2024-04-25T17:17:01.945 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T17:17:01.945 DEBUG:teuthology.run:Teuthology command: teuthology --name leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} --verbose -- /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560/orig.config.yaml 2024-04-25T17:17:02.005 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T17:17:02.089 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} email: null first_in_suite: false job_id: '7673560' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 3071830b3533a96301fd87d582ed5f17f0b618cd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2455 sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_ceph_ceph_3071830b3533a96301fd87d582ed5f17f0b618cd/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph suite_sha1: 3071830b3533a96301fd87d582ed5f17f0b618cd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNv7hWN3xpohe53vRVrnAB4MkRwwoJqHgMy5TMFJE7KVoRXVGBWbK//6orJckZSGDLq9weekpkgTK7IkWpjaAME= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJ8S3aYybZ0BCo44CqPJv9BgZsE7uxa63inWi1qLMetgIZ5oPKN8ASaJCRdSHRCqiZ0dmKVDgQGV5U5WXrd3pA= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH8irGOdpa7WxLAHIoGAsqHoyzfYqo/eum94UeChJ7UmDaWwbWu9Ix33CpBZrMtH9O/6tgpVpNKUG+yHYVzz7uE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_16:54:09 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T17:17:02.089 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_3071830b3533a96301fd87d582ed5f17f0b618cd/qa; will attempt to use it 2024-04-25T17:17:02.090 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_3071830b3533a96301fd87d582ed5f17f0b618cd/qa/tasks 2024-04-25T17:17:02.090 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T17:17:02.092 INFO:teuthology.task.internal:Checking packages... 2024-04-25T17:17:02.116 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd69f13d827952bb07bb9b7973bca14f97d764d6f' 2024-04-25T17:17:02.116 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T17:17:02.116 INFO:teuthology.packaging:ref: None 2024-04-25T17:17:02.116 INFO:teuthology.packaging:tag: None 2024-04-25T17:17:02.117 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-25T17:17:02.117 INFO:teuthology.packaging:sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:17:02.117 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-25T17:17:02.360 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.gd69f13d8 2024-04-25T17:17:02.362 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T17:17:02.370 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T17:17:02.370 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T17:17:02.385 INFO:teuthology.task.internal:Saving configuration 2024-04-25T17:17:02.408 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T17:17:02.414 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T17:17:02.433 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 17:11:52.628813', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNv7hWN3xpohe53vRVrnAB4MkRwwoJqHgMy5TMFJE7KVoRXVGBWbK//6orJckZSGDLq9weekpkgTK7IkWpjaAME='} 2024-04-25T17:17:02.453 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 17:11:52.630778', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJ8S3aYybZ0BCo44CqPJv9BgZsE7uxa63inWi1qLMetgIZ5oPKN8ASaJCRdSHRCqiZ0dmKVDgQGV5U5WXrd3pA='} 2024-04-25T17:17:02.470 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 17:11:52.632534', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH8irGOdpa7WxLAHIoGAsqHoyzfYqo/eum94UeChJ7UmDaWwbWu9Ix33CpBZrMtH9O/6tgpVpNKUG+yHYVzz7uE='} 2024-04-25T17:17:02.471 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T17:17:02.478 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T17:17:02.478 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T17:17:02.478 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T17:17:02.478 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T17:17:02.570 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcb0670d0d0>, signals=[15]) 2024-04-25T17:17:02.571 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T17:17:02.577 INFO:teuthology.task.internal:Opening connections... 2024-04-25T17:17:02.578 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:17:02.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T17:17:02.663 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T17:17:02.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T17:17:02.741 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T17:17:02.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T17:17:02.815 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T17:17:02.824 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-04-25T17:17:02.842 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-04-25T17:17:02.842 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-04-25T17:17:02.899 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-04-25T17:17:02.899 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-04-25T17:17:02.899 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-04-25T17:17:02.899 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-04-25T17:17:02.899 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-04-25T17:17:02.899 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-04-25T17:17:02.899 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T17:17:02.899 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-04-25T17:17:02.900 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-04-25T17:17:02.900 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T17:17:02.900 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-04-25T17:17:02.900 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T17:17:02.900 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T17:17:02.900 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T17:17:02.900 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-04-25T17:17:02.916 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-04-25T17:17:02.934 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-04-25T17:17:02.934 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-04-25T17:17:02.991 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-04-25T17:17:02.991 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-04-25T17:17:02.991 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-04-25T17:17:02.991 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-04-25T17:17:02.991 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-04-25T17:17:02.992 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-04-25T17:17:02.992 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T17:17:02.992 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-04-25T17:17:02.992 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-04-25T17:17:02.992 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T17:17:02.992 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-04-25T17:17:02.992 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T17:17:02.992 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T17:17:02.992 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T17:17:02.992 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-04-25T17:17:03.007 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-04-25T17:17:03.025 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-04-25T17:17:03.025 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-04-25T17:17:03.083 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-04-25T17:17:03.083 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-04-25T17:17:03.083 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-04-25T17:17:03.083 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-04-25T17:17:03.083 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-04-25T17:17:03.083 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-04-25T17:17:03.083 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T17:17:03.084 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-04-25T17:17:03.084 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-04-25T17:17:03.084 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T17:17:03.084 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-04-25T17:17:03.084 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T17:17:03.084 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T17:17:03.084 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T17:17:03.084 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-04-25T17:17:03.107 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T17:17:03.118 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T17:17:03.124 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T17:17:03.124 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-04-25T17:17:03.126 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-04-25T17:17:03.129 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-04-25T17:17:03.145 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T17:17:03.152 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T17:17:03.152 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-04-25T17:17:03.183 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-04-25T17:17:03.186 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-04-25T17:17:03.302 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T17:17:03.369 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T17:17:03.384 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-25T17:17:03.384 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-25T17:17:03.384 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-25T17:17:03.385 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T17:17:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:17:03.402 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-04-25T17:17:03.459 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-437.el9.x86_64 2024-04-25T17:17:03.460 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-437.el9.x86_64 2024-04-25T17:17:03.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-04-25T17:17:06.265 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 6.9 MB/s | 8.0 MB 00:01 2024-04-25T17:17:08.791 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-25T17:17:12.942 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T17:17:14.507 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 16 kB 00:00 2024-04-25T17:17:15.753 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-25T17:17:20.867 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-25T17:17:22.756 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T17:17:22.756 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T17:17:22.849 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-25T17:17:22.850 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-25T17:17:22.851 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-25T17:17:22.966 DEBUG:teuthology.orchestra.run.smithi098:> echo no | sudo yum reinstall kernel || true 2024-04-25T17:17:23.506 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 05:17:20 PM UTC. 2024-04-25T17:17:23.625 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalling: 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 255 k 2024-04-25T17:17:23.639 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 0 2024-04-25T17:17:23.640 INFO:teuthology.orchestra.run.smithi098.stderr:Operation aborted. 2024-04-25T17:17:23.695 INFO:teuthology.orchestra.run.smithi098.stdout:Is this ok [y/N]: 2024-04-25T17:17:23.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum reinstall -y kernel || true 2024-04-25T17:17:24.187 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 05:17:20 PM UTC. 2024-04-25T17:17:24.302 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-25T17:17:24.302 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T17:17:24.303 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-25T17:17:24.303 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T17:17:24.303 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalling: 2024-04-25T17:17:24.303 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T17:17:24.303 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:17:24.303 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-25T17:17:24.303 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T17:17:24.303 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:17:24.304 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 255 k 2024-04-25T17:17:24.304 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 0 2024-04-25T17:17:24.304 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-25T17:17:24.719 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.7 MB/s | 255 kB 00:00 2024-04-25T17:17:24.719 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-25T17:17:24.719 INFO:teuthology.orchestra.run.smithi098.stdout:Total 613 kB/s | 255 kB 00:00 2024-04-25T17:17:24.741 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-25T17:17:24.915 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-25T17:17:24.915 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-25T17:17:24.920 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-25T17:17:24.921 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-25T17:17:25.076 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-25T17:17:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T17:17:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T17:17:25.904 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T17:17:25.904 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T17:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T17:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalled: 2024-04-25T17:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T17:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:17:26.592 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-25T17:17:26.773 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T17:17:26.820 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T17:17:26.820 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T17:17:26.820 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T17:17:26.821 DEBUG:teuthology.parallel:result is None 2024-04-25T17:17:26.821 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T17:17:26.828 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T17:17:26.829 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T17:17:26.831 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T17:17:26.833 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T17:17:26.851 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T17:17:26.857 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T17:17:26.867 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T17:17:26.867 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T17:17:26.885 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T17:17:26.891 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T17:17:26.934 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T17:17:26.941 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T17:17:26.941 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T17:17:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:17:26.984 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T17:17:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:17:27.001 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T17:17:27.003 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T17:17:27.027 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T17:17:27.032 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T17:17:27.064 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T17:17:27.071 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T17:17:27.094 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T17:17:27.100 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T17:17:27.120 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T17:17:27.121 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T17:17:27.129 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T17:17:27.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T17:17:27.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T17:17:27.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T17:17:27.162 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T17:17:27.170 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T17:17:27.171 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T17:17:27.195 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T17:17:27.201 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T17:17:27.218 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T17:17:27.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T17:17:27.338 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T17:17:27.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T17:17:27.458 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:17:27.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T17:17:27.521 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T17:17:27.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T17:17:27.609 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T17:17:27.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T17:17:27.736 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:17:27.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T17:17:27.800 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T17:17:27.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T17:17:27.905 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T17:17:27.946 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T17:17:27.989 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:17:27.989 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T17:17:28.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-04-25T17:17:28.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-04-25T17:17:28.059 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-04-25T17:17:28.091 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T17:17:28.096 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T17:17:28.127 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T17:17:28.554 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T17:17:28.562 INFO:teuthology.task.internal:Starting timer... 2024-04-25T17:17:28.562 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T17:17:28.632 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T17:17:28.668 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T17:17:28.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-04-25T17:17:28.721 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-04-25T17:17:28.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-04-25T17:17:28.787 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-04-25T17:17:28.789 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-04-25T17:17:28.880 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-04-25T17:17:28.882 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T17:17:28.882 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-04-25T17:17:28.900 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-04-25T17:17:28.900 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-04-25T17:17:28.937 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-04-25T17:17:28.937 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-04-25T17:17:28.961 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-04-25T17:17:28.961 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-04-25T17:17:28.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T17:17:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:17:28.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T17:17:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:17:29.012 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T17:17:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:17:29.038 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T17:17:29.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-04-25T17:17:29.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-04-25T17:17:29.087 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-04-25T17:17:29.114 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T17:17:29.124 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T17:17:29.129 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T17:17:29.261 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T17:17:29.276 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T17:17:29.278 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi173.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T17:21:21.783 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-04-25T17:21:21.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-25T17:21:21.786 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T17:21:21.866 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-04-25T17:21:21.954 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-25T17:21:21.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-25T17:21:21.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T17:21:22.028 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-04-25T17:21:22.108 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-25T17:21:22.109 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-25T17:21:22.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T17:21:22.181 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-25T17:21:22.261 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-25T17:21:22.261 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T17:21:22.274 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T17:21:22.274 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T17:21:22.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T17:21:22.277 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T17:21:22.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T17:21:22.279 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T17:21:22.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T17:21:22.304 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T17:21:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T17:21:22.317 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T17:21:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T17:21:22.329 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T17:21:22.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-04-25T17:21:22.343 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T17:21:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-04-25T17:21:22.349 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-04-25T17:21:22.354 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-04-25T17:21:22.362 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T17:21:22.365 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-04-25T17:21:22.367 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T17:21:22.375 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T17:21:22.375 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-04-25T17:21:22.380 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T17:21:22.388 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T17:21:22.401 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T17:21:22.433 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-04-25T17:21:22.436 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T17:21:22.436 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-04-25T17:21:22.436 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.436 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.436 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.436 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.436 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-04-25T17:21:22.440 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T17:21:22.440 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-04-25T17:21:22.440 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.440 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.440 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.440 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.458 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-25T17:21:22.461 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T17:21:22.461 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-25T17:21:22.461 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.461 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.461 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.461 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:21:22.462 INFO:teuthology.run_tasks:Running task install... 2024-04-25T17:21:22.471 DEBUG:teuthology.task.install:project ceph 2024-04-25T17:21:22.471 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'}} 2024-04-25T17:21:22.471 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'} 2024-04-25T17:21:22.471 INFO:teuthology.task.install:Using flavor: default 2024-04-25T17:21:22.476 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T17:21:22.476 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T17:21:22.477 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-25T17:21:22.477 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:21:22.479 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-25T17:21:22.479 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:21:22.480 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-25T17:21:22.480 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:21:22.625 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-25T17:21:22.625 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-25T17:21:22.629 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-25T17:21:22.629 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-25T17:21:22.714 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-25T17:21:22.714 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-25T17:21:22.737 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T17:21:22.737 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:21:22.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T17:21:22.765 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T17:21:22.765 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T17:21:22.818 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T17:21:22.819 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:21:22.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T17:21:22.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T17:21:22.844 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T17:21:22.844 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T17:21:22.894 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T17:21:22.895 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:21:22.895 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T17:21:22.902 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T17:21:22.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T17:21:22.920 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T17:21:22.920 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T17:21:22.965 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-04-25T17:21:22.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-04-25T17:21:22.982 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T17:21:22.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T17:21:23.045 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-04-25T17:21:23.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-04-25T17:21:23.059 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T17:21:23.121 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-04-25T17:21:23.123 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-04-25T17:21:23.190 INFO:teuthology.orchestra.run.smithi173.stdout:48 files removed 2024-04-25T17:21:23.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T17:21:23.268 INFO:teuthology.orchestra.run.smithi098.stdout:48 files removed 2024-04-25T17:21:23.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T17:21:23.345 INFO:teuthology.orchestra.run.smithi203.stdout:48 files removed 2024-04-25T17:21:23.372 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T17:21:23.869 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 211 kB/s | 83 kB 00:00 2024-04-25T17:21:23.970 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 201 kB/s | 83 kB 00:00 2024-04-25T17:21:24.101 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 177 kB/s | 83 kB 00:00 2024-04-25T17:21:24.245 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-25T17:21:24.370 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 33 kB/s | 12 kB 00:00 2024-04-25T17:21:24.527 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-25T17:21:24.596 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-25T17:21:24.780 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-04-25T17:21:24.869 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-04-25T17:21:25.190 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-25T17:21:25.591 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-25T17:21:25.630 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-25T17:21:27.158 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-25T17:21:27.542 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-25T17:21:27.716 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-04-25T17:21:31.283 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T17:21:31.643 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T17:21:31.833 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T17:21:32.858 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-25T17:21:33.184 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-25T17:21:33.247 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-04-25T17:21:34.236 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-25T17:21:34.762 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-25T17:21:34.808 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-25T17:21:39.375 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-25T17:21:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T17:21:39.924 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-25T17:21:41.543 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T17:21:41.543 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T17:21:41.554 INFO:teuthology.orchestra.run.smithi173.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T17:21:41.555 INFO:teuthology.orchestra.run.smithi173.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T17:21:41.556 INFO:teuthology.orchestra.run.smithi173.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T17:21:41.557 INFO:teuthology.orchestra.run.smithi173.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T17:21:41.558 INFO:teuthology.orchestra.run.smithi173.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T17:21:41.561 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T17:21:41.561 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T17:21:41.621 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-04-25T17:21:41.650 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-25T17:21:41.650 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-04-25T17:21:41.650 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-25T17:21:41.650 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-04-25T17:21:41.650 INFO:teuthology.orchestra.run.smithi173.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T17:21:41.650 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-25T17:21:41.650 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T17:21:41.650 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-25T17:21:41.651 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-25T17:21:41.652 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T17:21:41.653 INFO:teuthology.orchestra.run.smithi173.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T17:21:41.654 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T17:21:41.655 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T17:21:41.656 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T17:21:41.657 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T17:21:41.658 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T17:21:41.659 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T17:21:41.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout:Install 148 Packages 2024-04-25T17:21:41.661 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 4 Packages 2024-04-25T17:21:41.662 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:21:41.662 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 222 M 2024-04-25T17:21:41.662 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-04-25T17:21:41.957 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T17:21:41.958 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T17:21:41.968 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T17:21:41.969 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T17:21:41.970 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T17:21:41.971 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T17:21:41.972 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T17:21:41.974 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T17:21:41.975 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T17:21:42.035 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-25T17:21:42.044 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-04-25T17:21:42.044 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-25T17:21:42.044 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-25T17:21:42.045 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T17:21:42.046 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T17:21:42.047 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T17:21:42.048 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T17:21:42.049 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T17:21:42.050 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T17:21:42.051 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T17:21:42.052 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T17:21:42.053 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T17:21:42.054 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T17:21:42.055 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:21:42.056 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-25T17:21:42.056 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-04-25T17:21:42.056 INFO:teuthology.orchestra.run.smithi098.stdout:Install 148 Packages 2024-04-25T17:21:42.056 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 4 Packages 2024-04-25T17:21:42.056 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:21:42.056 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 222 M 2024-04-25T17:21:42.056 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-25T17:21:42.109 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T17:21:42.110 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T17:21:42.120 INFO:teuthology.orchestra.run.smithi203.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T17:21:42.121 INFO:teuthology.orchestra.run.smithi203.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T17:21:42.122 INFO:teuthology.orchestra.run.smithi203.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T17:21:42.123 INFO:teuthology.orchestra.run.smithi203.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T17:21:42.124 INFO:teuthology.orchestra.run.smithi203.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T17:21:42.127 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T17:21:42.127 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T17:21:42.187 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-25T17:21:42.201 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-25T17:21:42.202 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-25T17:21:42.203 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-25T17:21:42.203 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-25T17:21:42.203 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-25T17:21:42.203 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-25T17:21:42.203 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-25T17:21:42.203 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-25T17:21:42.203 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-25T17:21:42.203 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-25T17:21:42.204 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-25T17:21:42.205 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T17:21:42.206 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T17:21:42.207 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T17:21:42.208 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T17:21:42.209 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T17:21:42.210 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T17:21:42.211 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T17:21:42.212 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:21:42.213 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-25T17:21:42.214 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-25T17:21:42.214 INFO:teuthology.orchestra.run.smithi203.stdout:Install 148 Packages 2024-04-25T17:21:42.214 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 4 Packages 2024-04-25T17:21:42.214 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:21:42.214 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 222 M 2024-04-25T17:21:42.214 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-25T17:21:42.551 INFO:teuthology.orchestra.run.smithi173.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 39 kB/s | 6.6 kB 00:00 2024-04-25T17:21:42.711 INFO:teuthology.orchestra.run.smithi098.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-25T17:21:42.868 INFO:teuthology.orchestra.run.smithi173.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 2.7 MB/s | 890 kB 00:00 2024-04-25T17:21:42.935 INFO:teuthology.orchestra.run.smithi173.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 2.1 MB/s | 145 kB 00:00 2024-04-25T17:21:43.036 INFO:teuthology.orchestra.run.smithi098.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 2.7 MB/s | 890 kB 00:00 2024-04-25T17:21:43.103 INFO:teuthology.orchestra.run.smithi098.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 2.1 MB/s | 145 kB 00:00 2024-04-25T17:21:43.284 INFO:teuthology.orchestra.run.smithi203.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 34 kB/s | 6.6 kB 00:00 2024-04-25T17:21:43.368 INFO:teuthology.orchestra.run.smithi173.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 5.4 MB/s | 2.4 MB 00:00 2024-04-25T17:21:43.502 INFO:teuthology.orchestra.run.smithi173.stdout:(5/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 4.8 MB/s | 5.4 MB 00:01 2024-04-25T17:21:43.727 INFO:teuthology.orchestra.run.smithi173.stdout:(6/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 4.2 MB/s | 1.5 MB 00:00 2024-04-25T17:21:43.737 INFO:teuthology.orchestra.run.smithi098.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 3.7 MB/s | 2.4 MB 00:00 2024-04-25T17:21:43.876 INFO:teuthology.orchestra.run.smithi203.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.5 MB/s | 890 kB 00:00 2024-04-25T17:21:44.035 INFO:teuthology.orchestra.run.smithi203.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 914 kB/s | 145 kB 00:00 2024-04-25T17:21:44.179 INFO:teuthology.orchestra.run.smithi098.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 3.4 MB/s | 1.5 MB 00:00 2024-04-25T17:21:44.346 INFO:teuthology.orchestra.run.smithi173.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 5.6 MB/s | 4.7 MB 00:00 2024-04-25T17:21:45.220 INFO:teuthology.orchestra.run.smithi098.stdout:(6/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 4.5 MB/s | 4.7 MB 00:01 2024-04-25T17:21:45.385 INFO:teuthology.orchestra.run.smithi203.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 1.7 MB/s | 2.4 MB 00:01 2024-04-25T17:21:45.621 INFO:teuthology.orchestra.run.smithi098.stdout:(7/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 1.7 MB/s | 5.4 MB 00:03 2024-04-25T17:21:45.853 INFO:teuthology.orchestra.run.smithi173.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 6.1 MB/s | 21 MB 00:03 2024-04-25T17:21:45.860 INFO:teuthology.orchestra.run.smithi203.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 3.1 MB/s | 1.5 MB 00:00 2024-04-25T17:21:45.928 INFO:teuthology.orchestra.run.smithi173.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 337 kB/s | 25 kB 00:00 2024-04-25T17:21:46.327 INFO:teuthology.orchestra.run.smithi203.stdout:(6/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 1.7 MB/s | 5.4 MB 00:03 2024-04-25T17:21:47.036 INFO:teuthology.orchestra.run.smithi173.stdout:(10/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 3.9 MB/s | 11 MB 00:02 2024-04-25T17:21:47.128 INFO:teuthology.orchestra.run.smithi173.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.4 MB/s | 225 kB 00:00 2024-04-25T17:21:47.211 INFO:teuthology.orchestra.run.smithi173.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 324 kB/s | 27 kB 00:00 2024-04-25T17:21:47.295 INFO:teuthology.orchestra.run.smithi173.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 380 kB/s | 32 kB 00:00 2024-04-25T17:21:47.478 INFO:teuthology.orchestra.run.smithi173.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 4.0 MB/s | 747 kB 00:00 2024-04-25T17:21:47.570 INFO:teuthology.orchestra.run.smithi173.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.7 MB/s | 163 kB 00:00 2024-04-25T17:21:47.654 INFO:teuthology.orchestra.run.smithi173.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.5 MB/s | 126 kB 00:00 2024-04-25T17:21:47.829 INFO:teuthology.orchestra.run.smithi173.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 2.8 MB/s | 500 kB 00:00 2024-04-25T17:21:48.104 INFO:teuthology.orchestra.run.smithi098.stdout:(8/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x86 5.8 MB/s | 17 MB 00:02 2024-04-25T17:21:48.196 INFO:teuthology.orchestra.run.smithi098.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 274 kB/s | 25 kB 00:00 2024-04-25T17:21:48.380 INFO:teuthology.orchestra.run.smithi173.stdout:(18/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 3.6 MB/s | 17 MB 00:04 2024-04-25T17:21:48.419 INFO:teuthology.orchestra.run.smithi203.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 1.8 MB/s | 4.7 MB 00:02 2024-04-25T17:21:48.488 INFO:teuthology.orchestra.run.smithi173.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 417 kB/s | 45 kB 00:00 2024-04-25T17:21:48.589 INFO:teuthology.orchestra.run.smithi173.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.3 MB/s | 135 kB 00:00 2024-04-25T17:21:48.697 INFO:teuthology.orchestra.run.smithi173.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.5 MB/s | 161 kB 00:00 2024-04-25T17:21:48.847 INFO:teuthology.orchestra.run.smithi173.stdout:(22/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 5.3 MB/s | 5.4 MB 00:01 2024-04-25T17:21:48.873 INFO:teuthology.orchestra.run.smithi173.stdout:(23/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.8 MB/s | 320 kB 00:00 2024-04-25T17:21:48.940 INFO:teuthology.orchestra.run.smithi173.stdout:(24/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 3.2 MB/s | 298 kB 00:00 2024-04-25T17:21:48.965 INFO:teuthology.orchestra.run.smithi173.stdout:(25/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.1 MB/s | 99 kB 00:00 2024-04-25T17:21:49.032 INFO:teuthology.orchestra.run.smithi173.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 923 kB/s | 85 kB 00:00 2024-04-25T17:21:49.133 INFO:teuthology.orchestra.run.smithi173.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.7 MB/s | 170 kB 00:00 2024-04-25T17:21:49.233 INFO:teuthology.orchestra.run.smithi173.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 342 kB/s | 34 kB 00:00 2024-04-25T17:21:49.342 INFO:teuthology.orchestra.run.smithi173.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.3 MB/s | 142 kB 00:00 2024-04-25T17:21:49.892 INFO:teuthology.orchestra.run.smithi173.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 3.3 MB/s | 3.1 MB 00:00 2024-04-25T17:21:50.217 INFO:teuthology.orchestra.run.smithi173.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 4.0 MB/s | 3.5 MB 00:00 2024-04-25T17:21:50.317 INFO:teuthology.orchestra.run.smithi173.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 2.4 MB/s | 249 kB 00:00 2024-04-25T17:21:50.401 INFO:teuthology.orchestra.run.smithi173.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 591 kB/s | 49 kB 00:00 2024-04-25T17:21:50.484 INFO:teuthology.orchestra.run.smithi173.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 198 kB/s | 16 kB 00:00 2024-04-25T17:21:50.585 INFO:teuthology.orchestra.run.smithi173.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 2.7 MB/s | 273 kB 00:00 2024-04-25T17:21:50.843 INFO:teuthology.orchestra.run.smithi173.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 2.9 MB/s | 762 kB 00:00 2024-04-25T17:21:50.935 INFO:teuthology.orchestra.run.smithi173.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 199 kB/s | 18 kB 00:00 2024-04-25T17:21:51.052 INFO:teuthology.orchestra.run.smithi173.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 689 kB/s | 80 kB 00:00 2024-04-25T17:21:51.127 INFO:teuthology.orchestra.run.smithi173.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 10 MB/s | 803 kB 00:00 2024-04-25T17:21:51.161 INFO:teuthology.orchestra.run.smithi173.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 5.7 MB/s | 195 kB 00:00 2024-04-25T17:21:51.211 INFO:teuthology.orchestra.run.smithi173.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-25T17:21:51.278 INFO:teuthology.orchestra.run.smithi173.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-25T17:21:51.306 INFO:teuthology.orchestra.run.smithi173.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 4.2 MB/s | 106 kB 00:00 2024-04-25T17:21:51.345 INFO:teuthology.orchestra.run.smithi173.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-25T17:21:51.379 INFO:teuthology.orchestra.run.smithi173.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 3.7 MB/s | 125 kB 00:00 2024-04-25T17:21:51.413 INFO:teuthology.orchestra.run.smithi173.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 6.3 MB/s | 215 kB 00:00 2024-04-25T17:21:51.463 INFO:teuthology.orchestra.run.smithi173.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 13 MB/s | 689 kB 00:00 2024-04-25T17:21:51.530 INFO:teuthology.orchestra.run.smithi173.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-25T17:21:51.564 INFO:teuthology.orchestra.run.smithi173.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 107 kB 00:00 2024-04-25T17:21:51.605 INFO:teuthology.orchestra.run.smithi173.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 517 kB/s | 21 kB 00:00 2024-04-25T17:21:51.647 INFO:teuthology.orchestra.run.smithi173.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 313 kB 00:00 2024-04-25T17:21:51.683 INFO:teuthology.orchestra.run.smithi173.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 924 kB/s | 33 kB 00:00 2024-04-25T17:21:51.792 INFO:teuthology.orchestra.run.smithi173.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-04-25T17:21:51.825 INFO:teuthology.orchestra.run.smithi173.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 549 kB/s | 18 kB 00:00 2024-04-25T17:21:51.838 INFO:teuthology.orchestra.run.smithi098.stdout:(10/152): ceph-common-19.0.0-3156.gd69f13d8.el7 2.3 MB/s | 21 MB 00:09 2024-04-25T17:21:51.859 INFO:teuthology.orchestra.run.smithi173.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 333 kB/s | 11 kB 00:00 2024-04-25T17:21:51.959 INFO:teuthology.orchestra.run.smithi173.stdout:(56/152): ceph-mgr-diskprediction-local-19.0.0- 3.6 MB/s | 7.4 MB 00:02 2024-04-25T17:21:51.976 INFO:teuthology.orchestra.run.smithi173.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-25T17:21:52.002 INFO:teuthology.orchestra.run.smithi173.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-25T17:21:52.027 INFO:teuthology.orchestra.run.smithi173.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-25T17:21:52.052 INFO:teuthology.orchestra.run.smithi173.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 582 kB 00:00 2024-04-25T17:21:52.069 INFO:teuthology.orchestra.run.smithi173.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 624 kB/s | 25 kB 00:00 2024-04-25T17:21:52.071 INFO:teuthology.orchestra.run.smithi098.stdout:(11/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 1.6 MB/s | 11 MB 00:06 2024-04-25T17:21:52.086 INFO:teuthology.orchestra.run.smithi173.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-25T17:21:52.096 INFO:teuthology.orchestra.run.smithi098.stdout:(12/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 870 kB/s | 225 kB 00:00 2024-04-25T17:21:52.111 INFO:teuthology.orchestra.run.smithi173.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-04-25T17:21:52.128 INFO:teuthology.orchestra.run.smithi173.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 4.5 MB/s | 188 kB 00:00 2024-04-25T17:21:52.145 INFO:teuthology.orchestra.run.smithi173.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 35 kB 00:00 2024-04-25T17:21:52.155 INFO:teuthology.orchestra.run.smithi098.stdout:(13/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 323 kB/s | 27 kB 00:00 2024-04-25T17:21:52.180 INFO:teuthology.orchestra.run.smithi098.stdout:(14/152): libcephfs-devel-19.0.0-3156.gd69f13d8 379 kB/s | 32 kB 00:00 2024-04-25T17:21:52.289 INFO:teuthology.orchestra.run.smithi098.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.5 MB/s | 163 kB 00:00 2024-04-25T17:21:52.304 INFO:teuthology.orchestra.run.smithi173.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-04-25T17:21:52.372 INFO:teuthology.orchestra.run.smithi098.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.5 MB/s | 126 kB 00:00 2024-04-25T17:21:52.396 INFO:teuthology.orchestra.run.smithi173.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-04-25T17:21:52.438 INFO:teuthology.orchestra.run.smithi173.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T17:21:52.471 INFO:teuthology.orchestra.run.smithi173.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-25T17:21:52.496 INFO:teuthology.orchestra.run.smithi173.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-25T17:21:52.539 INFO:teuthology.orchestra.run.smithi098.stdout:(17/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 1.9 MB/s | 747 kB 00:00 2024-04-25T17:21:52.565 INFO:teuthology.orchestra.run.smithi098.stdout:(18/152): libradosstriper1-19.0.0-3156.gd69f13d 2.5 MB/s | 500 kB 00:00 2024-04-25T17:21:52.640 INFO:teuthology.orchestra.run.smithi098.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 601 kB/s | 45 kB 00:00 2024-04-25T17:21:52.715 INFO:teuthology.orchestra.run.smithi098.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.8 MB/s | 135 kB 00:00 2024-04-25T17:21:52.790 INFO:teuthology.orchestra.run.smithi098.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 2.1 MB/s | 161 kB 00:00 2024-04-25T17:21:52.818 INFO:teuthology.orchestra.run.smithi203.stdout:(8/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 2.4 MB/s | 11 MB 00:04 2024-04-25T17:21:52.868 INFO:teuthology.orchestra.run.smithi203.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 506 kB/s | 25 kB 00:00 2024-04-25T17:21:52.907 INFO:teuthology.orchestra.run.smithi098.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 2.7 MB/s | 320 kB 00:00 2024-04-25T17:21:52.947 INFO:teuthology.orchestra.run.smithi173.stdout:(71/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 7.0 MB/s | 49 MB 00:07 2024-04-25T17:21:52.989 INFO:teuthology.orchestra.run.smithi173.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 2.1 MB/s | 1.0 MB 00:00 2024-04-25T17:21:53.033 INFO:teuthology.orchestra.run.smithi098.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.3 MB/s | 298 kB 00:00 2024-04-25T17:21:53.064 INFO:teuthology.orchestra.run.smithi173.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 9.8 MB/s | 5.8 MB 00:00 2024-04-25T17:21:53.089 INFO:teuthology.orchestra.run.smithi173.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 619 kB/s | 87 kB 00:00 2024-04-25T17:21:53.091 INFO:teuthology.orchestra.run.smithi098.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.7 MB/s | 99 kB 00:00 2024-04-25T17:21:53.159 INFO:teuthology.orchestra.run.smithi098.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.3 MB/s | 85 kB 00:00 2024-04-25T17:21:53.177 INFO:teuthology.orchestra.run.smithi203.stdout:(10/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 2.5 MB/s | 17 MB 00:06 2024-04-25T17:21:53.248 INFO:teuthology.orchestra.run.smithi173.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 6.0 MB 00:00 2024-04-25T17:21:53.273 INFO:teuthology.orchestra.run.smithi173.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 248 kB 00:00 2024-04-25T17:21:53.285 INFO:teuthology.orchestra.run.smithi203.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.0 MB/s | 225 kB 00:00 2024-04-25T17:21:53.298 INFO:teuthology.orchestra.run.smithi173.stdout:(77/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-04-25T17:21:53.323 INFO:teuthology.orchestra.run.smithi173.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 343 kB/s | 26 kB 00:00 2024-04-25T17:21:53.336 INFO:teuthology.orchestra.run.smithi203.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 541 kB/s | 27 kB 00:00 2024-04-25T17:21:53.349 INFO:teuthology.orchestra.run.smithi173.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 256 kB/s | 19 kB 00:00 2024-04-25T17:21:53.374 INFO:teuthology.orchestra.run.smithi173.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 697 kB/s | 35 kB 00:00 2024-04-25T17:21:53.386 INFO:teuthology.orchestra.run.smithi203.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 636 kB/s | 32 kB 00:00 2024-04-25T17:21:53.399 INFO:teuthology.orchestra.run.smithi173.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-25T17:21:53.424 INFO:teuthology.orchestra.run.smithi173.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-04-25T17:21:53.458 INFO:teuthology.orchestra.run.smithi173.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.8 MB/s | 222 kB 00:00 2024-04-25T17:21:53.500 INFO:teuthology.orchestra.run.smithi173.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-04-25T17:21:53.567 INFO:teuthology.orchestra.run.smithi173.stdout:(85/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-04-25T17:21:53.570 INFO:teuthology.orchestra.run.smithi203.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 4.0 MB/s | 747 kB 00:00 2024-04-25T17:21:53.592 INFO:teuthology.orchestra.run.smithi173.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 323 kB/s | 42 kB 00:00 2024-04-25T17:21:53.617 INFO:teuthology.orchestra.run.smithi173.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-04-25T17:21:53.637 INFO:teuthology.orchestra.run.smithi203.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 2.4 MB/s | 163 kB 00:00 2024-04-25T17:21:53.643 INFO:teuthology.orchestra.run.smithi173.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T17:21:53.668 INFO:teuthology.orchestra.run.smithi173.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-25T17:21:53.693 INFO:teuthology.orchestra.run.smithi173.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-04-25T17:21:53.695 INFO:teuthology.orchestra.run.smithi203.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 2.1 MB/s | 126 kB 00:00 2024-04-25T17:21:53.718 INFO:teuthology.orchestra.run.smithi173.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-04-25T17:21:53.744 INFO:teuthology.orchestra.run.smithi173.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2024-04-25T17:21:53.778 INFO:teuthology.orchestra.run.smithi173.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 777 kB/s | 44 kB 00:00 2024-04-25T17:21:53.819 INFO:teuthology.orchestra.run.smithi173.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.0 MB/s | 221 kB 00:00 2024-04-25T17:21:53.846 INFO:teuthology.orchestra.run.smithi203.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 3.3 MB/s | 500 kB 00:00 2024-04-25T17:21:53.861 INFO:teuthology.orchestra.run.smithi173.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 448 kB/s | 36 kB 00:00 2024-04-25T17:21:53.895 INFO:teuthology.orchestra.run.smithi173.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 656 kB/s | 47 kB 00:00 2024-04-25T17:21:53.953 INFO:teuthology.orchestra.run.smithi173.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.3 MB/s | 305 kB 00:00 2024-04-25T17:21:53.967 INFO:teuthology.orchestra.run.smithi098.stdout:(26/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 3.8 MB/s | 3.1 MB 00:00 2024-04-25T17:21:54.034 INFO:teuthology.orchestra.run.smithi098.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 2.5 MB/s | 170 kB 00:00 2024-04-25T17:21:54.110 INFO:teuthology.orchestra.run.smithi098.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 455 kB/s | 34 kB 00:00 2024-04-25T17:21:54.112 INFO:teuthology.orchestra.run.smithi173.stdout:(98/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-04-25T17:21:54.129 INFO:teuthology.orchestra.run.smithi173.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 250 kB/s | 58 kB 00:00 2024-04-25T17:21:54.154 INFO:teuthology.orchestra.run.smithi173.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 473 kB/s | 94 kB 00:00 2024-04-25T17:21:54.168 INFO:teuthology.orchestra.run.smithi098.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 2.4 MB/s | 142 kB 00:00 2024-04-25T17:21:54.171 INFO:teuthology.orchestra.run.smithi173.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-25T17:21:54.180 INFO:teuthology.orchestra.run.smithi203.stdout:(18/152): ceph-common-19.0.0-3156.gd69f13d8.el7 1.9 MB/s | 21 MB 00:11 2024-04-25T17:21:54.188 INFO:teuthology.orchestra.run.smithi173.stdout:(102/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T17:21:54.205 INFO:teuthology.orchestra.run.smithi173.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 444 kB/s | 22 kB 00:00 2024-04-25T17:21:54.247 INFO:teuthology.orchestra.run.smithi173.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-04-25T17:21:54.255 INFO:teuthology.orchestra.run.smithi203.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 602 kB/s | 45 kB 00:00 2024-04-25T17:21:54.361 INFO:teuthology.orchestra.run.smithi098.stdout:(30/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 2.9 MB/s | 5.4 MB 00:01 2024-04-25T17:21:54.372 INFO:teuthology.orchestra.run.smithi173.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-04-25T17:21:54.380 INFO:teuthology.orchestra.run.smithi203.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 1.1 MB/s | 135 kB 00:00 2024-04-25T17:21:54.472 INFO:teuthology.orchestra.run.smithi203.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.7 MB/s | 161 kB 00:00 2024-04-25T17:21:54.514 INFO:teuthology.orchestra.run.smithi173.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-04-25T17:21:54.539 INFO:teuthology.orchestra.run.smithi173.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-25T17:21:54.564 INFO:teuthology.orchestra.run.smithi173.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 518 kB/s | 26 kB 00:00 2024-04-25T17:21:54.598 INFO:teuthology.orchestra.run.smithi173.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2024-04-25T17:21:54.623 INFO:teuthology.orchestra.run.smithi203.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 2.1 MB/s | 320 kB 00:00 2024-04-25T17:21:54.623 INFO:teuthology.orchestra.run.smithi173.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-25T17:21:54.665 INFO:teuthology.orchestra.run.smithi173.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-25T17:21:54.741 INFO:teuthology.orchestra.run.smithi203.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.5 MB/s | 298 kB 00:00 2024-04-25T17:21:54.782 INFO:teuthology.orchestra.run.smithi173.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-25T17:21:54.816 INFO:teuthology.orchestra.run.smithi173.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-04-25T17:21:54.832 INFO:teuthology.orchestra.run.smithi203.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.1 MB/s | 99 kB 00:00 2024-04-25T17:21:54.858 INFO:teuthology.orchestra.run.smithi173.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-04-25T17:21:54.875 INFO:teuthology.orchestra.run.smithi173.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 471 kB/s | 43 kB 00:00 2024-04-25T17:21:54.892 INFO:teuthology.orchestra.run.smithi173.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 426 kB/s | 32 kB 00:00 2024-04-25T17:21:54.909 INFO:teuthology.orchestra.run.smithi173.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-25T17:21:54.958 INFO:teuthology.orchestra.run.smithi203.stdout:(25/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 4.8 MB/s | 5.4 MB 00:01 2024-04-25T17:21:54.959 INFO:teuthology.orchestra.run.smithi173.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-25T17:21:54.985 INFO:teuthology.orchestra.run.smithi203.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 567 kB/s | 85 kB 00:00 2024-04-25T17:21:54.992 INFO:teuthology.orchestra.run.smithi173.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-25T17:21:55.018 INFO:teuthology.orchestra.run.smithi173.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 606 kB/s | 35 kB 00:00 2024-04-25T17:21:55.043 INFO:teuthology.orchestra.run.smithi173.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-04-25T17:21:55.050 INFO:teuthology.orchestra.run.smithi203.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 2.5 MB/s | 170 kB 00:00 2024-04-25T17:21:55.061 INFO:teuthology.orchestra.run.smithi098.stdout:(31/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 7.2 MB/s | 49 MB 00:06 2024-04-25T17:21:55.068 INFO:teuthology.orchestra.run.smithi173.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-25T17:21:55.102 INFO:teuthology.orchestra.run.smithi173.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-25T17:21:55.117 INFO:teuthology.orchestra.run.smithi203.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 514 kB/s | 34 kB 00:00 2024-04-25T17:21:55.127 INFO:teuthology.orchestra.run.smithi173.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-04-25T17:21:55.128 INFO:teuthology.orchestra.run.smithi098.stdout:(32/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 3.7 MB/s | 3.5 MB 00:00 2024-04-25T17:21:55.153 INFO:teuthology.orchestra.run.smithi098.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3156.gd6 2.7 MB/s | 249 kB 00:00 2024-04-25T17:21:55.161 INFO:teuthology.orchestra.run.smithi173.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 303 kB/s | 18 kB 00:00 2024-04-25T17:21:55.184 INFO:teuthology.orchestra.run.smithi203.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 2.1 MB/s | 142 kB 00:00 2024-04-25T17:21:55.186 INFO:teuthology.orchestra.run.smithi173.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 396 kB/s | 23 kB 00:00 2024-04-25T17:21:55.195 INFO:teuthology.orchestra.run.smithi098.stdout:(34/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 741 kB/s | 49 kB 00:00 2024-04-25T17:21:55.220 INFO:teuthology.orchestra.run.smithi098.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3156.gd 246 kB/s | 16 kB 00:00 2024-04-25T17:21:55.228 INFO:teuthology.orchestra.run.smithi173.stdout:(127/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.7 MB/s | 2.0 MB 00:00 2024-04-25T17:21:55.245 INFO:teuthology.orchestra.run.smithi173.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-04-25T17:21:55.262 INFO:teuthology.orchestra.run.smithi173.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-25T17:21:55.304 INFO:teuthology.orchestra.run.smithi173.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 14 MB/s | 1.0 MB 00:00 2024-04-25T17:21:55.304 INFO:teuthology.orchestra.run.smithi098.stdout:(36/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 2.5 MB/s | 273 kB 00:00 2024-04-25T17:21:55.321 INFO:teuthology.orchestra.run.smithi173.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 794 kB/s | 46 kB 00:00 2024-04-25T17:21:55.346 INFO:teuthology.orchestra.run.smithi173.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 944 kB/s | 79 kB 00:00 2024-04-25T17:21:55.362 INFO:teuthology.orchestra.run.smithi098.stdout:(37/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 5.2 MB/s | 762 kB 00:00 2024-04-25T17:21:55.363 INFO:teuthology.orchestra.run.smithi173.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 982 kB/s | 58 kB 00:00 2024-04-25T17:21:55.387 INFO:teuthology.orchestra.run.smithi098.stdout:(38/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 218 kB/s | 18 kB 00:00 2024-04-25T17:21:55.396 INFO:teuthology.orchestra.run.smithi173.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-25T17:21:55.422 INFO:teuthology.orchestra.run.smithi173.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-04-25T17:21:55.447 INFO:teuthology.orchestra.run.smithi173.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-04-25T17:21:55.459 INFO:teuthology.orchestra.run.smithi203.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 6.2 MB/s | 3.1 MB 00:00 2024-04-25T17:21:55.472 INFO:teuthology.orchestra.run.smithi173.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-25T17:21:55.496 INFO:teuthology.orchestra.run.smithi098.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 603 kB/s | 80 kB 00:00 2024-04-25T17:21:55.497 INFO:teuthology.orchestra.run.smithi173.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-04-25T17:21:55.523 INFO:teuthology.orchestra.run.smithi173.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-25T17:21:55.547 INFO:teuthology.orchestra.run.smithi098.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-25T17:21:55.548 INFO:teuthology.orchestra.run.smithi173.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-04-25T17:21:55.572 INFO:teuthology.orchestra.run.smithi098.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-04-25T17:21:55.581 INFO:teuthology.orchestra.run.smithi173.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-04-25T17:21:55.597 INFO:teuthology.orchestra.run.smithi098.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-25T17:21:55.607 INFO:teuthology.orchestra.run.smithi173.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-04-25T17:21:55.632 INFO:teuthology.orchestra.run.smithi173.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-25T17:21:55.639 INFO:teuthology.orchestra.run.smithi098.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-25T17:21:55.657 INFO:teuthology.orchestra.run.smithi173.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-25T17:21:55.664 INFO:teuthology.orchestra.run.smithi098.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-25T17:21:55.683 INFO:teuthology.orchestra.run.smithi173.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-25T17:21:55.690 INFO:teuthology.orchestra.run.smithi098.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T17:21:55.708 INFO:teuthology.orchestra.run.smithi173.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-04-25T17:21:55.715 INFO:teuthology.orchestra.run.smithi098.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T17:21:55.740 INFO:teuthology.orchestra.run.smithi098.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T17:21:55.741 INFO:teuthology.orchestra.run.smithi173.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-25T17:21:55.774 INFO:teuthology.orchestra.run.smithi098.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 12 MB/s | 689 kB 00:00 2024-04-25T17:21:55.783 INFO:teuthology.orchestra.run.smithi173.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-25T17:21:55.816 INFO:teuthology.orchestra.run.smithi098.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-04-25T17:21:55.841 INFO:teuthology.orchestra.run.smithi098.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-25T17:21:55.850 INFO:teuthology.orchestra.run.smithi173.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 20 MB/s | 1.3 MB 00:00 2024-04-25T17:21:55.866 INFO:teuthology.orchestra.run.smithi098.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 436 kB/s | 21 kB 00:00 2024-04-25T17:21:55.892 INFO:teuthology.orchestra.run.smithi098.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T17:21:55.917 INFO:teuthology.orchestra.run.smithi098.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 660 kB/s | 33 kB 00:00 2024-04-25T17:21:55.934 INFO:teuthology.orchestra.run.smithi173.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-25T17:21:55.984 INFO:teuthology.orchestra.run.smithi098.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-04-25T17:21:55.993 INFO:teuthology.orchestra.run.smithi203.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 4.4 MB/s | 3.5 MB 00:00 2024-04-25T17:21:56.026 INFO:teuthology.orchestra.run.smithi098.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 171 kB/s | 18 kB 00:00 2024-04-25T17:21:56.051 INFO:teuthology.orchestra.run.smithi098.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 166 kB/s | 11 kB 00:00 2024-04-25T17:21:56.077 INFO:teuthology.orchestra.run.smithi098.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T17:21:56.094 INFO:teuthology.orchestra.run.smithi098.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-25T17:21:56.127 INFO:teuthology.orchestra.run.smithi098.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-25T17:21:56.135 INFO:teuthology.orchestra.run.smithi203.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.7 MB/s | 249 kB 00:00 2024-04-25T17:21:56.152 INFO:teuthology.orchestra.run.smithi098.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 582 kB 00:00 2024-04-25T17:21:56.178 INFO:teuthology.orchestra.run.smithi098.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 516 kB/s | 25 kB 00:00 2024-04-25T17:21:56.203 INFO:teuthology.orchestra.run.smithi098.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-04-25T17:21:56.235 INFO:teuthology.orchestra.run.smithi203.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 494 kB/s | 49 kB 00:00 2024-04-25T17:21:56.237 INFO:teuthology.orchestra.run.smithi098.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-25T17:21:56.262 INFO:teuthology.orchestra.run.smithi098.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-25T17:21:56.287 INFO:teuthology.orchestra.run.smithi098.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 706 kB/s | 35 kB 00:00 2024-04-25T17:21:56.319 INFO:teuthology.orchestra.run.smithi203.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 197 kB/s | 16 kB 00:00 2024-04-25T17:21:56.343 INFO:teuthology.orchestra.run.smithi173.stdout:(151/152): librados2-19.0.0-3156.gd69f13d8.el7. 5.4 MB/s | 3.4 MB 00:00 2024-04-25T17:21:56.388 INFO:teuthology.orchestra.run.smithi098.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-04-25T17:21:56.393 INFO:teuthology.orchestra.run.smithi173.stdout:(152/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 4.8 MB/s | 3.1 MB 00:00 2024-04-25T17:21:56.397 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-04-25T17:21:56.397 INFO:teuthology.orchestra.run.smithi173.stdout:Total 15 MB/s | 222 MB 00:14 2024-04-25T17:21:56.436 INFO:teuthology.orchestra.run.smithi203.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 2.3 MB/s | 273 kB 00:00 2024-04-25T17:21:56.463 INFO:teuthology.orchestra.run.smithi098.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 26 MB/s | 4.4 MB 00:00 2024-04-25T17:21:56.505 INFO:teuthology.orchestra.run.smithi098.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-25T17:21:56.520 INFO:teuthology.orchestra.run.smithi203.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 7.0 MB/s | 7.4 MB 00:01 2024-04-25T17:21:56.539 INFO:teuthology.orchestra.run.smithi098.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-25T17:21:56.564 INFO:teuthology.orchestra.run.smithi098.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-25T17:21:56.587 INFO:teuthology.orchestra.run.smithi203.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 274 kB/s | 18 kB 00:00 2024-04-25T17:21:56.689 INFO:teuthology.orchestra.run.smithi098.stdout:(71/152): postgresql-server-13.14-1.el9.x86_64. 39 MB/s | 5.8 MB 00:00 2024-04-25T17:21:56.695 INFO:teuthology.orchestra.run.smithi203.stdout:(38/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 2.9 MB/s | 762 kB 00:00 2024-04-25T17:21:56.720 INFO:teuthology.orchestra.run.smithi203.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 80 kB 00:00 2024-04-25T17:21:56.723 INFO:teuthology.orchestra.run.smithi098.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-04-25T17:21:56.748 INFO:teuthology.orchestra.run.smithi098.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-25T17:21:56.779 INFO:teuthology.orchestra.run.smithi203.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 803 kB 00:00 2024-04-25T17:21:56.805 INFO:teuthology.orchestra.run.smithi203.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-25T17:21:56.830 INFO:teuthology.orchestra.run.smithi203.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-04-25T17:21:56.872 INFO:teuthology.orchestra.run.smithi203.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-25T17:21:56.873 INFO:teuthology.orchestra.run.smithi098.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 40 MB/s | 6.0 MB 00:00 2024-04-25T17:21:56.898 INFO:teuthology.orchestra.run.smithi203.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-25T17:21:56.899 INFO:teuthology.orchestra.run.smithi098.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-04-25T17:21:56.923 INFO:teuthology.orchestra.run.smithi203.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T17:21:56.924 INFO:teuthology.orchestra.run.smithi098.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-25T17:21:56.948 INFO:teuthology.orchestra.run.smithi203.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T17:21:56.949 INFO:teuthology.orchestra.run.smithi098.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 516 kB/s | 26 kB 00:00 2024-04-25T17:21:56.974 INFO:teuthology.orchestra.run.smithi203.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T17:21:56.975 INFO:teuthology.orchestra.run.smithi098.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-04-25T17:21:56.999 INFO:teuthology.orchestra.run.smithi203.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 14 MB/s | 689 kB 00:00 2024-04-25T17:21:57.000 INFO:teuthology.orchestra.run.smithi098.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-25T17:21:57.025 INFO:teuthology.orchestra.run.smithi098.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-04-25T17:21:57.033 INFO:teuthology.orchestra.run.smithi203.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-04-25T17:21:57.058 INFO:teuthology.orchestra.run.smithi203.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T17:21:57.083 INFO:teuthology.orchestra.run.smithi203.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-25T17:21:57.109 INFO:teuthology.orchestra.run.smithi203.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T17:21:57.117 INFO:teuthology.orchestra.run.smithi098.stdout:(81/152): ceph-mgr-diskprediction-local-19.0.0- 2.7 MB/s | 7.4 MB 00:02 2024-04-25T17:21:57.134 INFO:teuthology.orchestra.run.smithi203.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 661 kB/s | 33 kB 00:00 2024-04-25T17:21:57.193 INFO:teuthology.orchestra.run.smithi098.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 5.1 MB 00:00 2024-04-25T17:21:57.201 INFO:teuthology.orchestra.run.smithi203.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-04-25T17:21:57.218 INFO:teuthology.orchestra.run.smithi098.stdout:(83/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 250 kB 00:00 2024-04-25T17:21:57.226 INFO:teuthology.orchestra.run.smithi203.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 206 kB/s | 18 kB 00:00 2024-04-25T17:21:57.252 INFO:teuthology.orchestra.run.smithi098.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-04-25T17:21:57.252 INFO:teuthology.orchestra.run.smithi203.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-25T17:21:57.278 INFO:teuthology.orchestra.run.smithi203.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T17:21:57.278 INFO:teuthology.orchestra.run.smithi098.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-25T17:21:57.302 INFO:teuthology.orchestra.run.smithi098.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 506 kB/s | 42 kB 00:00 2024-04-25T17:21:57.302 INFO:teuthology.orchestra.run.smithi203.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 906 kB/s | 45 kB 00:00 2024-04-25T17:21:57.327 INFO:teuthology.orchestra.run.smithi098.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-25T17:21:57.336 INFO:teuthology.orchestra.run.smithi203.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T17:21:57.352 INFO:teuthology.orchestra.run.smithi098.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T17:21:57.361 INFO:teuthology.orchestra.run.smithi203.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 10 MB/s | 582 kB 00:00 2024-04-25T17:21:57.378 INFO:teuthology.orchestra.run.smithi203.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 624 kB/s | 25 kB 00:00 2024-04-25T17:21:57.386 INFO:teuthology.orchestra.run.smithi098.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.3 MB/s | 279 kB 00:00 2024-04-25T17:21:57.395 INFO:teuthology.orchestra.run.smithi203.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-25T17:21:57.403 INFO:teuthology.orchestra.run.smithi098.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 682 kB/s | 51 kB 00:00 2024-04-25T17:21:57.420 INFO:teuthology.orchestra.run.smithi098.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-04-25T17:21:57.421 INFO:teuthology.orchestra.run.smithi203.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 292 kB 00:00 2024-04-25T17:21:57.446 INFO:teuthology.orchestra.run.smithi203.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T17:21:57.472 INFO:teuthology.orchestra.run.smithi203.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-25T17:21:57.495 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-04-25T17:21:57.545 INFO:teuthology.orchestra.run.smithi098.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 296 kB/s | 42 kB 00:00 2024-04-25T17:21:57.620 INFO:teuthology.orchestra.run.smithi098.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 223 kB/s | 44 kB 00:00 2024-04-25T17:21:57.637 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-04-25T17:21:57.637 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-04-25T17:21:57.655 INFO:teuthology.orchestra.run.smithi203.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 23 MB/s | 4.7 MB 00:00 2024-04-25T17:21:57.747 INFO:teuthology.orchestra.run.smithi203.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-25T17:21:57.787 INFO:teuthology.orchestra.run.smithi098.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-04-25T17:21:57.789 INFO:teuthology.orchestra.run.smithi203.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T17:21:57.813 INFO:teuthology.orchestra.run.smithi098.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 833 kB/s | 221 kB 00:00 2024-04-25T17:21:57.831 INFO:teuthology.orchestra.run.smithi203.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-25T17:21:57.838 INFO:teuthology.orchestra.run.smithi098.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 167 kB/s | 36 kB 00:00 2024-04-25T17:21:57.856 INFO:teuthology.orchestra.run.smithi203.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-25T17:21:57.863 INFO:teuthology.orchestra.run.smithi098.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-25T17:21:57.888 INFO:teuthology.orchestra.run.smithi098.stdout:(98/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-04-25T17:21:57.913 INFO:teuthology.orchestra.run.smithi098.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 771 kB/s | 58 kB 00:00 2024-04-25T17:21:57.939 INFO:teuthology.orchestra.run.smithi098.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-25T17:21:57.964 INFO:teuthology.orchestra.run.smithi098.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T17:21:57.989 INFO:teuthology.orchestra.run.smithi098.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 595 kB/s | 45 kB 00:00 2024-04-25T17:21:58.007 INFO:teuthology.orchestra.run.smithi203.stdout:(71/152): postgresql-server-13.14-1.el9.x86_64. 33 MB/s | 5.8 MB 00:00 2024-04-25T17:21:58.014 INFO:teuthology.orchestra.run.smithi098.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-04-25T17:21:58.049 INFO:teuthology.orchestra.run.smithi203.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-04-25T17:21:58.057 INFO:teuthology.orchestra.run.smithi098.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 13 MB/s | 863 kB 00:00 2024-04-25T17:21:58.074 INFO:teuthology.orchestra.run.smithi203.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-25T17:21:58.208 INFO:teuthology.orchestra.run.smithi203.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 38 MB/s | 6.0 MB 00:00 2024-04-25T17:21:58.242 INFO:teuthology.orchestra.run.smithi203.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-04-25T17:21:58.267 INFO:teuthology.orchestra.run.smithi203.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-25T17:21:58.292 INFO:teuthology.orchestra.run.smithi203.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 513 kB/s | 26 kB 00:00 2024-04-25T17:21:58.318 INFO:teuthology.orchestra.run.smithi203.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-25T17:21:58.340 INFO:teuthology.orchestra.run.smithi098.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 69 kB/s | 19 kB 00:00 2024-04-25T17:21:58.343 INFO:teuthology.orchestra.run.smithi203.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-25T17:21:58.368 INFO:teuthology.orchestra.run.smithi203.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-04-25T17:21:58.469 INFO:teuthology.orchestra.run.smithi203.stdout:(81/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-04-25T17:21:58.494 INFO:teuthology.orchestra.run.smithi203.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 250 kB 00:00 2024-04-25T17:21:58.519 INFO:teuthology.orchestra.run.smithi203.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-04-25T17:21:58.544 INFO:teuthology.orchestra.run.smithi203.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T17:21:58.578 INFO:teuthology.orchestra.run.smithi203.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 723 kB/s | 42 kB 00:00 2024-04-25T17:21:58.604 INFO:teuthology.orchestra.run.smithi203.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-25T17:21:58.624 INFO:teuthology.orchestra.run.smithi098.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 508 kB/s | 309 kB 00:00 2024-04-25T17:21:58.629 INFO:teuthology.orchestra.run.smithi203.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T17:21:58.654 INFO:teuthology.orchestra.run.smithi203.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-25T17:21:58.671 INFO:teuthology.orchestra.run.smithi203.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-25T17:21:58.688 INFO:teuthology.orchestra.run.smithi203.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-04-25T17:21:58.699 INFO:teuthology.orchestra.run.smithi098.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 773 kB/s | 548 kB 00:00 2024-04-25T17:21:58.732 INFO:teuthology.orchestra.run.smithi098.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-04-25T17:21:58.781 INFO:teuthology.orchestra.run.smithi203.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 454 kB/s | 42 kB 00:00 2024-04-25T17:21:58.824 INFO:teuthology.orchestra.run.smithi098.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-04-25T17:21:58.850 INFO:teuthology.orchestra.run.smithi098.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 576 kB/s | 67 kB 00:00 2024-04-25T17:21:58.950 INFO:teuthology.orchestra.run.smithi098.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-25T17:21:59.125 INFO:teuthology.orchestra.run.smithi098.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-04-25T17:21:59.131 INFO:teuthology.orchestra.run.smithi203.stdout:(92/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-04-25T17:21:59.156 INFO:teuthology.orchestra.run.smithi203.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 118 kB/s | 44 kB 00:00 2024-04-25T17:21:59.159 INFO:teuthology.orchestra.run.smithi098.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-04-25T17:21:59.181 INFO:teuthology.orchestra.run.smithi203.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-25T17:21:59.207 INFO:teuthology.orchestra.run.smithi203.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-04-25T17:21:59.226 INFO:teuthology.orchestra.run.smithi098.stdout:(114/152): libarrow-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2024-04-25T17:21:59.232 INFO:teuthology.orchestra.run.smithi203.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-25T17:21:59.252 INFO:teuthology.orchestra.run.smithi098.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 345 kB/s | 43 kB 00:00 2024-04-25T17:21:59.258 INFO:teuthology.orchestra.run.smithi203.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-25T17:21:59.277 INFO:teuthology.orchestra.run.smithi098.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 274 kB/s | 32 kB 00:00 2024-04-25T17:21:59.283 INFO:teuthology.orchestra.run.smithi203.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T17:21:59.317 INFO:teuthology.orchestra.run.smithi203.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-25T17:21:59.335 INFO:teuthology.orchestra.run.smithi098.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 137 kB/s | 15 kB 00:00 2024-04-25T17:21:59.342 INFO:teuthology.orchestra.run.smithi203.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T17:21:59.367 INFO:teuthology.orchestra.run.smithi203.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 895 kB/s | 45 kB 00:00 2024-04-25T17:21:59.369 INFO:teuthology.orchestra.run.smithi098.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-25T17:21:59.392 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-04-25T17:21:59.393 INFO:teuthology.orchestra.run.smithi203.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-04-25T17:21:59.393 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-04-25T17:21:59.426 INFO:teuthology.orchestra.run.smithi203.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 14 MB/s | 863 kB 00:00 2024-04-25T17:21:59.486 INFO:teuthology.orchestra.run.smithi098.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 1.7 MB/s | 358 kB 00:00 2024-04-25T17:21:59.529 INFO:teuthology.orchestra.run.smithi098.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 221 kB/s | 35 kB 00:00 2024-04-25T17:21:59.553 INFO:teuthology.orchestra.run.smithi098.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.4 MB/s | 303 kB 00:00 2024-04-25T17:21:59.604 INFO:teuthology.orchestra.run.smithi098.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.7 MB/s | 206 kB 00:00 2024-04-25T17:21:59.662 INFO:teuthology.orchestra.run.smithi098.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-25T17:21:59.712 INFO:teuthology.orchestra.run.smithi098.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-25T17:21:59.771 INFO:teuthology.orchestra.run.smithi098.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-25T17:21:59.821 INFO:teuthology.orchestra.run.smithi098.stdout:(126/152): python3-grpcio-1.46.7-10.el9.x86_64. 7.0 MB/s | 2.0 MB 00:00 2024-04-25T17:21:59.847 INFO:teuthology.orchestra.run.smithi098.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 173 kB/s | 23 kB 00:00 2024-04-25T17:21:59.880 INFO:teuthology.orchestra.run.smithi098.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 178 kB/s | 19 kB 00:00 2024-04-25T17:21:59.922 INFO:teuthology.orchestra.run.smithi098.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 195 kB/s | 20 kB 00:00 2024-04-25T17:21:59.968 INFO:teuthology.orchestra.run.smithi203.stdout:(104/152): ceph-test-19.0.0-3156.gd69f13d8.el7. 6.9 MB/s | 49 MB 00:07 2024-04-25T17:21:59.981 INFO:teuthology.orchestra.run.smithi098.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-04-25T17:22:00.031 INFO:teuthology.orchestra.run.smithi098.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-04-25T17:22:00.056 INFO:teuthology.orchestra.run.smithi098.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 589 kB/s | 79 kB 00:00 2024-04-25T17:22:00.082 INFO:teuthology.orchestra.run.smithi098.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 574 kB/s | 58 kB 00:00 2024-04-25T17:22:00.149 INFO:teuthology.orchestra.run.smithi098.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-25T17:22:00.165 INFO:teuthology.orchestra.run.smithi098.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-25T17:22:00.191 INFO:teuthology.orchestra.run.smithi098.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 827 kB/s | 90 kB 00:00 2024-04-25T17:22:00.235 INFO:teuthology.orchestra.run.smithi203.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-04-25T17:22:00.249 INFO:teuthology.orchestra.run.smithi098.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-04-25T17:22:00.275 INFO:teuthology.orchestra.run.smithi098.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-04-25T17:22:00.285 INFO:teuthology.orchestra.run.smithi203.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 360 kB/s | 309 kB 00:00 2024-04-25T17:22:00.300 INFO:teuthology.orchestra.run.smithi098.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-25T17:22:00.359 INFO:teuthology.orchestra.run.smithi098.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 543 kB/s | 59 kB 00:00 2024-04-25T17:22:00.369 INFO:teuthology.orchestra.run.smithi203.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 562 kB/s | 548 kB 00:00 2024-04-25T17:22:00.384 INFO:teuthology.orchestra.run.smithi098.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 328 kB/s | 36 kB 00:00 2024-04-25T17:22:00.402 INFO:teuthology.orchestra.run.smithi203.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-04-25T17:22:00.409 INFO:teuthology.orchestra.run.smithi098.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 469 kB/s | 51 kB 00:00 2024-04-25T17:22:00.476 INFO:teuthology.orchestra.run.smithi098.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-25T17:22:00.495 INFO:teuthology.orchestra.run.smithi203.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-04-25T17:22:00.510 INFO:teuthology.orchestra.run.smithi098.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 713 kB/s | 90 kB 00:00 2024-04-25T17:22:00.520 INFO:teuthology.orchestra.run.smithi203.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 576 kB/s | 67 kB 00:00 2024-04-25T17:22:00.535 INFO:teuthology.orchestra.run.smithi098.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-25T17:22:00.586 INFO:teuthology.orchestra.run.smithi098.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 183 kB/s | 20 kB 00:00 2024-04-25T17:22:00.619 INFO:teuthology.orchestra.run.smithi098.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-25T17:22:00.620 INFO:teuthology.orchestra.run.smithi203.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-25T17:22:00.669 INFO:teuthology.orchestra.run.smithi098.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T17:22:00.736 INFO:teuthology.orchestra.run.smithi098.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 20 MB/s | 1.3 MB 00:00 2024-04-25T17:22:00.820 INFO:teuthology.orchestra.run.smithi098.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-25T17:22:00.854 INFO:teuthology.orchestra.run.smithi203.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 2.3 MB/s | 548 kB 00:00 2024-04-25T17:22:01.020 INFO:teuthology.orchestra.run.smithi098.stdout:(151/152): librados2-19.0.0-3156.gd69f13d8.el7. 7.8 MB/s | 3.4 MB 00:00 2024-04-25T17:22:01.029 INFO:teuthology.orchestra.run.smithi203.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 248 kB/s | 43 kB 00:00 2024-04-25T17:22:01.163 INFO:teuthology.orchestra.run.smithi203.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 241 kB/s | 32 kB 00:00 2024-04-25T17:22:01.305 INFO:teuthology.orchestra.run.smithi203.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 106 kB/s | 15 kB 00:00 2024-04-25T17:22:01.447 INFO:teuthology.orchestra.run.smithi203.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2024-04-25T17:22:01.470 INFO:teuthology.orchestra.run.smithi098.stdout:(152/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 3.7 MB/s | 3.1 MB 00:00 2024-04-25T17:22:01.476 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-25T17:22:01.476 INFO:teuthology.orchestra.run.smithi098.stdout:Total 11 MB/s | 222 MB 00:19 2024-04-25T17:22:01.556 INFO:teuthology.orchestra.run.smithi203.stdout:(117/152): libarrow-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-04-25T17:22:01.598 INFO:teuthology.orchestra.run.smithi203.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-04-25T17:22:01.706 INFO:teuthology.orchestra.run.smithi203.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-04-25T17:22:01.732 INFO:teuthology.orchestra.run.smithi203.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 264 kB/s | 35 kB 00:00 2024-04-25T17:22:01.865 INFO:teuthology.orchestra.run.smithi203.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.3 MB/s | 206 kB 00:00 2024-04-25T17:22:02.007 INFO:teuthology.orchestra.run.smithi203.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-04-25T17:22:02.133 INFO:teuthology.orchestra.run.smithi203.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 85 kB/s | 11 kB 00:00 2024-04-25T17:22:02.241 INFO:teuthology.orchestra.run.smithi203.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-25T17:22:02.308 INFO:teuthology.orchestra.run.smithi203.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 3.5 MB/s | 2.0 MB 00:00 2024-04-25T17:22:02.350 INFO:teuthology.orchestra.run.smithi203.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-04-25T17:22:02.409 INFO:teuthology.orchestra.run.smithi203.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-04-25T17:22:02.476 INFO:teuthology.orchestra.run.smithi203.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 157 kB/s | 20 kB 00:00 2024-04-25T17:22:02.591 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-25T17:22:02.618 INFO:teuthology.orchestra.run.smithi203.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 327 kB/s | 46 kB 00:00 2024-04-25T17:22:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-25T17:22:02.736 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-25T17:22:02.769 INFO:teuthology.orchestra.run.smithi203.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 526 kB/s | 79 kB 00:00 2024-04-25T17:22:02.843 INFO:teuthology.orchestra.run.smithi203.stdout:(131/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 361 kB/s | 838 kB 00:02 2024-04-25T17:22:02.911 INFO:teuthology.orchestra.run.smithi203.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 406 kB/s | 58 kB 00:00 2024-04-25T17:22:03.094 INFO:teuthology.orchestra.run.smithi203.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 89 kB/s | 16 kB 00:00 2024-04-25T17:22:03.128 INFO:teuthology.orchestra.run.smithi203.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 959 kB/s | 272 kB 00:00 2024-04-25T17:22:03.153 INFO:teuthology.orchestra.run.smithi203.stdout:(135/152): python3-kubernetes-26.1.0-2.el9.noar 1.4 MB/s | 1.0 MB 00:00 2024-04-25T17:22:03.228 INFO:teuthology.orchestra.run.smithi203.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 672 kB/s | 90 kB 00:00 2024-04-25T17:22:03.254 INFO:teuthology.orchestra.run.smithi203.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 224 kB/s | 28 kB 00:00 2024-04-25T17:22:03.279 INFO:teuthology.orchestra.run.smithi203.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 245 kB/s | 31 kB 00:00 2024-04-25T17:22:03.379 INFO:teuthology.orchestra.run.smithi203.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-25T17:22:03.405 INFO:teuthology.orchestra.run.smithi203.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 392 kB/s | 59 kB 00:00 2024-04-25T17:22:03.430 INFO:teuthology.orchestra.run.smithi203.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 237 kB/s | 36 kB 00:00 2024-04-25T17:22:03.489 INFO:teuthology.orchestra.run.smithi203.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 468 kB/s | 51 kB 00:00 2024-04-25T17:22:03.564 INFO:teuthology.orchestra.run.smithi203.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-04-25T17:22:03.589 INFO:teuthology.orchestra.run.smithi203.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 564 kB/s | 90 kB 00:00 2024-04-25T17:22:03.681 INFO:teuthology.orchestra.run.smithi203.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-04-25T17:22:03.707 INFO:teuthology.orchestra.run.smithi203.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 1.9 MB/s | 427 kB 00:00 2024-04-25T17:22:03.757 INFO:teuthology.orchestra.run.smithi203.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-25T17:22:04.074 INFO:teuthology.orchestra.run.smithi203.stdout:(148/152): librados2-19.0.0-3156.gd69f13d8.el7. 9.3 MB/s | 3.4 MB 00:00 2024-04-25T17:22:04.132 INFO:teuthology.orchestra.run.smithi203.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-04-25T17:22:04.174 INFO:teuthology.orchestra.run.smithi203.stdout:(150/152): thrift-0.15.0-2.el9.x86_64.rpm 3.2 MB/s | 1.6 MB 00:00 2024-04-25T17:22:04.225 INFO:teuthology.orchestra.run.smithi203.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-04-25T17:22:04.275 INFO:teuthology.orchestra.run.smithi203.stdout:(152/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 6.1 MB/s | 3.1 MB 00:00 2024-04-25T17:22:04.280 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-25T17:22:04.280 INFO:teuthology.orchestra.run.smithi203.stdout:Total 10 MB/s | 222 MB 00:22 2024-04-25T17:22:04.520 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-25T17:22:04.562 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-25T17:22:05.369 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-25T17:22:05.462 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-04-25T17:22:05.513 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-25T17:22:05.513 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-25T17:22:05.571 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T17:22:05.685 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T17:22:05.865 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T17:22:05.914 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T17:22:05.939 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T17:22:05.983 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T17:22:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T17:22:06.432 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T17:22:06.553 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T17:22:06.591 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T17:22:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T17:22:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T17:22:06.850 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T17:22:06.900 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T17:22:06.928 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T17:22:07.023 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T17:22:07.046 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T17:22:07.092 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T17:22:07.114 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T17:22:07.133 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T17:22:07.290 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-25T17:22:07.292 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-25T17:22:07.314 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T17:22:07.347 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T17:22:07.453 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T17:22:07.542 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T17:22:07.579 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T17:22:07.614 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T17:22:07.688 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T17:22:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T17:22:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T17:22:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T17:22:07.851 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T17:22:07.922 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T17:22:07.974 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T17:22:08.012 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T17:22:08.056 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T17:22:08.133 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T17:22:08.167 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-25T17:22:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-25T17:22:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-25T17:22:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T17:22:08.378 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T17:22:08.452 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T17:22:08.588 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T17:22:08.639 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T17:22:09.153 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T17:22:09.234 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T17:22:09.263 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T17:22:09.306 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T17:22:09.344 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-25T17:22:09.386 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T17:22:09.550 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T17:22:09.622 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T17:22:09.676 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T17:22:09.741 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T17:22:09.773 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T17:22:09.809 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T17:22:09.856 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T17:22:09.892 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T17:22:09.934 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T17:22:09.977 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T17:22:10.111 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T17:22:10.157 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T17:22:10.187 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T17:22:10.228 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T17:22:10.324 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T17:22:10.371 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T17:22:10.411 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T17:22:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T17:22:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T17:22:10.537 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T17:22:10.574 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-25T17:22:10.662 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T17:22:10.666 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T17:22:10.693 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T17:22:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T17:22:10.720 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T17:22:10.764 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T17:22:10.766 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T17:22:10.803 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T17:22:10.808 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T17:22:10.834 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T17:22:10.834 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T17:22:10.872 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T17:22:10.877 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T17:22:10.907 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T17:22:10.940 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T17:22:10.982 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T17:22:11.016 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T17:22:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T17:22:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T17:22:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T17:22:11.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T17:22:11.309 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T17:22:11.319 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T17:22:11.440 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T17:22:11.478 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T17:22:11.517 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T17:22:11.569 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T17:22:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T17:22:11.599 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T17:22:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T17:22:11.804 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T17:22:11.832 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T17:22:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T17:22:11.958 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T17:22:12.030 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T17:22:12.051 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T17:22:12.073 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T17:22:12.268 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T17:22:12.291 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T17:22:12.301 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T17:22:12.335 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T17:22:12.368 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T17:22:12.408 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T17:22:12.467 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T17:22:12.505 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T17:22:12.526 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T17:22:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T17:22:12.569 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T17:22:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T17:22:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T17:22:12.642 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T17:22:12.676 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T17:22:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T17:22:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T17:22:12.780 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T17:22:12.815 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T17:22:12.851 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T17:22:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T17:22:12.950 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T17:22:12.994 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T17:22:13.010 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T17:22:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T17:22:13.089 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T17:22:13.113 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-25T17:22:13.156 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-25T17:22:13.185 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-25T17:22:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T17:22:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T17:22:13.375 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T17:22:13.519 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T17:22:13.541 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-25T17:22:13.578 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T17:22:13.629 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T17:22:13.667 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T17:22:13.721 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T17:22:13.731 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T17:22:13.755 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T17:22:13.789 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T17:22:13.840 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T17:22:13.840 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T17:22:14.136 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T17:22:14.187 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T17:22:14.222 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T17:22:14.252 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T17:22:14.294 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T17:22:14.307 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T17:22:14.332 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-25T17:22:14.374 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T17:22:14.422 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T17:22:14.450 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T17:22:14.525 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T17:22:14.534 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T17:22:14.562 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T17:22:14.594 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T17:22:14.649 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T17:22:14.712 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T17:22:14.722 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T17:22:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T17:22:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T17:22:14.813 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T17:22:14.815 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T17:22:14.861 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T17:22:14.897 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T17:22:14.918 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T17:22:14.940 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T17:22:14.956 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T17:22:14.982 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T17:22:15.018 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T17:22:15.048 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T17:22:15.064 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T17:22:15.073 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T17:22:15.119 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T17:22:15.155 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T17:22:15.172 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T17:22:15.183 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T17:22:15.202 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T17:22:15.215 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T17:22:15.216 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T17:22:15.243 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T17:22:15.248 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T17:22:15.337 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T17:22:15.364 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T17:22:15.377 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T17:22:15.417 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T17:22:15.442 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T17:22:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T17:22:15.467 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T17:22:15.514 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T17:22:15.528 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T17:22:15.543 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T17:22:15.549 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T17:22:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T17:22:15.614 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T17:22:15.644 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T17:22:15.657 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T17:22:15.669 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T17:22:15.675 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-25T17:22:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T17:22:15.709 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T17:22:15.711 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T17:22:15.736 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T17:22:15.749 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T17:22:15.770 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T17:22:15.786 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T17:22:15.826 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T17:22:15.854 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T17:22:15.857 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T17:22:15.888 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T17:22:15.907 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T17:22:15.922 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T17:22:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T17:22:15.956 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T17:22:15.974 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T17:22:15.997 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T17:22:16.032 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T17:22:16.050 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T17:22:16.086 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-25T17:22:16.119 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T17:22:16.137 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-25T17:22:16.170 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T17:22:16.174 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-25T17:22:16.190 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T17:22:16.202 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T17:22:16.223 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T17:22:16.248 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T17:22:16.252 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T17:22:16.286 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T17:22:16.306 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T17:22:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-25T17:22:16.347 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T17:22:16.381 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T17:22:16.395 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T17:22:16.514 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T17:22:16.575 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T17:22:17.111 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T17:22:17.120 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T17:22:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T17:22:17.197 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T17:22:17.201 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T17:22:17.226 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T17:22:17.269 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T17:22:17.301 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T17:22:17.315 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-25T17:22:17.360 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T17:22:17.365 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T17:22:17.446 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T17:22:17.487 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T17:22:17.535 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T17:22:17.610 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T17:22:17.655 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T17:22:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T17:22:17.684 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T17:22:17.737 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T17:22:17.762 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T17:22:17.769 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T17:22:17.806 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T17:22:17.861 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T17:22:17.906 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T17:22:17.949 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T17:22:17.991 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T17:22:18.127 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T17:22:18.180 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T17:22:18.211 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T17:22:18.252 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T17:22:18.346 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T17:22:18.395 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T17:22:18.419 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T17:22:18.443 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T17:22:18.485 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T17:22:18.510 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T17:22:18.541 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T17:22:18.584 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T17:22:18.588 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T17:22:18.629 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T17:22:18.713 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T17:22:18.743 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T17:22:18.771 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T17:22:18.806 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T17:22:18.845 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T17:22:18.885 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T17:22:18.923 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T17:22:18.958 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T17:22:18.992 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T17:22:19.033 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T17:22:19.068 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T17:22:19.164 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T17:22:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T17:22:19.207 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T17:22:19.224 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T17:22:19.247 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T17:22:19.302 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T17:22:19.309 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T17:22:19.309 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T17:22:19.309 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T17:22:19.309 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:19.334 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T17:22:19.400 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T17:22:19.448 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T17:22:19.773 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T17:22:19.848 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T17:22:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T17:22:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T17:22:20.172 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T17:22:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T17:22:20.221 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T17:22:20.263 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T17:22:20.272 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T17:22:20.346 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T17:22:20.368 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-25T17:22:20.370 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T17:22:20.404 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T17:22:20.438 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T17:22:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T17:22:20.557 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T17:22:20.737 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T17:22:20.764 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T17:22:20.851 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T17:22:20.885 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T17:22:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T17:22:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T17:22:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-25T17:22:21.511 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T17:22:21.626 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T17:22:22.857 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T17:22:22.950 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T17:22:22.986 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T17:22:23.028 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T17:22:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T17:22:23.358 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T17:22:23.373 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T17:22:23.400 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T17:22:23.491 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T17:22:23.538 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-25T17:22:23.574 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T17:22:23.918 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T17:22:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T17:22:24.059 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T17:22:24.059 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T17:22:24.059 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T17:22:24.061 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:24.066 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T17:22:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T17:22:24.094 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T17:22:24.166 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T17:22:24.188 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-25T17:22:26.421 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T17:22:26.469 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T17:22:27.018 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T17:22:27.056 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T17:22:27.131 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T17:22:27.132 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T17:22:27.132 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T17:22:27.132 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:27.165 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T17:22:30.924 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T17:22:30.924 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-04-25T17:22:30.924 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-04-25T17:22:30.924 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-04-25T17:22:30.924 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-04-25T17:22:30.924 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-04-25T17:22:30.924 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-04-25T17:22:30.924 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-04-25T17:22:30.924 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:31.095 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T17:22:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T17:22:31.114 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:31.114 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T17:22:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:32.615 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T17:22:32.640 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T17:22:32.640 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:32.640 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T17:22:32.640 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T17:22:32.640 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T17:22:32.640 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:32.874 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T17:22:32.896 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T17:22:32.896 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:32.897 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T17:22:32.897 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T17:22:32.897 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T17:22:32.897 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:33.350 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T17:22:33.371 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T17:22:33.372 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:33.372 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T17:22:33.372 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T17:22:33.372 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T17:22:33.372 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:33.480 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T17:22:33.527 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T17:22:33.582 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T17:22:33.767 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T17:22:33.825 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T17:22:33.894 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T17:22:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T17:22:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T17:22:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T17:22:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T17:22:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T17:22:34.654 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T17:22:34.690 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T17:22:34.724 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-25T17:22:34.784 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-25T17:22:34.815 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T17:22:35.580 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T17:22:35.581 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-04-25T17:22:35.581 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-04-25T17:22:35.581 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-04-25T17:22:35.581 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-04-25T17:22:35.581 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-04-25T17:22:35.581 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-04-25T17:22:35.581 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-04-25T17:22:35.581 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:35.750 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T17:22:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T17:22:35.765 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:35.765 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T17:22:35.765 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T17:22:35.891 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T17:22:36.856 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T17:22:36.884 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T17:22:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T17:22:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-25T17:22:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T17:22:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T17:22:37.208 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:37.209 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T17:22:37.209 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T17:22:37.209 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T17:22:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T17:22:37.324 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T17:22:37.324 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T17:22:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T17:22:37.325 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:37.325 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T17:22:37.325 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T17:22:37.325 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T17:22:37.325 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:37.361 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-25T17:22:37.409 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-25T17:22:37.513 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T17:22:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T17:22:37.536 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:37.536 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T17:22:37.536 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T17:22:37.536 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T17:22:37.536 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:37.981 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T17:22:38.004 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T17:22:38.004 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:38.004 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T17:22:38.004 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T17:22:38.004 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T17:22:38.004 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:38.102 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T17:22:38.151 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T17:22:38.206 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T17:22:38.331 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T17:22:38.363 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T17:22:38.386 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T17:22:38.386 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:38.386 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T17:22:38.386 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T17:22:38.386 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T17:22:38.386 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:38.392 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T17:22:38.431 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T17:22:38.439 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T17:22:38.452 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T17:22:38.452 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:38.452 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T17:22:38.453 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:38.498 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T17:22:38.508 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T17:22:38.520 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T17:22:38.520 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:38.520 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T17:22:38.520 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T17:22:38.520 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T17:22:38.520 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:38.598 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T17:22:38.739 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T17:22:38.782 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T17:22:38.814 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T17:22:38.824 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T17:22:38.841 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T17:22:38.846 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T17:22:38.846 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:38.846 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T17:22:38.846 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T17:22:38.847 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T17:22:38.847 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:38.849 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T17:22:38.850 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-04-25T17:22:38.850 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-04-25T17:22:38.850 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-04-25T17:22:38.850 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-04-25T17:22:38.850 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-04-25T17:22:38.850 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-04-25T17:22:38.850 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-04-25T17:22:38.850 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:39.033 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T17:22:39.047 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T17:22:39.048 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:39.048 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T17:22:39.048 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:39.244 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T17:22:39.273 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T17:22:39.306 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-25T17:22:39.367 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-25T17:22:39.397 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T17:22:40.432 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T17:22:40.465 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T17:22:40.544 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T17:22:40.569 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T17:22:40.569 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:40.569 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T17:22:40.569 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T17:22:40.569 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T17:22:40.569 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:40.795 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T17:22:40.817 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T17:22:40.817 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:40.817 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T17:22:40.817 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T17:22:40.817 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T17:22:40.817 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:41.271 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T17:22:41.292 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T17:22:41.293 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:41.293 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T17:22:41.293 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T17:22:41.293 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T17:22:41.293 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:41.393 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T17:22:41.425 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T17:22:41.441 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T17:22:41.457 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T17:22:41.496 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T17:22:41.559 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T17:22:41.692 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T17:22:41.720 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-25T17:22:41.738 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T17:22:41.742 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T17:22:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T17:22:41.766 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:41.766 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T17:22:41.766 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T17:22:41.766 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T17:22:41.766 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:41.817 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T17:22:41.835 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T17:22:41.846 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T17:22:41.906 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T17:22:41.936 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-25T17:22:41.994 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-25T17:22:42.046 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T17:22:42.089 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T17:22:42.122 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T17:22:42.149 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T17:22:42.512 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T17:22:42.548 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T17:22:42.589 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-25T17:22:42.659 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-25T17:22:42.697 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T17:22:42.917 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T17:22:42.955 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T17:22:42.978 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T17:22:42.979 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:42.979 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T17:22:42.979 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T17:22:42.979 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T17:22:42.979 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:43.022 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T17:22:43.044 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T17:22:43.044 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:43.044 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T17:22:43.044 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T17:22:43.112 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T17:22:43.112 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:43.112 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T17:22:43.112 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T17:22:43.112 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T17:22:43.112 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:43.415 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T17:22:43.418 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-25T17:22:43.438 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T17:22:43.438 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:43.438 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T17:22:43.438 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T17:22:43.438 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T17:22:43.438 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:43.460 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-25T17:22:43.501 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-25T17:22:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T17:22:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T17:22:43.742 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-25T17:22:43.745 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T17:22:43.774 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T17:22:43.797 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T17:22:43.819 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-25T17:22:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T17:22:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T17:22:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T17:22:44.756 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T17:22:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T17:22:44.891 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T17:22:45.066 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-25T17:22:45.093 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T17:22:45.116 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T17:22:45.116 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:45.116 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T17:22:45.116 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T17:22:45.116 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T17:22:45.116 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:45.161 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T17:22:45.171 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T17:22:45.271 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-25T17:22:45.325 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-25T17:22:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-25T17:22:45.438 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T17:22:45.478 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T17:22:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T17:22:45.592 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T17:22:45.592 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T17:22:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T17:22:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T17:22:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T17:22:45.801 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T17:22:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T17:22:46.274 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T17:22:46.307 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T17:22:46.330 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T17:22:46.331 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:46.331 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T17:22:46.331 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T17:22:46.331 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T17:22:46.331 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:46.374 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T17:22:46.396 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T17:22:46.396 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:46.396 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T17:22:46.396 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:46.442 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T17:22:46.464 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T17:22:46.464 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:46.464 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T17:22:46.464 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T17:22:46.464 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T17:22:46.464 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:46.768 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T17:22:46.790 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T17:22:46.790 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T17:22:46.790 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T17:22:46.790 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T17:22:46.790 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T17:22:46.790 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:22:48.002 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-25T17:22:48.043 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-25T17:22:48.084 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-25T17:22:48.116 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T17:22:48.144 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T17:22:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-25T17:22:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T17:22:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-25T17:22:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T17:22:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T17:22:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T17:22:49.911 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-25T17:22:50.000 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T17:22:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T17:22:50.125 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T17:22:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T17:22:50.163 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T17:22:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T17:22:50.280 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T17:22:50.280 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T17:22:50.391 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T17:22:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T17:22:51.496 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-25T17:22:51.537 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-25T17:22:51.578 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-25T17:22:51.618 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T17:22:51.646 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T17:22:51.824 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T17:22:51.824 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-25T17:22:51.824 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-25T17:22:51.824 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-25T17:22:51.824 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-25T17:22:51.825 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-25T17:22:51.826 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T17:22:51.827 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T17:22:51.828 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T17:22:51.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T17:22:51.830 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T17:22:51.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T17:22:51.832 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T17:22:51.833 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T17:22:51.834 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T17:22:51.835 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T17:22:51.836 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-25T17:22:51.883 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T17:22:51.913 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-25T17:22:52.525 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T17:22:52.561 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T17:22:52.580 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T17:22:53.407 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-25T17:22:53.499 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T17:22:53.531 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T17:22:53.614 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T17:22:53.653 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T17:22:53.653 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T17:22:53.720 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T17:22:53.745 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T17:22:53.745 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T17:22:53.822 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T17:22:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T17:22:53.916 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T17:22:53.916 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:53.916 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-04-25T17:22:53.916 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.916 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.917 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:53.918 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T17:22:53.919 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T17:22:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T17:22:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T17:22:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T17:22:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T17:22:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T17:22:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T17:22:53.921 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T17:22:53.922 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T17:22:53.922 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T17:22:53.922 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T17:22:53.922 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T17:22:53.922 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T17:22:53.923 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T17:22:53.924 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T17:22:53.925 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T17:22:53.926 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T17:22:53.927 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T17:22:53.928 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T17:22:53.929 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T17:22:53.929 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T17:22:53.929 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-04-25T17:22:54.300 DEBUG:teuthology.parallel:result is None 2024-04-25T17:22:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T17:22:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-25T17:22:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-25T17:22:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-25T17:22:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-25T17:22:56.509 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-25T17:22:56.510 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-25T17:22:56.511 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-25T17:22:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-25T17:22:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-25T17:22:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-25T17:22:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-25T17:22:56.523 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T17:22:56.524 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T17:22:56.525 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T17:22:56.526 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T17:22:56.527 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T17:22:56.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T17:22:56.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T17:22:56.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T17:22:56.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T17:22:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T17:22:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T17:22:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T17:22:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-25T17:22:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T17:22:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-25T17:22:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T17:22:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T17:22:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T17:22:56.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T17:22:58.602 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T17:22:58.603 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:58.604 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T17:22:58.605 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.606 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T17:22:58.607 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T17:22:58.608 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T17:22:58.609 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T17:22:58.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T17:22:58.611 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T17:22:58.612 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T17:22:58.613 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:22:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T17:22:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T17:22:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T17:22:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T17:22:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T17:22:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T17:22:58.614 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T17:22:58.615 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:22:58.615 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-25T17:22:58.981 DEBUG:teuthology.parallel:result is None 2024-04-25T17:23:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T17:23:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-25T17:23:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-25T17:23:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-25T17:23:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-25T17:23:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-25T17:23:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-25T17:23:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-25T17:23:00.121 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-25T17:23:00.122 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-25T17:23:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-25T17:23:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-25T17:23:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-25T17:23:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-25T17:23:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-25T17:23:00.123 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-25T17:23:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-25T17:23:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-25T17:23:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-25T17:23:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-25T17:23:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-25T17:23:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-25T17:23:00.135 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T17:23:00.136 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T17:23:00.137 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T17:23:00.138 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T17:23:00.139 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T17:23:00.140 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T17:23:00.141 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T17:23:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T17:23:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T17:23:00.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T17:23:02.186 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T17:23:02.186 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:23:02.186 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-04-25T17:23:02.186 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.186 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.186 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T17:23:02.186 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T17:23:02.186 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-25T17:23:02.186 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T17:23:02.186 INFO:teuthology.orchestra.run.smithi203.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:23:02.187 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T17:23:02.188 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T17:23:02.189 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T17:23:02.190 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T17:23:02.191 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T17:23:02.192 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T17:23:02.193 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T17:23:02.194 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.195 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T17:23:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T17:23:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T17:23:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T17:23:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T17:23:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T17:23:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T17:23:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T17:23:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T17:23:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T17:23:02.198 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:23:02.198 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-25T17:23:02.630 DEBUG:teuthology.parallel:result is None 2024-04-25T17:23:02.631 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:23:02.850 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T17:23:02.875 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-25T17:23:02.876 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-25T17:23:02.876 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-25T17:23:02.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:23:03.016 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T17:23:03.041 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-25T17:23:03.041 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-25T17:23:03.041 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-25T17:23:03.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:23:03.260 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T17:23:03.280 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-25T17:23:03.281 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-25T17:23:03.281 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-25T17:23:03.282 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T17:23:03.283 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:23:03.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T17:23:03.314 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:23:03.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T17:23:03.345 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:23:03.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T17:23:03.370 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T17:23:03.371 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:23:03.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T17:23:03.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T17:23:03.465 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:23:03.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T17:23:03.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T17:23:03.560 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:23:03.561 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T17:23:03.586 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T17:23:03.649 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T17:23:03.649 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:23:03.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T17:23:03.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T17:23:03.742 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:23:03.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T17:23:03.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T17:23:03.836 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:23:03.837 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T17:23:03.859 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T17:23:03.923 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T17:23:03.923 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:23:03.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T17:23:03.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T17:23:04.015 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:23:04.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T17:23:04.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T17:23:04.109 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:23:04.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T17:23:04.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T17:23:04.196 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T17:23:04.306 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'subvols': {'create': 2}} 2024-04-25T17:23:04.307 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:23:04.307 INFO:tasks.cephadm:Cluster fsid is 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:23:04.307 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T17:23:04.308 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.98', 'mon.b': '172.21.15.173', 'mon.c': '172.21.15.203'} 2024-04-25T17:23:04.308 INFO:tasks.cephadm:First mon is mon.a on smithi098 2024-04-25T17:23:04.308 INFO:tasks.cephadm:First mgr is x 2024-04-25T17:23:04.308 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T17:23:04.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-04-25T17:23:04.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-04-25T17:23:04.365 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-04-25T17:23:04.388 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T17:23:04.388 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:23:04.615 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78795/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-gd69f13d8', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.gd69f13d8'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 12:00:06.590653', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-25T17:23:04.845 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 d69f13d827952bb07bb9b7973bca14f97d764d6f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:23:04.847 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm 2024-04-25T17:23:04.847 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T17:23:05.264 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 25 17:23 /home/ubuntu/cephtest/cephadm 2024-04-25T17:23:05.265 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T17:23:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 25 17:23 /home/ubuntu/cephtest/cephadm 2024-04-25T17:23:05.699 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T17:23:06.315 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 25 17:23 /home/ubuntu/cephtest/cephadm 2024-04-25T17:23:06.316 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T17:23:06.335 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T17:23:06.354 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T17:23:06.389 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f on all hosts... 2024-04-25T17:23:06.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-25T17:23:06.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-25T17:23:06.396 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-25T17:23:08.034 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-25T17:23:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-25T17:23:08.095 INFO:teuthology.orchestra.run.smithi203.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-25T17:23:39.598 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:23:39.599 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-25T17:23:39.599 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-25T17:23:39.599 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-04-25T17:23:39.599 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-25T17:23:39.599 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:23:39.599 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:24:00.771 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T17:24:00.772 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-25T17:24:00.772 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-25T17:24:00.772 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-04-25T17:24:00.772 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-25T17:24:00.772 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-04-25T17:24:00.772 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T17:24:05.729 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-25T17:24:05.730 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-25T17:24:05.730 INFO:teuthology.orchestra.run.smithi203.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-25T17:24:05.730 INFO:teuthology.orchestra.run.smithi203.stdout: "repo_digests": [ 2024-04-25T17:24:05.730 INFO:teuthology.orchestra.run.smithi203.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-25T17:24:05.730 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-04-25T17:24:05.730 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-25T17:24:05.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-04-25T17:24:05.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-04-25T17:24:05.822 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-04-25T17:24:05.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-04-25T17:24:05.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-04-25T17:24:05.915 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-04-25T17:24:05.942 INFO:tasks.cephadm:Writing seed config... 2024-04-25T17:24:05.944 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T17:24:05.944 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T17:24:05.944 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T17:24:05.944 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T17:24:05.944 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T17:24:05.944 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-25T17:24:05.944 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T17:24:05.944 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T17:24:05.944 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T17:24:05.944 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T17:24:05.945 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-25T17:24:05.946 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T17:24:05.947 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T17:24:05.947 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T17:24:05.947 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T17:24:05.947 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T17:24:05.947 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T17:24:05.947 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T17:24:05.947 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-25T17:24:05.948 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:24:05.948 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T17:24:05.968 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 79754014-0328-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T17:24:05.968 DEBUG:teuthology.orchestra.run.smithi098:mon.a> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T17:24:06.012 DEBUG:teuthology.orchestra.run.smithi098:mgr.x> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T17:24:06.055 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T17:24:06.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f -v bootstrap --fsid 79754014-0328-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.98 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T17:24:06.246 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-25T17:24:06.247 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f', '-v', 'bootstrap', '--fsid', '79754014-0328-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.98', '--skip-admin-label'] 2024-04-25T17:24:06.247 INFO:teuthology.orchestra.run.smithi098.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T17:24:06.247 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying podman|docker is present... 2024-04-25T17:24:06.272 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T17:24:06.272 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying lvm2 is present... 2024-04-25T17:24:06.272 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying time synchronization is in place... 2024-04-25T17:24:06.280 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T17:24:06.280 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T17:24:06.286 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T17:24:06.286 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T17:24:06.292 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-04-25T17:24:06.298 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-04-25T17:24:06.298 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-04-25T17:24:06.298 INFO:teuthology.orchestra.run.smithi098.stdout:Repeating the final host check... 2024-04-25T17:24:06.321 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T17:24:06.321 INFO:teuthology.orchestra.run.smithi098.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T17:24:06.321 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl is present 2024-04-25T17:24:06.322 INFO:teuthology.orchestra.run.smithi098.stdout:lvcreate is present 2024-04-25T17:24:06.327 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T17:24:06.328 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T17:24:06.333 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T17:24:06.333 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T17:24:06.339 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-04-25T17:24:06.346 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-04-25T17:24:06.346 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-04-25T17:24:06.346 INFO:teuthology.orchestra.run.smithi098.stdout:Host looks OK 2024-04-25T17:24:06.346 INFO:teuthology.orchestra.run.smithi098.stdout:Cluster fsid: 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:06.346 INFO:teuthology.orchestra.run.smithi098.stdout:Acquiring lock 140485924666528 on /run/cephadm/79754014-0328-11ef-bc93-c7b262605968.lock 2024-04-25T17:24:06.346 INFO:teuthology.orchestra.run.smithi098.stdout:Lock 140485924666528 acquired on /run/cephadm/79754014-0328-11ef-bc93-c7b262605968.lock 2024-04-25T17:24:06.347 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 3300 ... 2024-04-25T17:24:06.347 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 6789 ... 2024-04-25T17:24:06.347 INFO:teuthology.orchestra.run.smithi098.stdout:Base mon IP(s) is [172.21.15.98:3300, 172.21.15.98:6789], mon addrv is [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-04-25T17:24:06.350 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.98 metric 100 2024-04-25T17:24:06.350 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.98 metric 100 2024-04-25T17:24:06.353 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T17:24:06.353 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-25T17:24:06.353 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-25T17:24:06.355 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T17:24:06.355 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T17:24:06.355 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T17:24:06.355 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-25T17:24:06.355 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 fe80::c454:7d32:cfc2:7ded/64 scope link noprefixroute 2024-04-25T17:24:06.355 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T17:24:06.356 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-04-25T17:24:06.356 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-04-25T17:24:06.356 INFO:teuthology.orchestra.run.smithi098.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T17:24:06.356 INFO:teuthology.orchestra.run.smithi098.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T17:24:06.357 INFO:teuthology.orchestra.run.smithi098.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-25T17:24:06.906 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-25T17:24:06.906 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-25T17:24:06.907 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T17:24:06.907 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:7f03586bcf9c6a2b4d690c118c4aa5b771f9c6f252d75a20fb283dd07093a738 2024-04-25T17:24:06.907 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-25T17:24:06.907 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying config sha256:da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-25T17:24:06.907 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T17:24:07.830 INFO:teuthology.orchestra.run.smithi098.stdout:ceph: stdout ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T17:24:07.830 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph version: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T17:24:07.830 INFO:teuthology.orchestra.run.smithi098.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T17:24:08.812 INFO:teuthology.orchestra.run.smithi098.stdout:stat: stdout 167 167 2024-04-25T17:24:08.812 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial keys... 2024-04-25T17:24:09.735 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQC5kSpmeZekCRAAPi+rQt64nRJ7Op60/1XW8Q== 2024-04-25T17:24:10.750 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQC6kSpm46syAxAAIGuqcWhWeQgEqN757T7fBw== 2024-04-25T17:24:11.660 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQC7kSpmCuMWBBAAK3C3LpBtmu2kwuTZpQ0ynw== 2024-04-25T17:24:11.661 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial monmap... 2024-04-25T17:24:12.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T17:24:12.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T17:24:12.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:12.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T17:24:12.544 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool for a [v2:172.21.15.98:3300,v1:172.21.15.98:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T17:24:12.544 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-04-25T17:24:12.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: set fsid to 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:12.544 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T17:24:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:24:12.545 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mon... 2024-04-25T17:24:13.542 INFO:teuthology.orchestra.run.smithi098.stdout:create mon.a on 2024-04-25T17:24:13.739 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T17:24:13.876 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T17:24:14.048 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-79754014-0328-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-79754014-0328-11ef-bc93-c7b262605968.target. 2024-04-25T17:24:14.048 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-79754014-0328-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-79754014-0328-11ef-bc93-c7b262605968.target. 2024-04-25T17:24:14.294 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-79754014-0328-11ef-bc93-c7b262605968@mon.a 2024-04-25T17:24:14.295 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-79754014-0328-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-79754014-0328-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T17:24:14.460 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-79754014-0328-11ef-bc93-c7b262605968.target.wants/ceph-79754014-0328-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-79754014-0328-11ef-bc93-c7b262605968@.service. 2024-04-25T17:24:15.000 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T17:24:15.000 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-25T17:24:15.007 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T17:24:15.007 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T17:24:15.007 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-25T17:24:15.007 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T17:24:15.008 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon to start... 2024-04-25T17:24:15.008 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon... 2024-04-25T17:24:16.084 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:15 smithi098 ceph-mon[28815]: from='client.? 172.21.15.98:0/3538297883' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T17:24:16.087 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T17:24:16.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout id: 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:16.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T17:24:16.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:16.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout services: 2024-04-25T17:24:16.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.57775s) 2024-04-25T17:24:16.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T17:24:16.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T17:24:16.088 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:16.089 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout data: 2024-04-25T17:24:16.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T17:24:16.094 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.452+0000 7f4a9262a700 1 Processor -- start 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.452+0000 7f4a9262a700 1 -- start start 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.452+0000 7f4a9262a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c107660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.452+0000 7f4a9262a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c107c30 con 0x7f4a8c107240 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.453+0000 7f4a8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.453+0000 7f4a8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53138/0 (socket says 172.21.15.98:53138) 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.453+0000 7f4a8bfff700 1 -- 172.21.15.98:0/1883583154 learned_addr learned my addr 172.21.15.98:0/1883583154 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.453+0000 7f4a8bfff700 1 -- 172.21.15.98:0/1883583154 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c108440 con 0x7f4a8c107240 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.453+0000 7f4a8bfff700 1 --2- 172.21.15.98:0/1883583154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c107660 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c0098a0 tx=0x7f4a7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42262aa8a7180ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:16.095 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.453+0000 7f4a8affd700 1 -- 172.21.15.98:0/1883583154 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a7c004160 con 0x7f4a8c107240 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.453+0000 7f4a8affd700 1 -- 172.21.15.98:0/1883583154 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a7c004300 con 0x7f4a8c107240 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.453+0000 7f4a8affd700 1 -- 172.21.15.98:0/1883583154 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a7c0036a0 con 0x7f4a8c107240 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.454+0000 7f4a9262a700 1 -- 172.21.15.98:0/1883583154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 msgr2=0x7f4a8c107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.454+0000 7f4a9262a700 1 --2- 172.21.15.98:0/1883583154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c107660 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c0098a0 tx=0x7f4a7c009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.454+0000 7f4a9262a700 1 -- 172.21.15.98:0/1883583154 shutdown_connections 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.454+0000 7f4a9262a700 1 --2- 172.21.15.98:0/1883583154 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c107660 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.454+0000 7f4a9262a700 1 -- 172.21.15.98:0/1883583154 >> 172.21.15.98:0/1883583154 conn(0x7f4a8c102820 msgr2=0x7f4a8c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.454+0000 7f4a9262a700 1 -- 172.21.15.98:0/1883583154 shutdown_connections 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.454+0000 7f4a9262a700 1 -- 172.21.15.98:0/1883583154 wait complete. 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.455+0000 7f4a9262a700 1 Processor -- start 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.455+0000 7f4a9262a700 1 -- start start 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.455+0000 7f4a9262a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.455+0000 7f4a9262a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c108970 con 0x7f4a8c107240 2024-04-25T17:24:16.096 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.455+0000 7f4a8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.455+0000 7f4a8bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c19a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53142/0 (socket says 172.21.15.98:53142) 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.455+0000 7f4a8bfff700 1 -- 172.21.15.98:0/3538297883 learned_addr learned my addr 172.21.15.98:0/3538297883 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.455+0000 7f4a8bfff700 1 -- 172.21.15.98:0/3538297883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c19ae30 con 0x7f4a8c107240 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.455+0000 7f4a8bfff700 1 --2- 172.21.15.98:0/3538297883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c19a8f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c000c00 tx=0x7f4a7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.456+0000 7f4a897fa700 1 -- 172.21.15.98:0/3538297883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a7c003db0 con 0x7f4a8c107240 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.456+0000 7f4a897fa700 1 -- 172.21.15.98:0/3538297883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a7c011630 con 0x7f4a8c107240 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.456+0000 7f4a9262a700 1 -- 172.21.15.98:0/3538297883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8c19b0c0 con 0x7f4a8c107240 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.456+0000 7f4a897fa700 1 -- 172.21.15.98:0/3538297883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a7c011950 con 0x7f4a8c107240 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.456+0000 7f4a9262a700 1 -- 172.21.15.98:0/3538297883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8c19ddb0 con 0x7f4a8c107240 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.456+0000 7f4a897fa700 1 -- 172.21.15.98:0/3538297883 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4a7c011af0 con 0x7f4a8c107240 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.456+0000 7f4a897fa700 1 -- 172.21.15.98:0/3538297883 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4a7c01c450 con 0x7f4a8c107240 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.457+0000 7f4a9262a700 1 -- 172.21.15.98:0/3538297883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8c0640a0 con 0x7f4a8c107240 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.459+0000 7f4a897fa700 1 -- 172.21.15.98:0/3538297883 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4a7c004260 con 0x7f4a8c107240 2024-04-25T17:24:16.097 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.531+0000 7f4a9262a700 1 -- 172.21.15.98:0/3538297883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f4a8c19e290 con 0x7f4a8c107240 2024-04-25T17:24:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.531+0000 7f4a897fa700 1 -- 172.21.15.98:0/3538297883 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f4a7c01b450 con 0x7f4a8c107240 2024-04-25T17:24:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.532+0000 7f4a9262a700 1 -- 172.21.15.98:0/3538297883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 msgr2=0x7f4a8c19a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.532+0000 7f4a9262a700 1 --2- 172.21.15.98:0/3538297883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c19a8f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c000c00 tx=0x7f4a7c004750 comp rx=0 tx=0).stop 2024-04-25T17:24:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.532+0000 7f4a9262a700 1 -- 172.21.15.98:0/3538297883 shutdown_connections 2024-04-25T17:24:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.532+0000 7f4a9262a700 1 --2- 172.21.15.98:0/3538297883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c107240 0x7f4a8c19a8f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.532+0000 7f4a9262a700 1 -- 172.21.15.98:0/3538297883 >> 172.21.15.98:0/3538297883 conn(0x7f4a8c102820 msgr2=0x7f4a8c18ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.532+0000 7f4a9262a700 1 -- 172.21.15.98:0/3538297883 shutdown_connections 2024-04-25T17:24:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:15.532+0000 7f4a9262a700 1 -- 172.21.15.98:0/3538297883 wait complete. 2024-04-25T17:24:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:mon is available 2024-04-25T17:24:16.098 INFO:teuthology.orchestra.run.smithi098.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T17:24:17.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:16 smithi098 ceph-mon[28815]: from='client.? 172.21.15.98:0/4237793003' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T17:24:17.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:16 smithi098 ceph-mon[28815]: from='client.? 172.21.15.98:0/4237793003' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T17:24:17.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:17.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.553+0000 7f44085fa700 1 Processor -- start 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.553+0000 7f44085fa700 1 -- start start 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.553+0000 7f44085fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400106e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.553+0000 7f44085fa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44001073e0 con 0x7f44001069f0 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.553+0000 7f4406396700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.553+0000 7f4406396700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400106e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53158/0 (socket says 172.21.15.98:53158) 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.553+0000 7f4406396700 1 -- 172.21.15.98:0/2451763704 learned_addr learned my addr 172.21.15.98:0/2451763704 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.554+0000 7f4406396700 1 -- 172.21.15.98:0/2451763704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4400107bf0 con 0x7f44001069f0 2024-04-25T17:24:17.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.554+0000 7f4406396700 1 --2- 172.21.15.98:0/2451763704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400106e10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0098a0 tx=0x7f43fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c968f7baef1fac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:17.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.554+0000 7f4405394700 1 -- 172.21.15.98:0/2451763704 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43fc004160 con 0x7f44001069f0 2024-04-25T17:24:17.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.554+0000 7f4405394700 1 -- 172.21.15.98:0/2451763704 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43fc004300 con 0x7f44001069f0 2024-04-25T17:24:17.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.554+0000 7f4405394700 1 -- 172.21.15.98:0/2451763704 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43fc0036a0 con 0x7f44001069f0 2024-04-25T17:24:17.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.554+0000 7f44085fa700 1 -- 172.21.15.98:0/2451763704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 msgr2=0x7f4400106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:17.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.554+0000 7f44085fa700 1 --2- 172.21.15.98:0/2451763704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400106e10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0098a0 tx=0x7f43fc009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:17.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f44085fa700 1 -- 172.21.15.98:0/2451763704 shutdown_connections 2024-04-25T17:24:17.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f44085fa700 1 --2- 172.21.15.98:0/2451763704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400106e10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:17.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f44085fa700 1 -- 172.21.15.98:0/2451763704 >> 172.21.15.98:0/2451763704 conn(0x7f4400102010 msgr2=0x7f4400104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:17.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f44085fa700 1 -- 172.21.15.98:0/2451763704 shutdown_connections 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f44085fa700 1 -- 172.21.15.98:0/2451763704 wait complete. 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f44085fa700 1 Processor -- start 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f44085fa700 1 -- start start 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f44085fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f44085fa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4400075440 con 0x7f44001069f0 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f4406396700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f4406396700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53166/0 (socket says 172.21.15.98:53166) 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.555+0000 7f4406396700 1 -- 172.21.15.98:0/4237793003 learned_addr learned my addr 172.21.15.98:0/4237793003 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.556+0000 7f4406396700 1 -- 172.21.15.98:0/4237793003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4400199cd0 con 0x7f44001069f0 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.556+0000 7f4406396700 1 --2- 172.21.15.98:0/4237793003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400199790 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f43fc000c00 tx=0x7f43fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.556+0000 7f43f77fe700 1 -- 172.21.15.98:0/4237793003 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43fc003db0 con 0x7f44001069f0 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.556+0000 7f43f77fe700 1 -- 172.21.15.98:0/4237793003 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43fc011630 con 0x7f44001069f0 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.556+0000 7f43f77fe700 1 -- 172.21.15.98:0/4237793003 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43fc011950 con 0x7f44001069f0 2024-04-25T17:24:17.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.556+0000 7f44085fa700 1 -- 172.21.15.98:0/4237793003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4400199f60 con 0x7f44001069f0 2024-04-25T17:24:17.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.556+0000 7f44085fa700 1 -- 172.21.15.98:0/4237793003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440019cc50 con 0x7f44001069f0 2024-04-25T17:24:17.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.556+0000 7f43f77fe700 1 -- 172.21.15.98:0/4237793003 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f43fc011af0 con 0x7f44001069f0 2024-04-25T17:24:17.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.556+0000 7f43f77fe700 1 -- 172.21.15.98:0/4237793003 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f43fc01c450 con 0x7f44001069f0 2024-04-25T17:24:17.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.557+0000 7f44085fa700 1 -- 172.21.15.98:0/4237793003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44000640a0 con 0x7f44001069f0 2024-04-25T17:24:17.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.559+0000 7f43f77fe700 1 -- 172.21.15.98:0/4237793003 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f43fc032020 con 0x7f44001069f0 2024-04-25T17:24:17.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.626+0000 7f44085fa700 1 -- 172.21.15.98:0/4237793003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f440019ce20 con 0x7f44001069f0 2024-04-25T17:24:17.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.629+0000 7f43f77fe700 1 -- 172.21.15.98:0/4237793003 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f43fc004260 con 0x7f44001069f0 2024-04-25T17:24:17.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.630+0000 7f43f77fe700 1 -- 172.21.15.98:0/4237793003 <== mon.0 v2:172.21.15.98:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f43fc01c6a0 con 0x7f44001069f0 2024-04-25T17:24:17.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.630+0000 7f44085fa700 1 -- 172.21.15.98:0/4237793003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 msgr2=0x7f4400199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:17.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.630+0000 7f44085fa700 1 --2- 172.21.15.98:0/4237793003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400199790 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f43fc000c00 tx=0x7f43fc004750 comp rx=0 tx=0).stop 2024-04-25T17:24:17.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.631+0000 7f44085fa700 1 -- 172.21.15.98:0/4237793003 shutdown_connections 2024-04-25T17:24:17.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.631+0000 7f44085fa700 1 --2- 172.21.15.98:0/4237793003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44001069f0 0x7f4400199790 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:17.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.631+0000 7f44085fa700 1 -- 172.21.15.98:0/4237793003 >> 172.21.15.98:0/4237793003 conn(0x7f4400102010 msgr2=0x7f440018fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:17.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.631+0000 7f44085fa700 1 -- 172.21.15.98:0/4237793003 shutdown_connections 2024-04-25T17:24:17.197 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:16.631+0000 7f44085fa700 1 -- 172.21.15.98:0/4237793003 wait complete. 2024-04-25T17:24:17.197 INFO:teuthology.orchestra.run.smithi098.stdout:Generating new minimal ceph.conf... 2024-04-25T17:24:18.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.585+0000 7f7999ad3700 1 Processor -- start 2024-04-25T17:24:18.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.585+0000 7f7999ad3700 1 -- start start 2024-04-25T17:24:18.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.585+0000 7f7999ad3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f79941087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:18.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.585+0000 7f7999ad3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7994108d90 con 0x7f79941083a0 2024-04-25T17:24:18.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.585+0000 7f79937fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f79941087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:18.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.585+0000 7f79937fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f79941087c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53174/0 (socket says 172.21.15.98:53174) 2024-04-25T17:24:18.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.585+0000 7f79937fe700 1 -- 172.21.15.98:0/1251624271 learned_addr learned my addr 172.21.15.98:0/1251624271 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:18.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.586+0000 7f79937fe700 1 -- 172.21.15.98:0/1251624271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7994109550 con 0x7f79941083a0 2024-04-25T17:24:18.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.586+0000 7f79937fe700 1 --2- 172.21.15.98:0/1251624271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f79941087c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f79840098a0 tx=0x7f7984009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8988d388c5d064d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:18.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.586+0000 7f7992ffd700 1 -- 172.21.15.98:0/1251624271 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7984004160 con 0x7f79941083a0 2024-04-25T17:24:18.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.586+0000 7f7992ffd700 1 -- 172.21.15.98:0/1251624271 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7984004300 con 0x7f79941083a0 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.586+0000 7f7992ffd700 1 -- 172.21.15.98:0/1251624271 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7984011af0 con 0x7f79941083a0 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.587+0000 7f7999ad3700 1 -- 172.21.15.98:0/1251624271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 msgr2=0x7f79941087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.587+0000 7f7999ad3700 1 --2- 172.21.15.98:0/1251624271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f79941087c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f79840098a0 tx=0x7f7984009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.587+0000 7f7999ad3700 1 -- 172.21.15.98:0/1251624271 shutdown_connections 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.587+0000 7f7999ad3700 1 --2- 172.21.15.98:0/1251624271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f79941087c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.587+0000 7f7999ad3700 1 -- 172.21.15.98:0/1251624271 >> 172.21.15.98:0/1251624271 conn(0x7f7994103940 msgr2=0x7f7994105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.587+0000 7f7999ad3700 1 -- 172.21.15.98:0/1251624271 shutdown_connections 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.587+0000 7f7999ad3700 1 -- 172.21.15.98:0/1251624271 wait complete. 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.587+0000 7f7999ad3700 1 Processor -- start 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.587+0000 7f7999ad3700 1 -- start start 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f7999ad3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f799407db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f7999ad3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f799410c900 con 0x7f79941083a0 2024-04-25T17:24:18.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f79937fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f799407db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f79937fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f799407db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53186/0 (socket says 172.21.15.98:53186) 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f79937fe700 1 -- 172.21.15.98:0/54475 learned_addr learned my addr 172.21.15.98:0/54475 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f79937fe700 1 -- 172.21.15.98:0/54475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799407e0a0 con 0x7f79941083a0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f79937fe700 1 --2- 172.21.15.98:0/54475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f799407db60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7984003f40 tx=0x7f7984011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f79917fa700 1 -- 172.21.15.98:0/54475 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79840038b0 con 0x7f79941083a0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f7999ad3700 1 -- 172.21.15.98:0/54475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799407c290 con 0x7f79941083a0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f79917fa700 1 -- 172.21.15.98:0/54475 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7984003a50 con 0x7f79941083a0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f79917fa700 1 -- 172.21.15.98:0/54475 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f798401a5c0 con 0x7f79941083a0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f7999ad3700 1 -- 172.21.15.98:0/54475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f799407c7d0 con 0x7f79941083a0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f79917fa700 1 -- 172.21.15.98:0/54475 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f798401ac50 con 0x7f79941083a0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.588+0000 7f79917fa700 1 -- 172.21.15.98:0/54475 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7984019700 con 0x7f79941083a0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.589+0000 7f7999ad3700 1 -- 172.21.15.98:0/54475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79940640a0 con 0x7f79941083a0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.591+0000 7f79917fa700 1 -- 172.21.15.98:0/54475 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f79840199a0 con 0x7f79941083a0 2024-04-25T17:24:18.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.658+0000 7f7999ad3700 1 -- 172.21.15.98:0/54475 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f799407ca80 con 0x7f79941083a0 2024-04-25T17:24:18.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.659+0000 7f79917fa700 1 -- 172.21.15.98:0/54475 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f7984019b80 con 0x7f79941083a0 2024-04-25T17:24:18.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.660+0000 7f7999ad3700 1 -- 172.21.15.98:0/54475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 msgr2=0x7f799407db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:18.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.660+0000 7f7999ad3700 1 --2- 172.21.15.98:0/54475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f799407db60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7984003f40 tx=0x7f7984011890 comp rx=0 tx=0).stop 2024-04-25T17:24:18.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.660+0000 7f7999ad3700 1 -- 172.21.15.98:0/54475 shutdown_connections 2024-04-25T17:24:18.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.660+0000 7f7999ad3700 1 --2- 172.21.15.98:0/54475 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79941083a0 0x7f799407db60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:18.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.660+0000 7f7999ad3700 1 -- 172.21.15.98:0/54475 >> 172.21.15.98:0/54475 conn(0x7f7994103940 msgr2=0x7f79941043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:18.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.660+0000 7f7999ad3700 1 -- 172.21.15.98:0/54475 shutdown_connections 2024-04-25T17:24:18.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:17.660+0000 7f7999ad3700 1 -- 172.21.15.98:0/54475 wait complete. 2024-04-25T17:24:18.212 INFO:teuthology.orchestra.run.smithi098.stdout:Restarting the monitor... 2024-04-25T17:24:18.230 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:17 smithi098 ceph-mon[28815]: from='client.? 172.21.15.98:0/54475' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:24:18.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:18 smithi098 systemd[1]: Stopping Ceph mon.a for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:24:18.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a[28811]: 2024-04-25T17:24:18.470+0000 7f8b9f004700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:24:18.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a[28811]: 2024-04-25T17:24:18.470+0000 7f8b9f004700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T17:24:18.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:18 smithi098 podman[29061]: 2024-04-25 17:24:18.644808101 +0000 UTC m=+0.271190839 container died 65423692bc4d84da2cba98e42640f9d9225ab142e4561bcde462469355a8cbfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T17:24:19.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:18 smithi098 podman[29061]: 2024-04-25 17:24:18.978866126 +0000 UTC m=+0.605248915 container cleanup 65423692bc4d84da2cba98e42640f9d9225ab142e4561bcde462469355a8cbfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T17:24:19.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:18 smithi098 bash[29061]: ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a 2024-04-25T17:24:19.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 podman[29074]: 2024-04-25 17:24:19.129717836 +0000 UTC m=+0.480877458 container remove 65423692bc4d84da2cba98e42640f9d9225ab142e4561bcde462469355a8cbfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-25T17:24:19.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T17:24:19.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 systemd[1]: Stopped Ceph mon.a for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:24:19.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 systemd[1]: Starting Ceph mon.a for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:24:19.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 podman[29173]: 2024-04-25 17:24:19.479118594 +0000 UTC m=+0.017952709 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T17:24:19.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 podman[29173]: 2024-04-25 17:24:19.580574268 +0000 UTC m=+0.119408367 container create 2fd9d01e54efa100d598d8a466c31b356cfc67f9df8336a5e9522851b39dbe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:24:19.882 INFO:teuthology.orchestra.run.smithi098.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T17:24:19.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 podman[29173]: 2024-04-25 17:24:19.794645026 +0000 UTC m=+0.333479121 container init 2fd9d01e54efa100d598d8a466c31b356cfc67f9df8336a5e9522851b39dbe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:24:19.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 podman[29173]: 2024-04-25 17:24:19.7976096 +0000 UTC m=+0.336443696 container start 2fd9d01e54efa100d598d8a466c31b356cfc67f9df8336a5e9522851b39dbe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:24:19.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T17:24:19.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-25T17:24:19.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: pidfile_write: ignore empty --pid-file 2024-04-25T17:24:19.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: load: jerasure load: lrc 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: RocksDB version: 7.9.2 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Git sha 0 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: DB SUMMARY 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: DB Session ID: CZD757GIVYWNZFJ07SZ1 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: CURRENT file: CURRENT 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104514 ; 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.error_if_exists: 0 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.create_if_missing: 0 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.paranoid_checks: 1 2024-04-25T17:24:19.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T17:24:19.958 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T17:24:19.958 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T17:24:19.958 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.env: 0x55cd02269cc0 2024-04-25T17:24:19.958 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T17:24:19.958 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.info_log: 0x55cd04497be0 2024-04-25T17:24:19.958 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T17:24:19.958 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.statistics: (nil) 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.use_fsync: 0 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_log_file_size: 0 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.allow_fallocate: 1 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.use_direct_reads: 0 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T17:24:19.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.db_log_dir: 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.wal_dir: 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.write_buffer_manager: 0x55cd0442b0e0 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T17:24:19.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.unordered_write: 0 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.row_cache: None 2024-04-25T17:24:19.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.wal_filter: None 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.two_write_queues: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.wal_compression: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.atomic_flush: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.log_readahead_size: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T17:24:19.963 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_background_jobs: 2 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_background_compactions: -1 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_subcompactions: 1 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T17:24:19.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_open_files: -1 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_background_flushes: -1 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Compression algorithms supported: 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: kZSTD supported: 0 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: kXpressCompression supported: 0 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: kZlibCompression supported: 1 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: kSnappyCompression supported: 1 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: kLZ4Compression supported: 1 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: kBZip2Compression supported: 0 2024-04-25T17:24:19.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.merge_operator: 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_filter: None 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cd04497dc0) 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T17:24:19.966 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-04-25T17:24:19.967 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-04-25T17:24:19.967 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-04-25T17:24:19.967 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-04-25T17:24:19.967 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T17:24:19.967 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 4 2024-04-25T17:24:19.967 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-04-25T17:24:19.967 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x55cd0445d1f0 2024-04-25T17:24:19.967 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-04-25T17:24:19.967 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-04-25T17:24:19.968 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: bloomfilter 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 5 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_auto_readahead_size: 262144 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout: prepopulate_block_cache: 0 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout: initial_auto_readahead_size: 8192 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compression: NoCompression 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T17:24:19.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.num_levels: 7 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T17:24:19.970 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T17:24:19.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T17:24:19.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T17:24:19.973 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.inplace_update_support: 0 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.bloom_locality: 0 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.max_successive_merges: 0 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T17:24:19.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.ttl: 2592000 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.enable_blob_files: false 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.min_blob_size: 0 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T17:24:19.975 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4011e1b1-52dd-467c-88a7-e205600812fb 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714065859825829, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714065859826968, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99741, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97623, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14028, "raw_average_key_size": 49, "raw_value_size": 89981, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714065859, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4011e1b1-52dd-467c-88a7-e205600812fb", "db_session_id": "CZD757GIVYWNZFJ07SZ1", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714065859827050, "job": 1, "event": "recovery_finished"} 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cd04572000 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: DB pointer 0x55cd0455c000 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T17:24:19.976 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 99.26 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 99.26 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T17:24:19.977 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T17:24:19.978 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 30.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 30.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache BinnedLRUCache@0x55cd0445d1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: mon.a@-1(???) e1 preinit fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: mon.a@-1(???).mds e1 new map 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: mon.a@-1(???).mds e1 print_map 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T17:24:19.979 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 bash[29173]: 2fd9d01e54efa100d598d8a466c31b356cfc67f9df8336a5e9522851b39dbe6f 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 systemd[1]: Started Ceph mon.a for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: monmap epoch 1 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: last_changed 2024-04-25T17:24:12.001663+0000 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: created 2024-04-25T17:24:12.001663+0000 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: min_mon_release 19 (squid) 2024-04-25T17:24:19.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: election_strategy: 1 2024-04-25T17:24:19.981 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T17:24:19.981 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: fsmap 2024-04-25T17:24:19.981 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T17:24:19.981 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:19 smithi098 ceph-mon[29192]: mgrmap e1: no daemons active 2024-04-25T17:24:21.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.367+0000 7f938a1c9700 1 Processor -- start 2024-04-25T17:24:21.000 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.367+0000 7f938a1c9700 1 -- start start 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.367+0000 7f938a1c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f93841087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.367+0000 7f938a1c9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9384108d70 con 0x7f9384108380 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.368+0000 7f93837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f93841087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.368+0000 7f93837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f93841087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53200/0 (socket says 172.21.15.98:53200) 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.368+0000 7f93837fe700 1 -- 172.21.15.98:0/780553916 learned_addr learned my addr 172.21.15.98:0/780553916 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.368+0000 7f93837fe700 1 -- 172.21.15.98:0/780553916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9384109530 con 0x7f9384108380 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.368+0000 7f93837fe700 1 --2- 172.21.15.98:0/780553916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f93841087a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f93740098a0 tx=0x7f9374009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e69dd7216ed1274 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.368+0000 7f9382ffd700 1 -- 172.21.15.98:0/780553916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9374004160 con 0x7f9384108380 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.368+0000 7f9382ffd700 1 -- 172.21.15.98:0/780553916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9374004300 con 0x7f9384108380 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.368+0000 7f9382ffd700 1 -- 172.21.15.98:0/780553916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9374011af0 con 0x7f9384108380 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.369+0000 7f938a1c9700 1 -- 172.21.15.98:0/780553916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 msgr2=0x7f93841087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.369+0000 7f938a1c9700 1 --2- 172.21.15.98:0/780553916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f93841087a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f93740098a0 tx=0x7f9374009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.369+0000 7f938a1c9700 1 -- 172.21.15.98:0/780553916 shutdown_connections 2024-04-25T17:24:21.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.369+0000 7f938a1c9700 1 --2- 172.21.15.98:0/780553916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f93841087a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.369+0000 7f938a1c9700 1 -- 172.21.15.98:0/780553916 >> 172.21.15.98:0/780553916 conn(0x7f9384103920 msgr2=0x7f9384105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.369+0000 7f938a1c9700 1 -- 172.21.15.98:0/780553916 shutdown_connections 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.369+0000 7f938a1c9700 1 -- 172.21.15.98:0/780553916 wait complete. 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.370+0000 7f938a1c9700 1 Processor -- start 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.370+0000 7f938a1c9700 1 -- start start 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.370+0000 7f938a1c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f938419c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.370+0000 7f938a1c9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f938410c8e0 con 0x7f9384108380 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.370+0000 7f93837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f938419c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.370+0000 7f93837fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f938419c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53216/0 (socket says 172.21.15.98:53216) 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.370+0000 7f93837fe700 1 -- 172.21.15.98:0/3508390598 learned_addr learned my addr 172.21.15.98:0/3508390598 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.370+0000 7f93837fe700 1 -- 172.21.15.98:0/3508390598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938419c740 con 0x7f9384108380 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.370+0000 7f93837fe700 1 --2- 172.21.15.98:0/3508390598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f938419c200 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9374003f40 tx=0x7f9374011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.371+0000 7f93817fa700 1 -- 172.21.15.98:0/3508390598 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93740038b0 con 0x7f9384108380 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.371+0000 7f93817fa700 1 -- 172.21.15.98:0/3508390598 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9374003a50 con 0x7f9384108380 2024-04-25T17:24:21.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.371+0000 7f938a1c9700 1 -- 172.21.15.98:0/3508390598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f938419c9d0 con 0x7f9384108380 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.371+0000 7f93817fa700 1 -- 172.21.15.98:0/3508390598 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f937401a5c0 con 0x7f9384108380 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.371+0000 7f938a1c9700 1 -- 172.21.15.98:0/3508390598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f938419f6c0 con 0x7f9384108380 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.371+0000 7f93817fa700 1 -- 172.21.15.98:0/3508390598 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f937401ac00 con 0x7f9384108380 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.371+0000 7f93817fa700 1 -- 172.21.15.98:0/3508390598 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9374019700 con 0x7f9384108380 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.372+0000 7f938a1c9700 1 -- 172.21.15.98:0/3508390598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93840640a0 con 0x7f9384108380 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.374+0000 7f93817fa700 1 -- 172.21.15.98:0/3508390598 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9374019900 con 0x7f9384108380 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.443+0000 7f938a1c9700 1 -- 172.21.15.98:0/3508390598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f938419f890 con 0x7f9384108380 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.445+0000 7f93817fa700 1 -- 172.21.15.98:0/3508390598 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f9374019ae0 con 0x7f9384108380 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.445+0000 7f938a1c9700 1 -- 172.21.15.98:0/3508390598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 msgr2=0x7f938419c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.445+0000 7f938a1c9700 1 --2- 172.21.15.98:0/3508390598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f938419c200 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9374003f40 tx=0x7f9374011890 comp rx=0 tx=0).stop 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.446+0000 7f938a1c9700 1 -- 172.21.15.98:0/3508390598 shutdown_connections 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.446+0000 7f938a1c9700 1 --2- 172.21.15.98:0/3508390598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9384108380 0x7f938419c200 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.446+0000 7f938a1c9700 1 -- 172.21.15.98:0/3508390598 >> 172.21.15.98:0/3508390598 conn(0x7f9384103920 msgr2=0x7f9384104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:21.003 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.446+0000 7f938a1c9700 1 -- 172.21.15.98:0/3508390598 shutdown_connections 2024-04-25T17:24:21.004 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:20.446+0000 7f938a1c9700 1 -- 172.21.15.98:0/3508390598 wait complete. 2024-04-25T17:24:21.004 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T17:24:21.027 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T17:24:21.027 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mgr... 2024-04-25T17:24:21.030 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T17:24:21.030 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T17:24:21.270 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.x 2024-04-25T17:24:21.270 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T17:24:21.418 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-79754014-0328-11ef-bc93-c7b262605968.target.wants/ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-79754014-0328-11ef-bc93-c7b262605968@.service. 2024-04-25T17:24:21.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:21 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3508390598' entity='client.admin' 2024-04-25T17:24:21.713 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:21 smithi098 podman[29407]: 2024-04-25 17:24:21.662187763 +0000 UTC m=+0.120594438 container create d234cf7edff53886131a7ccf9ab3e36263a6e54e0f23ba504b8c2c146e222b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T17:24:21.939 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T17:24:21.939 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-25T17:24:21.946 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T17:24:21.946 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T17:24:21.946 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-25T17:24:21.946 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T17:24:21.954 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T17:24:21.954 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-25T17:24:21.961 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T17:24:21.962 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T17:24:21.962 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-25T17:24:21.962 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T17:24:21.962 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr to start... 2024-04-25T17:24:21.962 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr... 2024-04-25T17:24:22.027 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:21 smithi098 podman[29407]: 2024-04-25 17:24:21.864534662 +0000 UTC m=+0.322941292 container init d234cf7edff53886131a7ccf9ab3e36263a6e54e0f23ba504b8c2c146e222b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T17:24:22.027 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:21 smithi098 podman[29407]: 2024-04-25 17:24:21.86692233 +0000 UTC m=+0.325328960 container start d234cf7edff53886131a7ccf9ab3e36263a6e54e0f23ba504b8c2c146e222b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-25T17:24:22.027 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:21 smithi098 bash[29407]: d234cf7edff53886131a7ccf9ab3e36263a6e54e0f23ba504b8c2c146e222b60 2024-04-25T17:24:22.027 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:21 smithi098 systemd[1]: Started Ceph mgr.x for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:24:22.320 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:22 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:22.025+0000 7ff153deb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T17:24:22.321 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:22 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:22.216+0000 7ff153deb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:24:22.321 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:22 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:22.319+0000 7ff153deb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T17:24:22.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:22 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2118406576' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T17:24:22.955 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:22 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:22.644+0000 7ff153deb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T17:24:22.956 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:22 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:22.734+0000 7ff153deb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T17:24:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T17:24:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:24:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T17:24:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T17:24:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T17:24:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T17:24:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T17:24:23.143 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T17:24:23.144 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T17:24:23.146 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T17:24:14.957928+0000", 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.434+0000 7fc477c72700 1 Processor -- start 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.434+0000 7fc477c72700 1 -- start start 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.435+0000 7fc477c72700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc47010b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.435+0000 7fc477c72700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc47007be60 con 0x7fc470108e00 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.435+0000 7fc475a0e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc47010b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:23.147 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.435+0000 7fc475a0e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc47010b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53240/0 (socket says 172.21.15.98:53240) 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.435+0000 7fc475a0e700 1 -- 172.21.15.98:0/3811923047 learned_addr learned my addr 172.21.15.98:0/3811923047 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.435+0000 7fc475a0e700 1 -- 172.21.15.98:0/3811923047 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47010b730 con 0x7fc470108e00 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.435+0000 7fc475a0e700 1 --2- 172.21.15.98:0/3811923047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc47010b1f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc46c0098a0 tx=0x7fc46c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f404026f966b8f2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.436+0000 7fc474a0c700 1 -- 172.21.15.98:0/3811923047 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc46c004160 con 0x7fc470108e00 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.436+0000 7fc474a0c700 1 -- 172.21.15.98:0/3811923047 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc46c004300 con 0x7fc470108e00 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.436+0000 7fc474a0c700 1 -- 172.21.15.98:0/3811923047 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc46c011af0 con 0x7fc470108e00 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.437+0000 7fc477c72700 1 -- 172.21.15.98:0/3811923047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 msgr2=0x7fc47010b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.437+0000 7fc477c72700 1 --2- 172.21.15.98:0/3811923047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc47010b1f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc46c0098a0 tx=0x7fc46c009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.437+0000 7fc477c72700 1 -- 172.21.15.98:0/3811923047 shutdown_connections 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.437+0000 7fc477c72700 1 --2- 172.21.15.98:0/3811923047 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc47010b1f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.437+0000 7fc477c72700 1 -- 172.21.15.98:0/3811923047 >> 172.21.15.98:0/3811923047 conn(0x7fc470102890 msgr2=0x7fc470104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:23.148 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.437+0000 7fc477c72700 1 -- 172.21.15.98:0/3811923047 shutdown_connections 2024-04-25T17:24:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.437+0000 7fc477c72700 1 -- 172.21.15.98:0/3811923047 wait complete. 2024-04-25T17:24:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.437+0000 7fc477c72700 1 Processor -- start 2024-04-25T17:24:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.437+0000 7fc477c72700 1 -- start start 2024-04-25T17:24:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.437+0000 7fc477c72700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc470108680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc477c72700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc47010c6f0 con 0x7fc470108e00 2024-04-25T17:24:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc475a0e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc470108680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc475a0e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc470108680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53252/0 (socket says 172.21.15.98:53252) 2024-04-25T17:24:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc475a0e700 1 -- 172.21.15.98:0/2118406576 learned_addr learned my addr 172.21.15.98:0/2118406576 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:23.149 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc475a0e700 1 -- 172.21.15.98:0/2118406576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc470108bc0 con 0x7fc470108e00 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc475a0e700 1 --2- 172.21.15.98:0/2118406576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc470108680 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc46c011ac0 tx=0x7fc46c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc466ffd700 1 -- 172.21.15.98:0/2118406576 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc46c003b20 con 0x7fc470108e00 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc477c72700 1 -- 172.21.15.98:0/2118406576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc470106db0 con 0x7fc470108e00 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc466ffd700 1 -- 172.21.15.98:0/2118406576 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc46c01a8c0 con 0x7fc470108e00 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc466ffd700 1 -- 172.21.15.98:0/2118406576 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc46c0248d0 con 0x7fc470108e00 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.438+0000 7fc477c72700 1 -- 172.21.15.98:0/2118406576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4701072f0 con 0x7fc470108e00 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.439+0000 7fc466ffd700 1 -- 172.21.15.98:0/2118406576 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc46c022020 con 0x7fc470108e00 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.439+0000 7fc466ffd700 1 -- 172.21.15.98:0/2118406576 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc46c0193c0 con 0x7fc470108e00 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.439+0000 7fc477c72700 1 -- 172.21.15.98:0/2118406576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4700640a0 con 0x7fc470108e00 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.441+0000 7fc466ffd700 1 -- 172.21.15.98:0/2118406576 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc46c01aa70 con 0x7fc470108e00 2024-04-25T17:24:23.150 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.513+0000 7fc477c72700 1 -- 172.21.15.98:0/2118406576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc4701075a0 con 0x7fc470108e00 2024-04-25T17:24:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.514+0000 7fc466ffd700 1 -- 172.21.15.98:0/2118406576 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc46c01a430 con 0x7fc470108e00 2024-04-25T17:24:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.515+0000 7fc477c72700 1 -- 172.21.15.98:0/2118406576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 msgr2=0x7fc470108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.515+0000 7fc477c72700 1 --2- 172.21.15.98:0/2118406576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc470108680 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc46c011ac0 tx=0x7fc46c011ba0 comp rx=0 tx=0).stop 2024-04-25T17:24:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.515+0000 7fc477c72700 1 -- 172.21.15.98:0/2118406576 shutdown_connections 2024-04-25T17:24:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.515+0000 7fc477c72700 1 --2- 172.21.15.98:0/2118406576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc470108e00 0x7fc470108680 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.515+0000 7fc477c72700 1 -- 172.21.15.98:0/2118406576 >> 172.21.15.98:0/2118406576 conn(0x7fc470102890 msgr2=0x7fc4701032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.515+0000 7fc477c72700 1 -- 172.21.15.98:0/2118406576 shutdown_connections 2024-04-25T17:24:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:22.515+0000 7fc477c72700 1 -- 172.21.15.98:0/2118406576 wait complete. 2024-04-25T17:24:23.151 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (1/15)... 2024-04-25T17:24:24.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:23 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:23.923+0000 7ff153deb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T17:24:24.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:24 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:24.090+0000 7ff153deb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T17:24:24.541 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:24 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:24.259+0000 7ff153deb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T17:24:24.541 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:24 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:24.540+0000 7ff153deb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T17:24:24.831 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:24 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:24.631+0000 7ff153deb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T17:24:24.832 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:24 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:24.830+0000 7ff153deb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T17:24:25.205 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:25 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:25.020+0000 7ff153deb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:24:25.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:25 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/37997545' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T17:24:25.956 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:25 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:25.636+0000 7ff153deb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T17:24:26.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:26.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T17:24:26.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:24:26.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T17:24:26.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T17:24:26.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T17:24:26.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T17:24:26.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:26.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:26.277 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T17:24:26.278 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T17:24:26.278 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T17:24:26.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T17:24:26.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T17:24:26.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T17:24:26.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:26.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T17:24:26.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T17:24:26.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T17:24:26.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T17:24:26.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T17:24:26.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T17:24:26.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T17:24:14.957928+0000", 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.552+0000 7fd9f2d38700 1 Processor -- start 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.552+0000 7fd9f2d38700 1 -- start start 2024-04-25T17:24:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.552+0000 7fd9f2d38700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec107660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.552+0000 7fd9f2d38700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ec107c30 con 0x7fd9ec107240 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.553+0000 7fd9f0ad4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.553+0000 7fd9f0ad4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53572/0 (socket says 172.21.15.98:53572) 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.553+0000 7fd9f0ad4700 1 -- 172.21.15.98:0/3272235245 learned_addr learned my addr 172.21.15.98:0/3272235245 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.553+0000 7fd9f0ad4700 1 -- 172.21.15.98:0/3272235245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ec108440 con 0x7fd9ec107240 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.553+0000 7fd9f0ad4700 1 --2- 172.21.15.98:0/3272235245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec107660 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00098a0 tx=0x7fd9e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04d088ec1a91ce7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.553+0000 7fd9eb7fe700 1 -- 172.21.15.98:0/3272235245 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9e0004160 con 0x7fd9ec107240 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.553+0000 7fd9eb7fe700 1 -- 172.21.15.98:0/3272235245 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd9e0004300 con 0x7fd9ec107240 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.553+0000 7fd9eb7fe700 1 -- 172.21.15.98:0/3272235245 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9e0011af0 con 0x7fd9ec107240 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.554+0000 7fd9f2d38700 1 -- 172.21.15.98:0/3272235245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 msgr2=0x7fd9ec107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.554+0000 7fd9f2d38700 1 --2- 172.21.15.98:0/3272235245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec107660 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd9e00098a0 tx=0x7fd9e0009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.554+0000 7fd9f2d38700 1 -- 172.21.15.98:0/3272235245 shutdown_connections 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.554+0000 7fd9f2d38700 1 --2- 172.21.15.98:0/3272235245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec107660 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:26.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.554+0000 7fd9f2d38700 1 -- 172.21.15.98:0/3272235245 >> 172.21.15.98:0/3272235245 conn(0x7fd9ec102820 msgr2=0x7fd9ec104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.554+0000 7fd9f2d38700 1 -- 172.21.15.98:0/3272235245 shutdown_connections 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.554+0000 7fd9f2d38700 1 -- 172.21.15.98:0/3272235245 wait complete. 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.555+0000 7fd9f2d38700 1 Processor -- start 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.555+0000 7fd9f2d38700 1 -- start start 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.555+0000 7fd9f2d38700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec19c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.555+0000 7fd9f2d38700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ec10b7f0 con 0x7fd9ec107240 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.555+0000 7fd9f0ad4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec19c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.555+0000 7fd9f0ad4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec19c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53588/0 (socket says 172.21.15.98:53588) 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.555+0000 7fd9f0ad4700 1 -- 172.21.15.98:0/37997545 learned_addr learned my addr 172.21.15.98:0/37997545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.555+0000 7fd9f0ad4700 1 -- 172.21.15.98:0/37997545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ec19c7a0 con 0x7fd9ec107240 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.555+0000 7fd9f0ad4700 1 --2- 172.21.15.98:0/37997545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec19c260 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0003f40 tx=0x7fd9e0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.555+0000 7fd9e9ffb700 1 -- 172.21.15.98:0/37997545 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9e00038b0 con 0x7fd9ec107240 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.556+0000 7fd9e9ffb700 1 -- 172.21.15.98:0/37997545 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd9e0003a50 con 0x7fd9ec107240 2024-04-25T17:24:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.556+0000 7fd9f2d38700 1 -- 172.21.15.98:0/37997545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9ec19ca30 con 0x7fd9ec107240 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.556+0000 7fd9e9ffb700 1 -- 172.21.15.98:0/37997545 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9e001a5c0 con 0x7fd9ec107240 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.556+0000 7fd9f2d38700 1 -- 172.21.15.98:0/37997545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9ec19f720 con 0x7fd9ec107240 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.556+0000 7fd9e9ffb700 1 -- 172.21.15.98:0/37997545 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd9e001ac50 con 0x7fd9ec107240 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.556+0000 7fd9e9ffb700 1 -- 172.21.15.98:0/37997545 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd9e0019700 con 0x7fd9ec107240 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.557+0000 7fd9f2d38700 1 -- 172.21.15.98:0/37997545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9ec0640a0 con 0x7fd9ec107240 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.559+0000 7fd9e9ffb700 1 -- 172.21.15.98:0/37997545 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd9e00199a0 con 0x7fd9ec107240 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.630+0000 7fd9f2d38700 1 -- 172.21.15.98:0/37997545 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd9ec02e180 con 0x7fd9ec107240 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.630+0000 7fd9e9ffb700 1 -- 172.21.15.98:0/37997545 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd9e0019b80 con 0x7fd9ec107240 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.631+0000 7fd9f2d38700 1 -- 172.21.15.98:0/37997545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 msgr2=0x7fd9ec19c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.631+0000 7fd9f2d38700 1 --2- 172.21.15.98:0/37997545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec19c260 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0003f40 tx=0x7fd9e0011890 comp rx=0 tx=0).stop 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.631+0000 7fd9f2d38700 1 -- 172.21.15.98:0/37997545 shutdown_connections 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.631+0000 7fd9f2d38700 1 --2- 172.21.15.98:0/37997545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd9ec107240 0x7fd9ec19c260 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.631+0000 7fd9f2d38700 1 -- 172.21.15.98:0/37997545 >> 172.21.15.98:0/37997545 conn(0x7fd9ec102820 msgr2=0x7fd9ec103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.631+0000 7fd9f2d38700 1 -- 172.21.15.98:0/37997545 shutdown_connections 2024-04-25T17:24:26.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:25.631+0000 7fd9f2d38700 1 -- 172.21.15.98:0/37997545 wait complete. 2024-04-25T17:24:26.292 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (2/15)... 2024-04-25T17:24:26.364 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:26 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:26.064+0000 7ff153deb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T17:24:26.364 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:26 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:26.159+0000 7ff153deb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T17:24:26.364 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:26 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:26.363+0000 7ff153deb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T17:24:27.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:26 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:26.882+0000 7ff153deb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T17:24:27.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:26 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:26.973+0000 7ff153deb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T17:24:27.486 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:27 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:27.212+0000 7ff153deb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T17:24:27.486 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:27 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:27.343+0000 7ff153deb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T17:24:27.486 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:27 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:27.438+0000 7ff153deb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T17:24:27.857 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:27 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:27.559+0000 7ff153deb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T17:24:27.857 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:27 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:27.856+0000 7ff153deb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T17:24:28.665 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:28.463+0000 7ff153deb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T17:24:28.956 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:28.684+0000 7ff153deb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T17:24:28.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: Activating manager daemon x 2024-04-25T17:24:28.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: mgrmap e2: x(active, starting, since 0.0030713s) 2024-04-25T17:24:28.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: from='mgr.14100 172.21.15.98:0/2651349168' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T17:24:28.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: from='mgr.14100 172.21.15.98:0/2651349168' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T17:24:28.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: from='mgr.14100 172.21.15.98:0/2651349168' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T17:24:28.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: from='mgr.14100 172.21.15.98:0/2651349168' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:24:28.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: from='mgr.14100 172.21.15.98:0/2651349168' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T17:24:28.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: Manager daemon x is now available 2024-04-25T17:24:28.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: from='mgr.14100 172.21.15.98:0/2651349168' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T17:24:28.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: from='mgr.14100 172.21.15.98:0/2651349168' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T17:24:28.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: from='mgr.14100 172.21.15.98:0/2651349168' entity='mgr.x' 2024-04-25T17:24:28.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: from='mgr.14100 172.21.15.98:0/2651349168' entity='mgr.x' 2024-04-25T17:24:28.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:28 smithi098 ceph-mon[29192]: from='mgr.14100 172.21.15.98:0/2651349168' entity='mgr.x' 2024-04-25T17:24:29.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T17:24:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:24:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T17:24:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T17:24:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T17:24:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T17:24:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T17:24:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T17:24:29.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T17:24:29.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T17:24:29.333 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T17:24:29.333 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:29.334 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:29.335 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T17:24:14.957928+0000", 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.689+0000 7fc83d8ab700 1 Processor -- start 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.689+0000 7fc83d8ab700 1 -- start start 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.689+0000 7fc83d8ab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc8381087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.689+0000 7fc83d8ab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc838108d90 con 0x7fc8381083a0 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.690+0000 7fc836ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc8381087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.690+0000 7fc836ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc8381087c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53610/0 (socket says 172.21.15.98:53610) 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.690+0000 7fc836ffd700 1 -- 172.21.15.98:0/734584627 learned_addr learned my addr 172.21.15.98:0/734584627 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.690+0000 7fc836ffd700 1 -- 172.21.15.98:0/734584627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc838109550 con 0x7fc8381083a0 2024-04-25T17:24:29.336 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.690+0000 7fc836ffd700 1 --2- 172.21.15.98:0/734584627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc8381087c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc8280098a0 tx=0x7fc828009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddbe060556060208 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.691+0000 7fc835ffb700 1 -- 172.21.15.98:0/734584627 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc828004160 con 0x7fc8381083a0 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.691+0000 7fc835ffb700 1 -- 172.21.15.98:0/734584627 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc828004300 con 0x7fc8381083a0 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.691+0000 7fc835ffb700 1 -- 172.21.15.98:0/734584627 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc828011af0 con 0x7fc8381083a0 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.691+0000 7fc83d8ab700 1 -- 172.21.15.98:0/734584627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 msgr2=0x7fc8381087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.691+0000 7fc83d8ab700 1 --2- 172.21.15.98:0/734584627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc8381087c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc8280098a0 tx=0x7fc828009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.691+0000 7fc83d8ab700 1 -- 172.21.15.98:0/734584627 shutdown_connections 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.692+0000 7fc83d8ab700 1 --2- 172.21.15.98:0/734584627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc8381087c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.692+0000 7fc83d8ab700 1 -- 172.21.15.98:0/734584627 >> 172.21.15.98:0/734584627 conn(0x7fc838103920 msgr2=0x7fc838105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.692+0000 7fc83d8ab700 1 -- 172.21.15.98:0/734584627 shutdown_connections 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.692+0000 7fc83d8ab700 1 -- 172.21.15.98:0/734584627 wait complete. 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.692+0000 7fc83d8ab700 1 Processor -- start 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.692+0000 7fc83d8ab700 1 -- start start 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc83d8ab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc83807db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc83d8ab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc83810c900 con 0x7fc8381083a0 2024-04-25T17:24:29.337 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc836ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc83807db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc836ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc83807db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53630/0 (socket says 172.21.15.98:53630) 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc836ffd700 1 -- 172.21.15.98:0/1521282973 learned_addr learned my addr 172.21.15.98:0/1521282973 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc836ffd700 1 -- 172.21.15.98:0/1521282973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83807e0a0 con 0x7fc8381083a0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc836ffd700 1 --2- 172.21.15.98:0/1521282973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc83807db60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc828000c00 tx=0x7fc828011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc83c8a9700 1 -- 172.21.15.98:0/1521282973 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc828003b20 con 0x7fc8381083a0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc83d8ab700 1 -- 172.21.15.98:0/1521282973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83807c290 con 0x7fc8381083a0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc83d8ab700 1 -- 172.21.15.98:0/1521282973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83807c7d0 con 0x7fc8381083a0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.693+0000 7fc83c8a9700 1 -- 172.21.15.98:0/1521282973 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc82801a8c0 con 0x7fc8381083a0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.694+0000 7fc83c8a9700 1 -- 172.21.15.98:0/1521282973 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc828024940 con 0x7fc8381083a0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.694+0000 7fc83c8a9700 1 -- 172.21.15.98:0/1521282973 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 2) ==== 46314+0+0 (secure 0 0 0) 0x7fc828022020 con 0x7fc8381083a0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.694+0000 7fc83d8ab700 1 -- 172.21.15.98:0/1521282973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc824005360 con 0x7fc8381083a0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.695+0000 7fc83c8a9700 1 -- 172.21.15.98:0/1521282973 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc82804cff0 con 0x7fc8381083a0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.697+0000 7fc83c8a9700 1 -- 172.21.15.98:0/1521282973 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc82800b3e0 con 0x7fc8381083a0 2024-04-25T17:24:29.338 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.771+0000 7fc83d8ab700 1 -- 172.21.15.98:0/1521282973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc824005650 con 0x7fc8381083a0 2024-04-25T17:24:29.339 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.771+0000 7fc83c8a9700 1 -- 172.21.15.98:0/1521282973 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc82802d4a0 con 0x7fc8381083a0 2024-04-25T17:24:29.339 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.772+0000 7fc83d8ab700 1 -- 172.21.15.98:0/1521282973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 msgr2=0x7fc83807db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:29.339 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.772+0000 7fc83d8ab700 1 --2- 172.21.15.98:0/1521282973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc83807db60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc828000c00 tx=0x7fc828011b70 comp rx=0 tx=0).stop 2024-04-25T17:24:29.339 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.772+0000 7fc83d8ab700 1 -- 172.21.15.98:0/1521282973 shutdown_connections 2024-04-25T17:24:29.339 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.772+0000 7fc83d8ab700 1 --2- 172.21.15.98:0/1521282973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8381083a0 0x7fc83807db60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:29.339 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.772+0000 7fc83d8ab700 1 -- 172.21.15.98:0/1521282973 >> 172.21.15.98:0/1521282973 conn(0x7fc838103920 msgr2=0x7fc838104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:29.339 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.772+0000 7fc83d8ab700 1 -- 172.21.15.98:0/1521282973 shutdown_connections 2024-04-25T17:24:29.339 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:28.772+0000 7fc83d8ab700 1 -- 172.21.15.98:0/1521282973 wait complete. 2024-04-25T17:24:29.339 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (3/15)... 2024-04-25T17:24:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:29 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1521282973' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T17:24:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:29 smithi098 ceph-mon[29192]: mgrmap e3: x(active, since 1.00739s) 2024-04-25T17:24:31.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:31 smithi098 ceph-mon[29192]: mgrmap e4: x(active, since 2s) 2024-04-25T17:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T17:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T17:24:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:32.597 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T17:24:32.598 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:32.599 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T17:24:14.957928+0000", 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T17:24:32.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.713+0000 7fb810b0b700 1 Processor -- start 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.713+0000 7fb810b0b700 1 -- start start 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.713+0000 7fb810b0b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c10b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.713+0000 7fb810b0b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c07be60 con 0x7fb80c108e20 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.714+0000 7fb80a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c10b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.714+0000 7fb80a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c10b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53702/0 (socket says 172.21.15.98:53702) 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.714+0000 7fb80a59c700 1 -- 172.21.15.98:0/3042620448 learned_addr learned my addr 172.21.15.98:0/3042620448 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.714+0000 7fb80a59c700 1 -- 172.21.15.98:0/3042620448 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80c10b750 con 0x7fb80c108e20 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.714+0000 7fb80a59c700 1 --2- 172.21.15.98:0/3042620448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c10b210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009e80 tx=0x7fb7fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5563428179badc1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.714+0000 7fb80959a700 1 -- 172.21.15.98:0/3042620448 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7fc017070 con 0x7fb80c108e20 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.714+0000 7fb80959a700 1 -- 172.21.15.98:0/3042620448 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb7fc00b8c0 con 0x7fb80c108e20 2024-04-25T17:24:32.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.714+0000 7fb80959a700 1 -- 172.21.15.98:0/3042620448 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7fc004490 con 0x7fb80c108e20 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.715+0000 7fb810b0b700 1 -- 172.21.15.98:0/3042620448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 msgr2=0x7fb80c10b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.715+0000 7fb810b0b700 1 --2- 172.21.15.98:0/3042620448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c10b210 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009e80 tx=0x7fb7fc00b330 comp rx=0 tx=0).stop 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.715+0000 7fb810b0b700 1 -- 172.21.15.98:0/3042620448 shutdown_connections 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.715+0000 7fb810b0b700 1 --2- 172.21.15.98:0/3042620448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c10b210 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.715+0000 7fb810b0b700 1 -- 172.21.15.98:0/3042620448 >> 172.21.15.98:0/3042620448 conn(0x7fb80c102850 msgr2=0x7fb80c104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.715+0000 7fb810b0b700 1 -- 172.21.15.98:0/3042620448 shutdown_connections 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.715+0000 7fb810b0b700 1 -- 172.21.15.98:0/3042620448 wait complete. 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.716+0000 7fb810b0b700 1 Processor -- start 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.716+0000 7fb810b0b700 1 -- start start 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.716+0000 7fb810b0b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c1086a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.716+0000 7fb810b0b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c10c710 con 0x7fb80c108e20 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.716+0000 7fb80a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c1086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:32.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.716+0000 7fb80a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c1086a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53710/0 (socket says 172.21.15.98:53710) 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.716+0000 7fb80a59c700 1 -- 172.21.15.98:0/4044557312 learned_addr learned my addr 172.21.15.98:0/4044557312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.716+0000 7fb80a59c700 1 -- 172.21.15.98:0/4044557312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80c108be0 con 0x7fb80c108e20 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.716+0000 7fb80a59c700 1 --2- 172.21.15.98:0/4044557312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c1086a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00b7b0 tx=0x7fb7fc00be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.716+0000 7fb7fb7fe700 1 -- 172.21.15.98:0/4044557312 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7fc020020 con 0x7fb80c108e20 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.717+0000 7fb7fb7fe700 1 -- 172.21.15.98:0/4044557312 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb7fc003d10 con 0x7fb80c108e20 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.717+0000 7fb810b0b700 1 -- 172.21.15.98:0/4044557312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80c106dd0 con 0x7fb80c108e20 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.717+0000 7fb7fb7fe700 1 -- 172.21.15.98:0/4044557312 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7fc017040 con 0x7fb80c108e20 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.717+0000 7fb810b0b700 1 -- 172.21.15.98:0/4044557312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb80c107310 con 0x7fb80c108e20 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.717+0000 7fb7fb7fe700 1 -- 172.21.15.98:0/4044557312 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fb7fc023020 con 0x7fb80c108e20 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.717+0000 7fb810b0b700 1 -- 172.21.15.98:0/4044557312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb80c0640a0 con 0x7fb80c108e20 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.718+0000 7fb7fb7fe700 1 --2- 172.21.15.98:0/4044557312 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7fb7f4039620 0x7fb7f403bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.718+0000 7fb7fb7fe700 1 -- 172.21.15.98:0/4044557312 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb7fc00f050 con 0x7fb80c108e20 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.718+0000 7fb809d9b700 1 --2- 172.21.15.98:0/4044557312 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7fb7f4039620 0x7fb7f403bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:32.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.718+0000 7fb809d9b700 1 --2- 172.21.15.98:0/4044557312 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7fb7f4039620 0x7fb7f403bae0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb800006fd0 tx=0x7fb800006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:31.723+0000 7fb7fb7fe700 1 -- 172.21.15.98:0/4044557312 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7fc02a020 con 0x7fb80c108e20 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.002+0000 7fb810b0b700 1 -- 172.21.15.98:0/4044557312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb80c107c50 con 0x7fb80c108e20 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.002+0000 7fb7fb7fe700 1 -- 172.21.15.98:0/4044557312 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fb7fc003ec0 con 0x7fb80c108e20 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.005+0000 7fb810b0b700 1 -- 172.21.15.98:0/4044557312 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7fb7f4039620 msgr2=0x7fb7f403bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.005+0000 7fb810b0b700 1 --2- 172.21.15.98:0/4044557312 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7fb7f4039620 0x7fb7f403bae0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb800006fd0 tx=0x7fb800006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.005+0000 7fb810b0b700 1 -- 172.21.15.98:0/4044557312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 msgr2=0x7fb80c1086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.005+0000 7fb810b0b700 1 --2- 172.21.15.98:0/4044557312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c1086a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00b7b0 tx=0x7fb7fc00be90 comp rx=0 tx=0).stop 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.005+0000 7fb810b0b700 1 -- 172.21.15.98:0/4044557312 shutdown_connections 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.005+0000 7fb810b0b700 1 --2- 172.21.15.98:0/4044557312 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7fb7f4039620 0x7fb7f403bae0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.005+0000 7fb810b0b700 1 --2- 172.21.15.98:0/4044557312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb80c108e20 0x7fb80c1086a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.005+0000 7fb810b0b700 1 -- 172.21.15.98:0/4044557312 >> 172.21.15.98:0/4044557312 conn(0x7fb80c102850 msgr2=0x7fb80c1032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.006+0000 7fb810b0b700 1 -- 172.21.15.98:0/4044557312 shutdown_connections 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:32.006+0000 7fb810b0b700 1 -- 172.21.15.98:0/4044557312 wait complete. 2024-04-25T17:24:32.604 INFO:teuthology.orchestra.run.smithi098.stdout:mgr is available 2024-04-25T17:24:33.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:32 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4044557312' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T17:24:33.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:33.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T17:24:33.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:33.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T17:24:33.824 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.016+0000 7f459851b700 1 Processor -- start 2024-04-25T17:24:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.016+0000 7f459851b700 1 -- start start 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.016+0000 7f459851b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590106a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.016+0000 7f459851b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4590107040 con 0x7f4590106650 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.016+0000 7f45962b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590106a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.016+0000 7f45962b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590106a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53720/0 (socket says 172.21.15.98:53720) 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.016+0000 7f45962b7700 1 -- 172.21.15.98:0/2752171253 learned_addr learned my addr 172.21.15.98:0/2752171253 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.016+0000 7f45962b7700 1 -- 172.21.15.98:0/2752171253 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45901095e0 con 0x7f4590106650 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.016+0000 7f45962b7700 1 --2- 172.21.15.98:0/2752171253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590106a70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f45840098a0 tx=0x7f4584009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7159ed97c8d06f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.017+0000 7f45952b5700 1 -- 172.21.15.98:0/2752171253 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4584004160 con 0x7f4590106650 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.017+0000 7f45952b5700 1 -- 172.21.15.98:0/2752171253 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4584004300 con 0x7f4590106650 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.017+0000 7f45952b5700 1 -- 172.21.15.98:0/2752171253 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4584011af0 con 0x7f4590106650 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.017+0000 7f459851b700 1 -- 172.21.15.98:0/2752171253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 msgr2=0x7f4590106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.017+0000 7f459851b700 1 --2- 172.21.15.98:0/2752171253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590106a70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f45840098a0 tx=0x7f4584009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:33.826 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.017+0000 7f459851b700 1 -- 172.21.15.98:0/2752171253 shutdown_connections 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.017+0000 7f459851b700 1 --2- 172.21.15.98:0/2752171253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590106a70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.017+0000 7f459851b700 1 -- 172.21.15.98:0/2752171253 >> 172.21.15.98:0/2752171253 conn(0x7f4590101fe0 msgr2=0x7f4590104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.017+0000 7f459851b700 1 -- 172.21.15.98:0/2752171253 shutdown_connections 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.017+0000 7f459851b700 1 -- 172.21.15.98:0/2752171253 wait complete. 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.018+0000 7f459851b700 1 Processor -- start 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.018+0000 7f459851b700 1 -- start start 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.018+0000 7f459851b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590108e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.018+0000 7f459851b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f459010c990 con 0x7f4590106650 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.018+0000 7f45962b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590108e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.018+0000 7f45962b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590108e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53736/0 (socket says 172.21.15.98:53736) 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.018+0000 7f45962b7700 1 -- 172.21.15.98:0/1772407114 learned_addr learned my addr 172.21.15.98:0/1772407114 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.018+0000 7f45962b7700 1 -- 172.21.15.98:0/1772407114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45901093a0 con 0x7f4590106650 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.018+0000 7f45962b7700 1 --2- 172.21.15.98:0/1772407114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590108e60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4584003f40 tx=0x7f4584011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.019+0000 7f45837fe700 1 -- 172.21.15.98:0/1772407114 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45840038b0 con 0x7f4590106650 2024-04-25T17:24:33.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.019+0000 7f45837fe700 1 -- 172.21.15.98:0/1772407114 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4584003a50 con 0x7f4590106650 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.019+0000 7f45837fe700 1 -- 172.21.15.98:0/1772407114 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f458401a5c0 con 0x7f4590106650 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.019+0000 7f459851b700 1 -- 172.21.15.98:0/1772407114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4590107590 con 0x7f4590106650 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.019+0000 7f459851b700 1 -- 172.21.15.98:0/1772407114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4590107ad0 con 0x7f4590106650 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.019+0000 7f45837fe700 1 -- 172.21.15.98:0/1772407114 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f4584024430 con 0x7f4590106650 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.020+0000 7f459851b700 1 -- 172.21.15.98:0/1772407114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45900640a0 con 0x7f4590106650 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.020+0000 7f45837fe700 1 --2- 172.21.15.98:0/1772407114 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f457c0395d0 0x7f457c03ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.020+0000 7f45837fe700 1 -- 172.21.15.98:0/1772407114 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f458404da50 con 0x7f4590106650 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.020+0000 7f4595ab6700 1 --2- 172.21.15.98:0/1772407114 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f457c0395d0 0x7f457c03ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.020+0000 7f4595ab6700 1 --2- 172.21.15.98:0/1772407114 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f457c0395d0 0x7f457c03ba90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f458c006fd0 tx=0x7f458c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.025+0000 7f45837fe700 1 -- 172.21.15.98:0/1772407114 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f458404d470 con 0x7f4590106650 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.221+0000 7f459851b700 1 -- 172.21.15.98:0/1772407114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f4590107d50 con 0x7f4590106650 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.222+0000 7f45837fe700 1 -- 172.21.15.98:0/1772407114 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f458402d3f0 con 0x7f4590106650 2024-04-25T17:24:33.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.224+0000 7f459851b700 1 -- 172.21.15.98:0/1772407114 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f457c0395d0 msgr2=0x7f457c03ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.224+0000 7f459851b700 1 --2- 172.21.15.98:0/1772407114 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f457c0395d0 0x7f457c03ba90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f458c006fd0 tx=0x7f458c006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.224+0000 7f459851b700 1 -- 172.21.15.98:0/1772407114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 msgr2=0x7f4590108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.224+0000 7f459851b700 1 --2- 172.21.15.98:0/1772407114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590108e60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4584003f40 tx=0x7f4584011890 comp rx=0 tx=0).stop 2024-04-25T17:24:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.224+0000 7f459851b700 1 -- 172.21.15.98:0/1772407114 shutdown_connections 2024-04-25T17:24:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.224+0000 7f459851b700 1 --2- 172.21.15.98:0/1772407114 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f457c0395d0 0x7f457c03ba90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.224+0000 7f459851b700 1 --2- 172.21.15.98:0/1772407114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4590106650 0x7f4590108e60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.224+0000 7f459851b700 1 -- 172.21.15.98:0/1772407114 >> 172.21.15.98:0/1772407114 conn(0x7f4590101fe0 msgr2=0x7f4590102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.224+0000 7f459851b700 1 -- 172.21.15.98:0/1772407114 shutdown_connections 2024-04-25T17:24:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:33.224+0000 7f459851b700 1 -- 172.21.15.98:0/1772407114 wait complete. 2024-04-25T17:24:33.829 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling cephadm module... 2024-04-25T17:24:33.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:33 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1772407114' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T17:24:34.765 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:34 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3549959515' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T17:24:35.063 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:34 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: ignoring --setuser ceph since I am not root 2024-04-25T17:24:35.063 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:34 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: ignoring --setgroup ceph since I am not root 2024-04-25T17:24:35.063 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:34 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:34.918+0000 7ff36bcd9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T17:24:35.313 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:35 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:35.103+0000 7ff36bcd9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:24:35.313 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:35 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:35.204+0000 7ff36bcd9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T17:24:35.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.248+0000 7f4a91d9d700 1 Processor -- start 2024-04-25T17:24:35.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.248+0000 7f4a91d9d700 1 -- start start 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.249+0000 7f4a91d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c10b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.249+0000 7f4a91d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c07be60 con 0x7f4a8c108e00 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.249+0000 7f4a8b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c10b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.249+0000 7f4a8b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c10b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37582/0 (socket says 172.21.15.98:37582) 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.249+0000 7f4a8b7fe700 1 -- 172.21.15.98:0/1497156170 learned_addr learned my addr 172.21.15.98:0/1497156170 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.249+0000 7f4a8b7fe700 1 -- 172.21.15.98:0/1497156170 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c10b730 con 0x7f4a8c108e00 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.249+0000 7f4a8b7fe700 1 --2- 172.21.15.98:0/1497156170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c10b1f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c0098a0 tx=0x7f4a7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723915cb3aa92cbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.249+0000 7f4a8a7fc700 1 -- 172.21.15.98:0/1497156170 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a7c004160 con 0x7f4a8c108e00 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.249+0000 7f4a8a7fc700 1 -- 172.21.15.98:0/1497156170 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4a7c004300 con 0x7f4a8c108e00 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.249+0000 7f4a8a7fc700 1 -- 172.21.15.98:0/1497156170 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a7c011af0 con 0x7f4a8c108e00 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.250+0000 7f4a91d9d700 1 -- 172.21.15.98:0/1497156170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 msgr2=0x7f4a8c10b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:35.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.250+0000 7f4a91d9d700 1 --2- 172.21.15.98:0/1497156170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c10b1f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c0098a0 tx=0x7f4a7c009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.250+0000 7f4a91d9d700 1 -- 172.21.15.98:0/1497156170 shutdown_connections 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.250+0000 7f4a91d9d700 1 --2- 172.21.15.98:0/1497156170 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c10b1f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.250+0000 7f4a91d9d700 1 -- 172.21.15.98:0/1497156170 >> 172.21.15.98:0/1497156170 conn(0x7f4a8c1027f0 msgr2=0x7f4a8c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.250+0000 7f4a91d9d700 1 -- 172.21.15.98:0/1497156170 shutdown_connections 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.250+0000 7f4a91d9d700 1 -- 172.21.15.98:0/1497156170 wait complete. 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.250+0000 7f4a91d9d700 1 Processor -- start 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a91d9d700 1 -- start start 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a91d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c108680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a91d9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c10c6f0 con 0x7f4a8c108e00 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a8b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c108680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a8b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c108680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37590/0 (socket says 172.21.15.98:37590) 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a8b7fe700 1 -- 172.21.15.98:0/3549959515 learned_addr learned my addr 172.21.15.98:0/3549959515 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a8b7fe700 1 -- 172.21.15.98:0/3549959515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c108bc0 con 0x7f4a8c108e00 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a8b7fe700 1 --2- 172.21.15.98:0/3549959515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c108680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c003f40 tx=0x7f4a7c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:35.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a88ff9700 1 -- 172.21.15.98:0/3549959515 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a7c0038b0 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a88ff9700 1 -- 172.21.15.98:0/3549959515 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4a7c003a50 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a91d9d700 1 -- 172.21.15.98:0/3549959515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8c106db0 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.251+0000 7f4a88ff9700 1 -- 172.21.15.98:0/3549959515 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a7c01a5c0 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.252+0000 7f4a91d9d700 1 -- 172.21.15.98:0/3549959515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8c1072f0 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.252+0000 7f4a88ff9700 1 -- 172.21.15.98:0/3549959515 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f4a7c01a8c0 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.252+0000 7f4a88ff9700 1 --2- 172.21.15.98:0/3549959515 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f4a740395d0 0x7f4a7403ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.252+0000 7f4a91d9d700 1 -- 172.21.15.98:0/3549959515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8c0640a0 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.253+0000 7f4a88ff9700 1 -- 172.21.15.98:0/3549959515 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4a7c04da80 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.253+0000 7f4a8affd700 1 --2- 172.21.15.98:0/3549959515 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f4a740395d0 0x7f4a7403ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.253+0000 7f4a8affd700 1 --2- 172.21.15.98:0/3549959515 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f4a740395d0 0x7f4a7403ba90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4a80006fd0 tx=0x7f4a80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.258+0000 7f4a88ff9700 1 -- 172.21.15.98:0/3549959515 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a7c019c70 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.508+0000 7f4a91d9d700 1 -- 172.21.15.98:0/3549959515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f4a8c107570 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.746+0000 7f4a88ff9700 1 -- 172.21.15.98:0/3549959515 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f4a7c04d4a0 con 0x7f4a8c108e00 2024-04-25T17:24:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.747+0000 7f4a88ff9700 1 -- 172.21.15.98:0/3549959515 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f4a7c02d910 con 0x7f4a8c108e00 2024-04-25T17:24:35.326 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.748+0000 7f4a91d9d700 1 -- 172.21.15.98:0/3549959515 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f4a740395d0 msgr2=0x7f4a7403ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:35.326 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.748+0000 7f4a91d9d700 1 --2- 172.21.15.98:0/3549959515 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f4a740395d0 0x7f4a7403ba90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4a80006fd0 tx=0x7f4a80006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:35.326 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.748+0000 7f4a91d9d700 1 -- 172.21.15.98:0/3549959515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 msgr2=0x7f4a8c108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:35.326 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.748+0000 7f4a91d9d700 1 --2- 172.21.15.98:0/3549959515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c108680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c003f40 tx=0x7f4a7c011890 comp rx=0 tx=0).stop 2024-04-25T17:24:35.326 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.749+0000 7f4a91d9d700 1 -- 172.21.15.98:0/3549959515 shutdown_connections 2024-04-25T17:24:35.326 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.749+0000 7f4a91d9d700 1 --2- 172.21.15.98:0/3549959515 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f4a740395d0 0x7f4a7403ba90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:35.326 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.749+0000 7f4a91d9d700 1 --2- 172.21.15.98:0/3549959515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4a8c108e00 0x7f4a8c108680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:35.326 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.749+0000 7f4a91d9d700 1 -- 172.21.15.98:0/3549959515 >> 172.21.15.98:0/3549959515 conn(0x7f4a8c1027f0 msgr2=0x7f4a8c1032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:35.326 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.749+0000 7f4a91d9d700 1 -- 172.21.15.98:0/3549959515 shutdown_connections 2024-04-25T17:24:35.326 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:34.749+0000 7f4a91d9d700 1 -- 172.21.15.98:0/3549959515 wait complete. 2024-04-25T17:24:35.572 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:35 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:35.526+0000 7ff36bcd9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T17:24:35.956 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:35 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:35.614+0000 7ff36bcd9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T17:24:35.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:35 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3549959515' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T17:24:35.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:35 smithi098 ceph-mon[29192]: mgrmap e5: x(active, since 6s) 2024-04-25T17:24:36.679 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T17:24:36.679 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T17:24:36.679 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T17:24:36.679 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T17:24:36.679 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T17:24:36.679 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T17:24:36.679 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.773+0000 7f922a6a3700 1 Processor -- start 2024-04-25T17:24:36.679 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.773+0000 7f922a6a3700 1 -- start start 2024-04-25T17:24:36.679 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.773+0000 7f922a6a3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224106a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:36.679 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.773+0000 7f922a6a3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9224107020 con 0x7f9224106630 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.773+0000 7f9223fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224106a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.773+0000 7f9223fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224106a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37620/0 (socket says 172.21.15.98:37620) 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.773+0000 7f9223fff700 1 -- 172.21.15.98:0/19092755 learned_addr learned my addr 172.21.15.98:0/19092755 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.773+0000 7f9223fff700 1 -- 172.21.15.98:0/19092755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92241095c0 con 0x7f9224106630 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.774+0000 7f9223fff700 1 --2- 172.21.15.98:0/19092755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224106a50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f92140098a0 tx=0x7f9214009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149a26d1d66d057d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.774+0000 7f9222ffd700 1 -- 172.21.15.98:0/19092755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9214004160 con 0x7f9224106630 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.774+0000 7f9222ffd700 1 -- 172.21.15.98:0/19092755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9214004300 con 0x7f9224106630 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.774+0000 7f9222ffd700 1 -- 172.21.15.98:0/19092755 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9214011af0 con 0x7f9224106630 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.775+0000 7f922a6a3700 1 -- 172.21.15.98:0/19092755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 msgr2=0x7f9224106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.775+0000 7f922a6a3700 1 --2- 172.21.15.98:0/19092755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224106a50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f92140098a0 tx=0x7f9214009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.775+0000 7f922a6a3700 1 -- 172.21.15.98:0/19092755 shutdown_connections 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.775+0000 7f922a6a3700 1 --2- 172.21.15.98:0/19092755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224106a50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.775+0000 7f922a6a3700 1 -- 172.21.15.98:0/19092755 >> 172.21.15.98:0/19092755 conn(0x7f9224101fa0 msgr2=0x7f92241043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:36.680 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.775+0000 7f922a6a3700 1 -- 172.21.15.98:0/19092755 shutdown_connections 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.775+0000 7f922a6a3700 1 -- 172.21.15.98:0/19092755 wait complete. 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.775+0000 7f922a6a3700 1 Processor -- start 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.775+0000 7f922a6a3700 1 -- start start 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f922a6a3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224108e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f922a6a3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f922410c970 con 0x7f9224106630 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f9223fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224108e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f9223fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224108e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37626/0 (socket says 172.21.15.98:37626) 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f9223fff700 1 -- 172.21.15.98:0/3730850433 learned_addr learned my addr 172.21.15.98:0/3730850433 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f9223fff700 1 -- 172.21.15.98:0/3730850433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9224109380 con 0x7f9224106630 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f9223fff700 1 --2- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224108e40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9214000c00 tx=0x7f9214011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f92217fa700 1 -- 172.21.15.98:0/3730850433 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9214003b20 con 0x7f9224106630 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f92217fa700 1 -- 172.21.15.98:0/3730850433 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f921401a8c0 con 0x7f9224106630 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f922a6a3700 1 -- 172.21.15.98:0/3730850433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9224107570 con 0x7f9224106630 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f92217fa700 1 -- 172.21.15.98:0/3730850433 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9214024940 con 0x7f9224106630 2024-04-25T17:24:36.681 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.776+0000 7f922a6a3700 1 -- 172.21.15.98:0/3730850433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9224107ab0 con 0x7f9224106630 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.777+0000 7f92217fa700 1 -- 172.21.15.98:0/3730850433 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f9214022020 con 0x7f9224106630 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.777+0000 7f92217fa700 1 --2- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f920c0396c0 0x7f920c03bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.777+0000 7f922a6a3700 1 -- 172.21.15.98:0/3730850433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92240640a0 con 0x7f9224106630 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.777+0000 7f92237fe700 1 -- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f920c0396c0 msgr2=0x7f920c03bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3033733589 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.778+0000 7f92217fa700 1 -- 172.21.15.98:0/3730850433 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f921404e430 con 0x7f9224106630 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.778+0000 7f92237fe700 1 --2- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f920c0396c0 0x7f920c03bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.783+0000 7f92217fa700 1 -- 172.21.15.98:0/3730850433 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f921404e190 con 0x7f9224106630 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.979+0000 7f92237fe700 1 -- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f920c0396c0 msgr2=0x7f920c03bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3033733589 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:35.979+0000 7f92237fe700 1 --2- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f920c0396c0 0x7f920c03bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.078+0000 7f922a6a3700 1 -- 172.21.15.98:0/3730850433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f9224107d30 con 0x7f9224106630 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.079+0000 7f92217fa700 1 -- 172.21.15.98:0/3730850433 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f9214021d80 con 0x7f9224106630 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.081+0000 7f922a6a3700 1 -- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f920c0396c0 msgr2=0x7f920c03bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:24:36.682 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.081+0000 7f922a6a3700 1 --2- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f920c0396c0 0x7f920c03bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:36.683 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.081+0000 7f922a6a3700 1 -- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 msgr2=0x7f9224108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:36.683 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.081+0000 7f922a6a3700 1 --2- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224108e40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9214000c00 tx=0x7f9214011b70 comp rx=0 tx=0).stop 2024-04-25T17:24:36.683 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.081+0000 7f922a6a3700 1 -- 172.21.15.98:0/3730850433 shutdown_connections 2024-04-25T17:24:36.683 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.081+0000 7f922a6a3700 1 --2- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f920c0396c0 0x7f920c03bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:36.683 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.081+0000 7f922a6a3700 1 --2- 172.21.15.98:0/3730850433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9224106630 0x7f9224108e40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:36.683 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.081+0000 7f922a6a3700 1 -- 172.21.15.98:0/3730850433 >> 172.21.15.98:0/3730850433 conn(0x7f9224101fa0 msgr2=0x7f9224102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:36.683 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.081+0000 7f922a6a3700 1 -- 172.21.15.98:0/3730850433 shutdown_connections 2024-04-25T17:24:36.683 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:36.081+0000 7f922a6a3700 1 -- 172.21.15.98:0/3730850433 wait complete. 2024-04-25T17:24:36.683 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-04-25T17:24:36.683 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 5... 2024-04-25T17:24:36.940 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:36 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3730850433' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T17:24:36.940 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:36 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:36.834+0000 7ff36bcd9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T17:24:37.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:37 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:37.007+0000 7ff36bcd9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T17:24:37.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:37 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:37.182+0000 7ff36bcd9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T17:24:37.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:37 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:37.468+0000 7ff36bcd9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T17:24:37.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:37 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:37.560+0000 7ff36bcd9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T17:24:38.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:37 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:37.762+0000 7ff36bcd9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T17:24:38.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:37 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:37.956+0000 7ff36bcd9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:24:38.956 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:38 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:38.568+0000 7ff36bcd9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T17:24:39.291 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:38 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:38.993+0000 7ff36bcd9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T17:24:39.291 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:39 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:39.089+0000 7ff36bcd9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T17:24:39.291 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:39 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:39.290+0000 7ff36bcd9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T17:24:40.149 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:39 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:39.832+0000 7ff36bcd9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T17:24:40.150 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:39 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:39.923+0000 7ff36bcd9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T17:24:40.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:40.167+0000 7ff36bcd9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T17:24:40.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:40.300+0000 7ff36bcd9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T17:24:40.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:40.395+0000 7ff36bcd9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T17:24:40.810 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:40.511+0000 7ff36bcd9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T17:24:40.811 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:40.809+0000 7ff36bcd9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T17:24:41.674 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:41.407+0000 7ff36bcd9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T17:24:41.674 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:24:41.620+0000 7ff36bcd9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T17:24:41.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: Active manager daemon x restarted 2024-04-25T17:24:41.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: Activating manager daemon x 2024-04-25T17:24:41.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T17:24:41.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: mgrmap e6: x(active, starting, since 0.00353401s) 2024-04-25T17:24:41.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:24:41.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T17:24:41.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T17:24:41.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T17:24:41.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T17:24:41.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: Manager daemon x is now available 2024-04-25T17:24:41.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:41.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:41.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:24:41.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:24:41.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:41 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T17:24:42.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:42 smithi098 ceph-mon[29192]: Found migration_current of "None". Setting to last migration. 2024-04-25T17:24:42.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:42 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T17:24:42.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:42 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:42.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:42 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:42.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:42 smithi098 ceph-mon[29192]: mgrmap e7: x(active, since 1.00526s) 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.140+0000 7f1ddb9c8700 1 Processor -- start 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.140+0000 7f1ddb9c8700 1 -- start start 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.140+0000 7f1ddb9c8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd4107050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.140+0000 7f1ddb9c8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd407be20 con 0x7f1dd4106c30 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.140+0000 7f1dd9764700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd4107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.140+0000 7f1dd9764700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd4107050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37638/0 (socket says 172.21.15.98:37638) 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.140+0000 7f1dd9764700 1 -- 172.21.15.98:0/2684559766 learned_addr learned my addr 172.21.15.98:0/2684559766 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.141+0000 7f1dd9764700 1 -- 172.21.15.98:0/2684559766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd4107590 con 0x7f1dd4106c30 2024-04-25T17:24:43.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.141+0000 7f1dd9764700 1 --2- 172.21.15.98:0/2684559766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd4107050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00098a0 tx=0x7f1dd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8969ee33ed85cf87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.141+0000 7f1dcbfff700 1 -- 172.21.15.98:0/2684559766 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dd0004160 con 0x7f1dd4106c30 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.141+0000 7f1dcbfff700 1 -- 172.21.15.98:0/2684559766 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1dd0004300 con 0x7f1dd4106c30 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.141+0000 7f1dcbfff700 1 -- 172.21.15.98:0/2684559766 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dd0011af0 con 0x7f1dd4106c30 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.142+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/2684559766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 msgr2=0x7f1dd4107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.142+0000 7f1ddb9c8700 1 --2- 172.21.15.98:0/2684559766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd4107050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00098a0 tx=0x7f1dd0009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.142+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/2684559766 shutdown_connections 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.142+0000 7f1ddb9c8700 1 --2- 172.21.15.98:0/2684559766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd4107050 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.142+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/2684559766 >> 172.21.15.98:0/2684559766 conn(0x7f1dd41027f0 msgr2=0x7f1dd4104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.142+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/2684559766 shutdown_connections 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.142+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/2684559766 wait complete. 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.142+0000 7f1ddb9c8700 1 Processor -- start 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.142+0000 7f1ddb9c8700 1 -- start start 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1ddb9c8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd419e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1ddb9c8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd41085c0 con 0x7f1dd4106c30 2024-04-25T17:24:43.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1dd9764700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd419e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1dd9764700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd419e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37646/0 (socket says 172.21.15.98:37646) 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1dd9764700 1 -- 172.21.15.98:0/3593678693 learned_addr learned my addr 172.21.15.98:0/3593678693 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1dd9764700 1 -- 172.21.15.98:0/3593678693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd419e910 con 0x7f1dd4106c30 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1dd9764700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd419e3d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0011ac0 tx=0x7f1dd0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dd0003b20 con 0x7f1dd4106c30 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1dd001a8c0 con 0x7f1dd4106c30 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/3593678693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd419eba0 con 0x7f1dd4106c30 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1dd00248d0 con 0x7f1dd4106c30 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.143+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/3593678693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd41a1890 con 0x7f1dd4106c30 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.144+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f1dd0022020 con 0x7f1dd4106c30 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.144+0000 7f1dca7fc700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 0x7f1dc003bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.144+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 --> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1dc003c240 con 0x7f1dc0039670 2024-04-25T17:24:43.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.144+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1dd004d630 con 0x7f1dd4106c30 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.144+0000 7f1dd8f63700 1 -- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 msgr2=0x7f1dc003bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3033733589 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.144+0000 7f1dd8f63700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 0x7f1dc003bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.345+0000 7f1dd8f63700 1 -- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 msgr2=0x7f1dc003bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3033733589 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.345+0000 7f1dd8f63700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 0x7f1dc003bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.746+0000 7f1dd8f63700 1 -- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 msgr2=0x7f1dc003bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3033733589 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:37.746+0000 7f1dd8f63700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 0x7f1dc003bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:38.547+0000 7f1dd8f63700 1 -- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 msgr2=0x7f1dc003bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3033733589 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:38.547+0000 7f1dd8f63700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 0x7f1dc003bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:40.148+0000 7f1dd8f63700 1 -- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 msgr2=0x7f1dc003bb30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3033733589 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:40.148+0000 7f1dd8f63700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 0x7f1dc003bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:41.623+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f1dd002d3f0 con 0x7f1dd4106c30 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:41.624+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 msgr2=0x7f1dc003bb30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:41.624+0000 7f1dca7fc700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 0x7f1dc003bb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.624+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f1dd004f1c0 con 0x7f1dd4106c30 2024-04-25T17:24:43.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.624+0000 7f1dca7fc700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f1dc003cf60 0x7f1dc003f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.624+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 --> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1dc003c240 con 0x7f1dc003cf60 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.625+0000 7f1dd8f63700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f1dc003cf60 0x7f1dc003f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.625+0000 7f1dd8f63700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f1dc003cf60 0x7f1dc003f350 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4003a10 tx=0x7f1dc4009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.626+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 <== mgr.14120 v2:172.21.15.98:6800/3170662380 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f1dc003c240 con 0x7f1dc003cf60 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.632+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/3593678693 --> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f1dd4069b50 con 0x7f1dc003cf60 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1dca7fc700 1 -- 172.21.15.98:0/3593678693 <== mgr.14120 v2:172.21.15.98:6800/3170662380 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f1dd4069b50 con 0x7f1dc003cf60 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f1dc003cf60 msgr2=0x7f1dc003f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f1dc003cf60 0x7f1dc003f350 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4003a10 tx=0x7f1dc4009290 comp rx=0 tx=0).stop 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 msgr2=0x7f1dd419e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd419e3d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0011ac0 tx=0x7f1dd0011ba0 comp rx=0 tx=0).stop 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/3593678693 shutdown_connections 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f1dc003cf60 0x7f1dc003f350 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1dd4106c30 0x7f1dd419e3d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 --2- 172.21.15.98:0/3593678693 >> [v2:172.21.15.98:6800/3033733589,v1:172.21.15.98:6801/3033733589] conn(0x7f1dc0039670 0x7f1dc003bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:43.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/3593678693 >> 172.21.15.98:0/3593678693 conn(0x7f1dd41027f0 msgr2=0x7f1dd4195040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:43.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/3593678693 shutdown_connections 2024-04-25T17:24:43.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:42.633+0000 7f1ddb9c8700 1 -- 172.21.15.98:0/3593678693 wait complete. 2024-04-25T17:24:43.233 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 5 is available 2024-04-25T17:24:43.233 INFO:teuthology.orchestra.run.smithi098.stdout:Setting orchestrator backend to cephadm... 2024-04-25T17:24:43.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:43 smithi098 ceph-mon[29192]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T17:24:43.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:43 smithi098 ceph-mon[29192]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T17:24:43.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:43 smithi098 ceph-mon[29192]: [25/Apr/2024:17:24:42] ENGINE Bus STARTING 2024-04-25T17:24:43.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:43 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:24:44.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.603+0000 7f8bde5ea700 1 Processor -- start 2024-04-25T17:24:44.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.603+0000 7f8bde5ea700 1 -- start start 2024-04-25T17:24:44.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.603+0000 7f8bde5ea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd807c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.603+0000 7f8bde5ea700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd807c8c0 con 0x7f8bd807de90 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.603+0000 7f8bd7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd807c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.604+0000 7f8bd7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd807c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54968/0 (socket says 172.21.15.98:54968) 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.604+0000 7f8bd7fff700 1 -- 172.21.15.98:0/4113423211 learned_addr learned my addr 172.21.15.98:0/4113423211 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.604+0000 7f8bd7fff700 1 -- 172.21.15.98:0/4113423211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd807ca40 con 0x7f8bd807de90 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.604+0000 7f8bd7fff700 1 --2- 172.21.15.98:0/4113423211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd807c2f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80098a0 tx=0x7f8bc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0452bb85994dd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.604+0000 7f8bd6ffd700 1 -- 172.21.15.98:0/4113423211 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bc8004160 con 0x7f8bd807de90 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.604+0000 7f8bd6ffd700 1 -- 172.21.15.98:0/4113423211 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8bc8004300 con 0x7f8bd807de90 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.604+0000 7f8bd6ffd700 1 -- 172.21.15.98:0/4113423211 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bc8011af0 con 0x7f8bd807de90 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.605+0000 7f8bde5ea700 1 -- 172.21.15.98:0/4113423211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 msgr2=0x7f8bd807c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.605+0000 7f8bde5ea700 1 --2- 172.21.15.98:0/4113423211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd807c2f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80098a0 tx=0x7f8bc8009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.605+0000 7f8bde5ea700 1 -- 172.21.15.98:0/4113423211 shutdown_connections 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.605+0000 7f8bde5ea700 1 --2- 172.21.15.98:0/4113423211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd807c2f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:44.433 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.605+0000 7f8bde5ea700 1 -- 172.21.15.98:0/4113423211 >> 172.21.15.98:0/4113423211 conn(0x7f8bd81038e0 msgr2=0x7f8bd8105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.605+0000 7f8bde5ea700 1 -- 172.21.15.98:0/4113423211 shutdown_connections 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.605+0000 7f8bde5ea700 1 -- 172.21.15.98:0/4113423211 wait complete. 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.605+0000 7f8bde5ea700 1 Processor -- start 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.605+0000 7f8bde5ea700 1 -- start start 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bde5ea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd819e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bde5ea700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd810e8f0 con 0x7f8bd807de90 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bd7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd819e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bd7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd819e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54972/0 (socket says 172.21.15.98:54972) 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bd7fff700 1 -- 172.21.15.98:0/774220693 learned_addr learned my addr 172.21.15.98:0/774220693 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bd7fff700 1 -- 172.21.15.98:0/774220693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd819e910 con 0x7f8bd807de90 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bd7fff700 1 --2- 172.21.15.98:0/774220693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd819e3d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8003f40 tx=0x7f8bc8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bd57fa700 1 -- 172.21.15.98:0/774220693 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bc80038b0 con 0x7f8bd807de90 2024-04-25T17:24:44.434 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bd57fa700 1 -- 172.21.15.98:0/774220693 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8bc8003a50 con 0x7f8bd807de90 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bd57fa700 1 -- 172.21.15.98:0/774220693 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bc801a5c0 con 0x7f8bd807de90 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bde5ea700 1 -- 172.21.15.98:0/774220693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd819eba0 con 0x7f8bd807de90 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.606+0000 7f8bde5ea700 1 -- 172.21.15.98:0/774220693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd81a1890 con 0x7f8bd807de90 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.607+0000 7f8bd57fa700 1 -- 172.21.15.98:0/774220693 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f8bc801a8c0 con 0x7f8bd807de90 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.607+0000 7f8bde5ea700 1 -- 172.21.15.98:0/774220693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd80640a0 con 0x7f8bd807de90 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.607+0000 7f8bd57fa700 1 --2- 172.21.15.98:0/774220693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f8bc0039510 0x7f8bc003b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.607+0000 7f8bd57fa700 1 -- 172.21.15.98:0/774220693 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8bc804dbb0 con 0x7f8bd807de90 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.608+0000 7f8bd77fe700 1 --2- 172.21.15.98:0/774220693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f8bc0039510 0x7f8bc003b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.608+0000 7f8bd77fe700 1 --2- 172.21.15.98:0/774220693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f8bc0039510 0x7f8bc003b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc006fd0 tx=0x7f8bcc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.613+0000 7f8bd57fa700 1 -- 172.21.15.98:0/774220693 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8bc8019c70 con 0x7f8bd807de90 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.679+0000 7f8bd57fa700 1 -- 172.21.15.98:0/774220693 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f8bc802d910 con 0x7f8bd807de90 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.828+0000 7f8bde5ea700 1 -- 172.21.15.98:0/774220693 --> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f8bd8104ab0 con 0x7f8bc0039510 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.834+0000 7f8bd57fa700 1 -- 172.21.15.98:0/774220693 <== mgr.14120 v2:172.21.15.98:6800/3170662380 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8bd8104ab0 con 0x7f8bc0039510 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.836+0000 7f8bde5ea700 1 -- 172.21.15.98:0/774220693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f8bc0039510 msgr2=0x7f8bc003b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:44.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.836+0000 7f8bde5ea700 1 --2- 172.21.15.98:0/774220693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f8bc0039510 0x7f8bc003b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc006fd0 tx=0x7f8bcc006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:44.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.836+0000 7f8bde5ea700 1 -- 172.21.15.98:0/774220693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 msgr2=0x7f8bd819e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:44.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.836+0000 7f8bde5ea700 1 --2- 172.21.15.98:0/774220693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd819e3d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8003f40 tx=0x7f8bc8011890 comp rx=0 tx=0).stop 2024-04-25T17:24:44.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.837+0000 7f8bde5ea700 1 -- 172.21.15.98:0/774220693 shutdown_connections 2024-04-25T17:24:44.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.837+0000 7f8bde5ea700 1 --2- 172.21.15.98:0/774220693 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f8bc0039510 0x7f8bc003b9d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:44.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.837+0000 7f8bde5ea700 1 --2- 172.21.15.98:0/774220693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd807de90 0x7f8bd819e3d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:44.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.837+0000 7f8bde5ea700 1 -- 172.21.15.98:0/774220693 >> 172.21.15.98:0/774220693 conn(0x7f8bd81038e0 msgr2=0x7f8bd81043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:44.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.837+0000 7f8bde5ea700 1 -- 172.21.15.98:0/774220693 shutdown_connections 2024-04-25T17:24:44.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:43.837+0000 7f8bde5ea700 1 -- 172.21.15.98:0/774220693 wait complete. 2024-04-25T17:24:44.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:44 smithi098 ceph-mon[29192]: [25/Apr/2024:17:24:42] ENGINE Serving on https://172.21.15.98:7150 2024-04-25T17:24:44.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:44 smithi098 ceph-mon[29192]: [25/Apr/2024:17:24:42] ENGINE Serving on http://172.21.15.98:8765 2024-04-25T17:24:44.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:44 smithi098 ceph-mon[29192]: [25/Apr/2024:17:24:42] ENGINE Bus STARTED 2024-04-25T17:24:44.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:44 smithi098 ceph-mon[29192]: mgrmap e8: x(active, since 2s) 2024-04-25T17:24:44.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:44 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:44.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:44 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:24:45.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T17:24:45.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.805+0000 7efd8ddda700 1 Processor -- start 2024-04-25T17:24:45.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.805+0000 7efd8ddda700 1 -- start start 2024-04-25T17:24:45.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.805+0000 7efd8ddda700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd881076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:45.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.805+0000 7efd8ddda700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd88107c70 con 0x7efd88107280 2024-04-25T17:24:45.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.806+0000 7efd877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd881076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:45.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.806+0000 7efd877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd881076a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54988/0 (socket says 172.21.15.98:54988) 2024-04-25T17:24:45.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.806+0000 7efd877fe700 1 -- 172.21.15.98:0/552578733 learned_addr learned my addr 172.21.15.98:0/552578733 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:45.640 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.806+0000 7efd877fe700 1 -- 172.21.15.98:0/552578733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd88108480 con 0x7efd88107280 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.806+0000 7efd877fe700 1 --2- 172.21.15.98:0/552578733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd881076a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efd780098a0 tx=0x7efd78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854830f9f7f06a86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.806+0000 7efd867fc700 1 -- 172.21.15.98:0/552578733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd78004160 con 0x7efd88107280 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.806+0000 7efd867fc700 1 -- 172.21.15.98:0/552578733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efd78004300 con 0x7efd88107280 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.806+0000 7efd867fc700 1 -- 172.21.15.98:0/552578733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd78011af0 con 0x7efd88107280 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.807+0000 7efd8ddda700 1 -- 172.21.15.98:0/552578733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 msgr2=0x7efd881076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.807+0000 7efd8ddda700 1 --2- 172.21.15.98:0/552578733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd881076a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efd780098a0 tx=0x7efd78009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.807+0000 7efd8ddda700 1 -- 172.21.15.98:0/552578733 shutdown_connections 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.807+0000 7efd8ddda700 1 --2- 172.21.15.98:0/552578733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd881076a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.807+0000 7efd8ddda700 1 -- 172.21.15.98:0/552578733 >> 172.21.15.98:0/552578733 conn(0x7efd88102860 msgr2=0x7efd88104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.807+0000 7efd8ddda700 1 -- 172.21.15.98:0/552578733 shutdown_connections 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.807+0000 7efd8ddda700 1 -- 172.21.15.98:0/552578733 wait complete. 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.807+0000 7efd8ddda700 1 Processor -- start 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.807+0000 7efd8ddda700 1 -- start start 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd8ddda700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd8819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:45.641 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd8ddda700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd8810b830 con 0x7efd88107280 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd8819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd8819c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54994/0 (socket says 172.21.15.98:54994) 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd877fe700 1 -- 172.21.15.98:0/2309355810 learned_addr learned my addr 172.21.15.98:0/2309355810 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd877fe700 1 -- 172.21.15.98:0/2309355810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8819c790 con 0x7efd88107280 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd877fe700 1 --2- 172.21.15.98:0/2309355810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd8819c250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efd78000c00 tx=0x7efd78011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd84ff9700 1 -- 172.21.15.98:0/2309355810 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd78003b20 con 0x7efd88107280 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd8ddda700 1 -- 172.21.15.98:0/2309355810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd8819ca20 con 0x7efd88107280 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd84ff9700 1 -- 172.21.15.98:0/2309355810 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efd7801a8c0 con 0x7efd88107280 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd8ddda700 1 -- 172.21.15.98:0/2309355810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd8819f710 con 0x7efd88107280 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.808+0000 7efd84ff9700 1 -- 172.21.15.98:0/2309355810 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd78004300 con 0x7efd88107280 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.809+0000 7efd84ff9700 1 -- 172.21.15.98:0/2309355810 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7efd78022020 con 0x7efd88107280 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.809+0000 7efd8ddda700 1 -- 172.21.15.98:0/2309355810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd88069b50 con 0x7efd88107280 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.809+0000 7efd84ff9700 1 --2- 172.21.15.98:0/2309355810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7efd70039810 0x7efd7003bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:45.642 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.809+0000 7efd84ff9700 1 -- 172.21.15.98:0/2309355810 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efd7804d870 con 0x7efd88107280 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.809+0000 7efd86ffd700 1 --2- 172.21.15.98:0/2309355810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7efd70039810 0x7efd7003bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.810+0000 7efd86ffd700 1 --2- 172.21.15.98:0/2309355810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7efd70039810 0x7efd7003bcd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efd7c006fd0 tx=0x7efd7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:44.814+0000 7efd84ff9700 1 -- 172.21.15.98:0/2309355810 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd7802c9a0 con 0x7efd88107280 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.014+0000 7efd8ddda700 1 -- 172.21.15.98:0/2309355810 --> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7efd88076600 con 0x7efd70039810 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.015+0000 7efd84ff9700 1 -- 172.21.15.98:0/2309355810 <== mgr.14120 v2:172.21.15.98:6800/3170662380 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7efd88076600 con 0x7efd70039810 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.017+0000 7efd8ddda700 1 -- 172.21.15.98:0/2309355810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7efd70039810 msgr2=0x7efd7003bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.017+0000 7efd8ddda700 1 --2- 172.21.15.98:0/2309355810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7efd70039810 0x7efd7003bcd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efd7c006fd0 tx=0x7efd7c006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.017+0000 7efd8ddda700 1 -- 172.21.15.98:0/2309355810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 msgr2=0x7efd8819c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.017+0000 7efd8ddda700 1 --2- 172.21.15.98:0/2309355810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd8819c250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efd78000c00 tx=0x7efd78011b70 comp rx=0 tx=0).stop 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.017+0000 7efd8ddda700 1 -- 172.21.15.98:0/2309355810 shutdown_connections 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.017+0000 7efd8ddda700 1 --2- 172.21.15.98:0/2309355810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7efd70039810 0x7efd7003bcd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.017+0000 7efd8ddda700 1 --2- 172.21.15.98:0/2309355810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd88107280 0x7efd8819c250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.017+0000 7efd8ddda700 1 -- 172.21.15.98:0/2309355810 >> 172.21.15.98:0/2309355810 conn(0x7efd88102860 msgr2=0x7efd88103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.017+0000 7efd8ddda700 1 -- 172.21.15.98:0/2309355810 shutdown_connections 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:45.017+0000 7efd8ddda700 1 -- 172.21.15.98:0/2309355810 wait complete. 2024-04-25T17:24:45.643 INFO:teuthology.orchestra.run.smithi098.stdout:Generating ssh key... 2024-04-25T17:24:45.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:45 smithi098 ceph-mon[29192]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:24:46.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-mon[29192]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:24:47.282 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: Generating public/private rsa key pair. 2024-04-25T17:24:47.283 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: Your identification has been saved in /tmp/tmpwjz3usit/key. 2024-04-25T17:24:47.283 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: Your public key has been saved in /tmp/tmpwjz3usit/key.pub. 2024-04-25T17:24:47.283 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: The key fingerprint is: 2024-04-25T17:24:47.283 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: SHA256:6/PFpQw3lBvv4ZMgT3jaQn7t9zFkk0he7l+dhGBbfHw ceph-79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:47.283 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: The key's randomart image is: 2024-04-25T17:24:47.283 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: +---[RSA 3072]----+ 2024-04-25T17:24:47.283 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: | | 2024-04-25T17:24:47.283 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: | ... | 2024-04-25T17:24:47.283 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: | o++ +E| 2024-04-25T17:24:47.283 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: | .o*+* o| 2024-04-25T17:24:47.284 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: | S =.B++B | 2024-04-25T17:24:47.284 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: | + % O=o+| 2024-04-25T17:24:47.284 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: | . + O *=o| 2024-04-25T17:24:47.284 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: | .. + . .*| 2024-04-25T17:24:47.284 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: | .o. ..+| 2024-04-25T17:24:47.284 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:24:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: +----[SHA256]-----+ 2024-04-25T17:24:47.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.050+0000 7f5e069ed700 1 Processor -- start 2024-04-25T17:24:47.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.050+0000 7f5e069ed700 1 -- start start 2024-04-25T17:24:47.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.050+0000 7f5e069ed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0010b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:47.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.050+0000 7f5e069ed700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0007be80 con 0x7f5e00108e00 2024-04-25T17:24:47.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.050+0000 7f5dfffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0010b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:47.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.050+0000 7f5dfffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0010b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55010/0 (socket says 172.21.15.98:55010) 2024-04-25T17:24:47.589 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.050+0000 7f5dfffff700 1 -- 172.21.15.98:0/2823840701 learned_addr learned my addr 172.21.15.98:0/2823840701 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.050+0000 7f5dfffff700 1 -- 172.21.15.98:0/2823840701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0010b730 con 0x7f5e00108e00 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.051+0000 7f5dfffff700 1 --2- 172.21.15.98:0/2823840701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0010b1f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5df00098a0 tx=0x7f5df0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85cb2ec5a5ac672 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.051+0000 7f5dfeffd700 1 -- 172.21.15.98:0/2823840701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5df0004160 con 0x7f5e00108e00 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.051+0000 7f5dfeffd700 1 -- 172.21.15.98:0/2823840701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5df0004300 con 0x7f5e00108e00 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.051+0000 7f5dfeffd700 1 -- 172.21.15.98:0/2823840701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5df0011af0 con 0x7f5e00108e00 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.051+0000 7f5e069ed700 1 -- 172.21.15.98:0/2823840701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 msgr2=0x7f5e0010b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.051+0000 7f5e069ed700 1 --2- 172.21.15.98:0/2823840701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0010b1f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5df00098a0 tx=0x7f5df0009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.051+0000 7f5e069ed700 1 -- 172.21.15.98:0/2823840701 shutdown_connections 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.051+0000 7f5e069ed700 1 --2- 172.21.15.98:0/2823840701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0010b1f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.051+0000 7f5e069ed700 1 -- 172.21.15.98:0/2823840701 >> 172.21.15.98:0/2823840701 conn(0x7f5e00102850 msgr2=0x7f5e00104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.052+0000 7f5e069ed700 1 -- 172.21.15.98:0/2823840701 shutdown_connections 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.052+0000 7f5e069ed700 1 -- 172.21.15.98:0/2823840701 wait complete. 2024-04-25T17:24:47.590 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.052+0000 7f5e069ed700 1 Processor -- start 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.052+0000 7f5e069ed700 1 -- start start 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.052+0000 7f5e069ed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0019e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.052+0000 7f5e069ed700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0010c680 con 0x7f5e00108e00 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.053+0000 7f5dfffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0019e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.053+0000 7f5dfffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0019e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55018/0 (socket says 172.21.15.98:55018) 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.053+0000 7f5dfffff700 1 -- 172.21.15.98:0/2470680508 learned_addr learned my addr 172.21.15.98:0/2470680508 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.053+0000 7f5dfffff700 1 -- 172.21.15.98:0/2470680508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0019e8d0 con 0x7f5e00108e00 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.053+0000 7f5dfffff700 1 --2- 172.21.15.98:0/2470680508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0019e390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5df0003f40 tx=0x7f5df0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.053+0000 7f5dfd7fa700 1 -- 172.21.15.98:0/2470680508 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5df00038b0 con 0x7f5e00108e00 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.053+0000 7f5dfd7fa700 1 -- 172.21.15.98:0/2470680508 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5df0003a50 con 0x7f5e00108e00 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.053+0000 7f5e069ed700 1 -- 172.21.15.98:0/2470680508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e0019ebc0 con 0x7f5e00108e00 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.053+0000 7f5dfd7fa700 1 -- 172.21.15.98:0/2470680508 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5df001a5c0 con 0x7f5e00108e00 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.053+0000 7f5e069ed700 1 -- 172.21.15.98:0/2470680508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e001a1850 con 0x7f5e00108e00 2024-04-25T17:24:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.054+0000 7f5dfd7fa700 1 -- 172.21.15.98:0/2470680508 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5df001a8c0 con 0x7f5e00108e00 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.054+0000 7f5e069ed700 1 -- 172.21.15.98:0/2470680508 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e000640a0 con 0x7f5e00108e00 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.054+0000 7f5dfd7fa700 1 --2- 172.21.15.98:0/2470680508 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f5de8039620 0x7f5de803bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.054+0000 7f5dfd7fa700 1 -- 172.21.15.98:0/2470680508 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5df004dd40 con 0x7f5e00108e00 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.054+0000 7f5dff7fe700 1 --2- 172.21.15.98:0/2470680508 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f5de8039620 0x7f5de803bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.055+0000 7f5dff7fe700 1 --2- 172.21.15.98:0/2470680508 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f5de8039620 0x7f5de803bae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5df4006fd0 tx=0x7f5df4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.059+0000 7f5dfd7fa700 1 -- 172.21.15.98:0/2470680508 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5df0019c70 con 0x7f5e00108e00 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.258+0000 7f5e069ed700 1 -- 172.21.15.98:0/2470680508 --> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f5e001a1a20 con 0x7f5de8039620 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.997+0000 7f5dfd7fa700 1 -- 172.21.15.98:0/2470680508 <== mgr.14120 v2:172.21.15.98:6800/3170662380 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5e001a1a20 con 0x7f5de8039620 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.999+0000 7f5e069ed700 1 -- 172.21.15.98:0/2470680508 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f5de8039620 msgr2=0x7f5de803bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.999+0000 7f5e069ed700 1 --2- 172.21.15.98:0/2470680508 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f5de8039620 0x7f5de803bae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5df4006fd0 tx=0x7f5df4006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.999+0000 7f5e069ed700 1 -- 172.21.15.98:0/2470680508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 msgr2=0x7f5e0019e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:46.999+0000 7f5e069ed700 1 --2- 172.21.15.98:0/2470680508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0019e390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5df0003f40 tx=0x7f5df0011890 comp rx=0 tx=0).stop 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:47.000+0000 7f5e069ed700 1 -- 172.21.15.98:0/2470680508 shutdown_connections 2024-04-25T17:24:47.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:47.000+0000 7f5e069ed700 1 --2- 172.21.15.98:0/2470680508 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f5de8039620 0x7f5de803bae0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:47.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:47.000+0000 7f5e069ed700 1 --2- 172.21.15.98:0/2470680508 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e00108e00 0x7f5e0019e390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:47.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:47.000+0000 7f5e069ed700 1 -- 172.21.15.98:0/2470680508 >> 172.21.15.98:0/2470680508 conn(0x7f5e00102850 msgr2=0x7f5e00103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:47.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:47.000+0000 7f5e069ed700 1 -- 172.21.15.98:0/2470680508 shutdown_connections 2024-04-25T17:24:47.593 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:47.000+0000 7f5e069ed700 1 -- 172.21.15.98:0/2470680508 wait complete. 2024-04-25T17:24:47.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:47 smithi098 ceph-mon[29192]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:24:47.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:47 smithi098 ceph-mon[29192]: Generating ssh key... 2024-04-25T17:24:47.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:47 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:47.834 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:47 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:48.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDitixWwEzjIanHWKHEIREwnhwxWLgffJkEk79ZMEaBm106qwF+2ZHM21Y8SfEpW6emkfVuz79bion9coU/R7Xai6Tf6fVErXSQOTrnWpvS89xFOrRGvy4H4ZhWbMrno3syHiLkgdp8Jf7A9tx6T6SdqvbOnmqhG7gRnNYLRLroRlot09Jwgr81L0RrGKNMYm+LgD1Ysirbrj4/cCVyQGGJBIys2sRY93noP+IFDU98RJ+021mXa24haoYRbygvxZ42ntVvGMJ1uR+IPfqTs2aZbVwCGXPiXNlTCbIGtb2C9DpWzujuf4UQqOAKKX88OvApdssgxwMSCfDBJLKvjuH2NrVGCd27h2wHGByPeRpdf6yPEqPT90l6VwbcNjtHwqcM51Zys4FmV60wBBF56kOmuKBGpulRB2+ar5MTLj2QFOp5pc+oC7OLlJWM/wrUxVMJ4dKS0KGKsqL/GpuhFpW3fmyAhxmIiJczyqfKrENU6+3836Rylu4ZNdV1dQ6FKb8= ceph-79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.040+0000 7faa50ecd700 1 Processor -- start 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.040+0000 7faa50ecd700 1 -- start start 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.040+0000 7faa50ecd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c108790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.040+0000 7faa50ecd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c108d60 con 0x7faa4c108370 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.040+0000 7faa4a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c108790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.040+0000 7faa4a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c108790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55034/0 (socket says 172.21.15.98:55034) 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.041+0000 7faa4a59c700 1 -- 172.21.15.98:0/1272476600 learned_addr learned my addr 172.21.15.98:0/1272476600 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.041+0000 7faa4a59c700 1 -- 172.21.15.98:0/1272476600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c109520 con 0x7faa4c108370 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.041+0000 7faa4a59c700 1 --2- 172.21.15.98:0/1272476600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c108790 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faa400098a0 tx=0x7faa40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26cee22215856eef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.041+0000 7faa4959a700 1 -- 172.21.15.98:0/1272476600 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa40004160 con 0x7faa4c108370 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.041+0000 7faa4959a700 1 -- 172.21.15.98:0/1272476600 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7faa40004300 con 0x7faa4c108370 2024-04-25T17:24:48.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.041+0000 7faa4959a700 1 -- 172.21.15.98:0/1272476600 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa40011af0 con 0x7faa4c108370 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.042+0000 7faa50ecd700 1 -- 172.21.15.98:0/1272476600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 msgr2=0x7faa4c108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.042+0000 7faa50ecd700 1 --2- 172.21.15.98:0/1272476600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c108790 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faa400098a0 tx=0x7faa40009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.042+0000 7faa50ecd700 1 -- 172.21.15.98:0/1272476600 shutdown_connections 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.042+0000 7faa50ecd700 1 --2- 172.21.15.98:0/1272476600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c108790 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.042+0000 7faa50ecd700 1 -- 172.21.15.98:0/1272476600 >> 172.21.15.98:0/1272476600 conn(0x7faa4c1038f0 msgr2=0x7faa4c105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.042+0000 7faa50ecd700 1 -- 172.21.15.98:0/1272476600 shutdown_connections 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.042+0000 7faa50ecd700 1 -- 172.21.15.98:0/1272476600 wait complete. 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa50ecd700 1 Processor -- start 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa50ecd700 1 -- start start 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa50ecd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c07db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa50ecd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c10c8d0 con 0x7faa4c108370 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa4a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c07db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa4a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c07db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55038/0 (socket says 172.21.15.98:55038) 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa4a59c700 1 -- 172.21.15.98:0/3854751374 learned_addr learned my addr 172.21.15.98:0/3854751374 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:48.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa4a59c700 1 -- 172.21.15.98:0/3854751374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c07e0a0 con 0x7faa4c108370 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa4a59c700 1 --2- 172.21.15.98:0/3854751374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c07db60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faa40003f40 tx=0x7faa40011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa3b7fe700 1 -- 172.21.15.98:0/3854751374 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa400038b0 con 0x7faa4c108370 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa3b7fe700 1 -- 172.21.15.98:0/3854751374 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7faa40003a50 con 0x7faa4c108370 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.043+0000 7faa50ecd700 1 -- 172.21.15.98:0/3854751374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c07c290 con 0x7faa4c108370 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.044+0000 7faa3b7fe700 1 -- 172.21.15.98:0/3854751374 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa4001a5c0 con 0x7faa4c108370 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.044+0000 7faa50ecd700 1 -- 172.21.15.98:0/3854751374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c07c7d0 con 0x7faa4c108370 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.045+0000 7faa3b7fe700 1 -- 172.21.15.98:0/3854751374 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7faa4001a8c0 con 0x7faa4c108370 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.045+0000 7faa50ecd700 1 -- 172.21.15.98:0/3854751374 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c0640a0 con 0x7faa4c108370 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.045+0000 7faa3b7fe700 1 --2- 172.21.15.98:0/3854751374 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa34039620 0x7faa3403bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.045+0000 7faa3b7fe700 1 -- 172.21.15.98:0/3854751374 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faa4004dc40 con 0x7faa4c108370 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.045+0000 7faa49d9b700 1 --2- 172.21.15.98:0/3854751374 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa34039620 0x7faa3403bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.045+0000 7faa49d9b700 1 --2- 172.21.15.98:0/3854751374 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa34039620 0x7faa3403bae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faa3c006fd0 tx=0x7faa3c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.050+0000 7faa3b7fe700 1 -- 172.21.15.98:0/3854751374 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa40019c70 con 0x7faa4c108370 2024-04-25T17:24:48.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.253+0000 7faa50ecd700 1 -- 172.21.15.98:0/3854751374 --> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7faa4c104af0 con 0x7faa34039620 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.254+0000 7faa3b7fe700 1 -- 172.21.15.98:0/3854751374 <== mgr.14120 v2:172.21.15.98:6800/3170662380 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7faa4c104af0 con 0x7faa34039620 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.256+0000 7faa50ecd700 1 -- 172.21.15.98:0/3854751374 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa34039620 msgr2=0x7faa3403bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.256+0000 7faa50ecd700 1 --2- 172.21.15.98:0/3854751374 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa34039620 0x7faa3403bae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7faa3c006fd0 tx=0x7faa3c006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.256+0000 7faa50ecd700 1 -- 172.21.15.98:0/3854751374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 msgr2=0x7faa4c07db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.256+0000 7faa50ecd700 1 --2- 172.21.15.98:0/3854751374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c07db60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faa40003f40 tx=0x7faa40011890 comp rx=0 tx=0).stop 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.256+0000 7faa50ecd700 1 -- 172.21.15.98:0/3854751374 shutdown_connections 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.256+0000 7faa50ecd700 1 --2- 172.21.15.98:0/3854751374 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa34039620 0x7faa3403bae0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.256+0000 7faa50ecd700 1 --2- 172.21.15.98:0/3854751374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa4c108370 0x7faa4c07db60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.256+0000 7faa50ecd700 1 -- 172.21.15.98:0/3854751374 >> 172.21.15.98:0/3854751374 conn(0x7faa4c1038f0 msgr2=0x7faa4c1043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.256+0000 7faa50ecd700 1 -- 172.21.15.98:0/3854751374 shutdown_connections 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:48.256+0000 7faa50ecd700 1 -- 172.21.15.98:0/3854751374 wait complete. 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T17:24:48.841 INFO:teuthology.orchestra.run.smithi098.stdout:Adding host smithi098... 2024-04-25T17:24:49.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:48 smithi098 ceph-mon[29192]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:24:50.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:49 smithi098 ceph-mon[29192]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:24:51.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:50 smithi098 ceph-mon[29192]: Deploying cephadm binary to smithi098 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Added host 'smithi098' with addr '172.21.15.98' 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.280+0000 7f67a65b6700 1 Processor -- start 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.280+0000 7f67a65b6700 1 -- start start 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.280+0000 7f67a65b6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a010b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.280+0000 7f67a65b6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f67a007bea0 con 0x7f67a0108df0 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.280+0000 7f679ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a010b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.280+0000 7f679ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a010b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55054/0 (socket says 172.21.15.98:55054) 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.281+0000 7f679ffff700 1 -- 172.21.15.98:0/3798414112 learned_addr learned my addr 172.21.15.98:0/3798414112 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.281+0000 7f679ffff700 1 -- 172.21.15.98:0/3798414112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a010b720 con 0x7f67a0108df0 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.281+0000 7f679ffff700 1 --2- 172.21.15.98:0/3798414112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a010b1e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f67900098a0 tx=0x7f6790009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cb6171d3a88f8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:54.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.281+0000 7f679effd700 1 -- 172.21.15.98:0/3798414112 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6790004160 con 0x7f67a0108df0 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.281+0000 7f679effd700 1 -- 172.21.15.98:0/3798414112 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6790004300 con 0x7f67a0108df0 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.281+0000 7f679effd700 1 -- 172.21.15.98:0/3798414112 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6790011af0 con 0x7f67a0108df0 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.282+0000 7f67a65b6700 1 -- 172.21.15.98:0/3798414112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 msgr2=0x7f67a010b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.282+0000 7f67a65b6700 1 --2- 172.21.15.98:0/3798414112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a010b1e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f67900098a0 tx=0x7f6790009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.282+0000 7f67a65b6700 1 -- 172.21.15.98:0/3798414112 shutdown_connections 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.282+0000 7f67a65b6700 1 --2- 172.21.15.98:0/3798414112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a010b1e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.282+0000 7f67a65b6700 1 -- 172.21.15.98:0/3798414112 >> 172.21.15.98:0/3798414112 conn(0x7f67a0102860 msgr2=0x7f67a0104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.282+0000 7f67a65b6700 1 -- 172.21.15.98:0/3798414112 shutdown_connections 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.282+0000 7f67a65b6700 1 -- 172.21.15.98:0/3798414112 wait complete. 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.282+0000 7f67a65b6700 1 Processor -- start 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.282+0000 7f67a65b6700 1 -- start start 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f67a65b6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a019c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f67a65b6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010c6f0 con 0x7f67a0108df0 2024-04-25T17:24:54.171 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f679ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a019c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f679ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a019c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55070/0 (socket says 172.21.15.98:55070) 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f679ffff700 1 -- 172.21.15.98:0/3995730827 learned_addr learned my addr 172.21.15.98:0/3995730827 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f679ffff700 1 -- 172.21.15.98:0/3995730827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a019c710 con 0x7f67a0108df0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f679ffff700 1 --2- 172.21.15.98:0/3995730827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a019c1d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6790003f40 tx=0x7f6790011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f679d7fa700 1 -- 172.21.15.98:0/3995730827 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67900038b0 con 0x7f67a0108df0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f67a65b6700 1 -- 172.21.15.98:0/3995730827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a019c9a0 con 0x7f67a0108df0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f679d7fa700 1 -- 172.21.15.98:0/3995730827 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6790003a50 con 0x7f67a0108df0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f679d7fa700 1 -- 172.21.15.98:0/3995730827 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f679001a5c0 con 0x7f67a0108df0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.283+0000 7f67a65b6700 1 -- 172.21.15.98:0/3995730827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a019f690 con 0x7f67a0108df0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.284+0000 7f679d7fa700 1 -- 172.21.15.98:0/3995730827 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f679001a8c0 con 0x7f67a0108df0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.284+0000 7f67a65b6700 1 -- 172.21.15.98:0/3995730827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a00640a0 con 0x7f67a0108df0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.284+0000 7f679d7fa700 1 --2- 172.21.15.98:0/3995730827 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f6788039620 0x7f678803bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.285+0000 7f679d7fa700 1 -- 172.21.15.98:0/3995730827 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f679004db00 con 0x7f67a0108df0 2024-04-25T17:24:54.172 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.285+0000 7f679f7fe700 1 --2- 172.21.15.98:0/3995730827 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f6788039620 0x7f678803bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.285+0000 7f679f7fe700 1 --2- 172.21.15.98:0/3995730827 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f6788039620 0x7f678803bae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6794006fd0 tx=0x7f6794006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.290+0000 7f679d7fa700 1 -- 172.21.15.98:0/3995730827 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6790019c70 con 0x7f67a0108df0 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:49.490+0000 7f67a65b6700 1 -- 172.21.15.98:0/3995730827 --> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}) -- 0x7f67a0103a10 con 0x7f6788039620 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.540+0000 7f679d7fa700 1 -- 172.21.15.98:0/3995730827 <== mgr.14120 v2:172.21.15.98:6800/3170662380 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f67a0103a10 con 0x7f6788039620 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.543+0000 7f67a65b6700 1 -- 172.21.15.98:0/3995730827 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f6788039620 msgr2=0x7f678803bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.543+0000 7f67a65b6700 1 --2- 172.21.15.98:0/3995730827 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f6788039620 0x7f678803bae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6794006fd0 tx=0x7f6794006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.543+0000 7f67a65b6700 1 -- 172.21.15.98:0/3995730827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 msgr2=0x7f67a019c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.543+0000 7f67a65b6700 1 --2- 172.21.15.98:0/3995730827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a019c1d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6790003f40 tx=0x7f6790011890 comp rx=0 tx=0).stop 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.543+0000 7f67a65b6700 1 -- 172.21.15.98:0/3995730827 shutdown_connections 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.543+0000 7f67a65b6700 1 --2- 172.21.15.98:0/3995730827 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f6788039620 0x7f678803bae0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.543+0000 7f67a65b6700 1 --2- 172.21.15.98:0/3995730827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0108df0 0x7f67a019c1d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.543+0000 7f67a65b6700 1 -- 172.21.15.98:0/3995730827 >> 172.21.15.98:0/3995730827 conn(0x7f67a0102860 msgr2=0x7f67a0103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.544+0000 7f67a65b6700 1 -- 172.21.15.98:0/3995730827 shutdown_connections 2024-04-25T17:24:54.173 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:53.544+0000 7f67a65b6700 1 -- 172.21.15.98:0/3995730827 wait complete. 2024-04-25T17:24:54.174 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mon service... 2024-04-25T17:24:54.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:54 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:54.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:54 smithi098 ceph-mon[29192]: Added host smithi098 2024-04-25T17:24:54.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:54 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.601+0000 7f45b8051700 1 Processor -- start 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.601+0000 7f45b8051700 1 -- start start 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.602+0000 7f45b8051700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b007c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.602+0000 7f45b8051700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45b007c880 con 0x7f45b007de50 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.602+0000 7f45b5ded700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b007c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.602+0000 7f45b5ded700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b007c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54434/0 (socket says 172.21.15.98:54434) 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.602+0000 7f45b5ded700 1 -- 172.21.15.98:0/3820436631 learned_addr learned my addr 172.21.15.98:0/3820436631 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.602+0000 7f45b5ded700 1 -- 172.21.15.98:0/3820436631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b007ca00 con 0x7f45b007de50 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.602+0000 7f45b5ded700 1 --2- 172.21.15.98:0/3820436631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b007c2b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009e80 tx=0x7f45a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa80a93e3fd8f60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:55.673 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.602+0000 7f45b4deb700 1 -- 172.21.15.98:0/3820436631 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45a0017070 con 0x7f45b007de50 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.602+0000 7f45b4deb700 1 -- 172.21.15.98:0/3820436631 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f45a000b8c0 con 0x7f45b007de50 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.602+0000 7f45b4deb700 1 -- 172.21.15.98:0/3820436631 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45a0004490 con 0x7f45b007de50 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.603+0000 7f45b8051700 1 -- 172.21.15.98:0/3820436631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 msgr2=0x7f45b007c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.603+0000 7f45b8051700 1 --2- 172.21.15.98:0/3820436631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b007c2b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009e80 tx=0x7f45a000b330 comp rx=0 tx=0).stop 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.603+0000 7f45b8051700 1 -- 172.21.15.98:0/3820436631 shutdown_connections 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.603+0000 7f45b8051700 1 --2- 172.21.15.98:0/3820436631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b007c2b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.603+0000 7f45b8051700 1 -- 172.21.15.98:0/3820436631 >> 172.21.15.98:0/3820436631 conn(0x7f45b0103990 msgr2=0x7f45b0105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.603+0000 7f45b8051700 1 -- 172.21.15.98:0/3820436631 shutdown_connections 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.603+0000 7f45b8051700 1 -- 172.21.15.98:0/3820436631 wait complete. 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.603+0000 7f45b8051700 1 Processor -- start 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45b8051700 1 -- start start 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45b8051700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45b8051700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45b010e9a0 con 0x7f45b007de50 2024-04-25T17:24:55.674 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45b5ded700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45b5ded700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54448/0 (socket says 172.21.15.98:54448) 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45b5ded700 1 -- 172.21.15.98:0/864957276 learned_addr learned my addr 172.21.15.98:0/864957276 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45b5ded700 1 -- 172.21.15.98:0/864957276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b019e970 con 0x7f45b007de50 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45b5ded700 1 --2- 172.21.15.98:0/864957276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b019e430 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f45a000b7b0 tx=0x7f45a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45aeffd700 1 -- 172.21.15.98:0/864957276 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45a0020020 con 0x7f45b007de50 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45b8051700 1 -- 172.21.15.98:0/864957276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b019ec00 con 0x7f45b007de50 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45aeffd700 1 -- 172.21.15.98:0/864957276 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f45a0012430 con 0x7f45b007de50 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45aeffd700 1 -- 172.21.15.98:0/864957276 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45a0017040 con 0x7f45b007de50 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.604+0000 7f45b8051700 1 -- 172.21.15.98:0/864957276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b01a18f0 con 0x7f45b007de50 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.605+0000 7f45aeffd700 1 -- 172.21.15.98:0/864957276 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f45a0023020 con 0x7f45b007de50 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.605+0000 7f45b8051700 1 -- 172.21.15.98:0/864957276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b00640a0 con 0x7f45b007de50 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.605+0000 7f45aeffd700 1 --2- 172.21.15.98:0/864957276 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f4598039620 0x7f459803bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.605+0000 7f45aeffd700 1 -- 172.21.15.98:0/864957276 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f45a000f050 con 0x7f45b007de50 2024-04-25T17:24:55.675 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.606+0000 7f45b55ec700 1 --2- 172.21.15.98:0/864957276 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f4598039620 0x7f459803bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.606+0000 7f45b55ec700 1 --2- 172.21.15.98:0/864957276 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f4598039620 0x7f459803bae0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f45a4006fd0 tx=0x7f45a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.611+0000 7f45aeffd700 1 -- 172.21.15.98:0/864957276 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45a002a020 con 0x7f45b007de50 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.825+0000 7f45b8051700 1 -- 172.21.15.98:0/864957276 --> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f45b0104b20 con 0x7f4598039620 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.832+0000 7f45aeffd700 1 -- 172.21.15.98:0/864957276 <== mgr.14120 v2:172.21.15.98:6800/3170662380 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f45b0104b20 con 0x7f4598039620 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.835+0000 7f45b8051700 1 -- 172.21.15.98:0/864957276 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f4598039620 msgr2=0x7f459803bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.835+0000 7f45b8051700 1 --2- 172.21.15.98:0/864957276 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f4598039620 0x7f459803bae0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f45a4006fd0 tx=0x7f45a4006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.835+0000 7f45b8051700 1 -- 172.21.15.98:0/864957276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 msgr2=0x7f45b019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.835+0000 7f45b8051700 1 --2- 172.21.15.98:0/864957276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b019e430 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f45a000b7b0 tx=0x7f45a000be20 comp rx=0 tx=0).stop 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.835+0000 7f45b8051700 1 -- 172.21.15.98:0/864957276 shutdown_connections 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.835+0000 7f45b8051700 1 --2- 172.21.15.98:0/864957276 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f4598039620 0x7f459803bae0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.835+0000 7f45b8051700 1 --2- 172.21.15.98:0/864957276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45b007de50 0x7f45b019e430 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.835+0000 7f45b8051700 1 -- 172.21.15.98:0/864957276 >> 172.21.15.98:0/864957276 conn(0x7f45b0103990 msgr2=0x7f45b0104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.835+0000 7f45b8051700 1 -- 172.21.15.98:0/864957276 shutdown_connections 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:54.835+0000 7f45b8051700 1 -- 172.21.15.98:0/864957276 wait complete. 2024-04-25T17:24:55.676 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mgr service... 2024-04-25T17:24:55.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:55 smithi098 ceph-mon[29192]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:24:55.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:55 smithi098 ceph-mon[29192]: Saving service mon spec with placement count:5 2024-04-25T17:24:55.924 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:55 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:57.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T17:24:57.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.133+0000 7fca01bb3700 1 Processor -- start 2024-04-25T17:24:57.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.133+0000 7fca01bb3700 1 -- start start 2024-04-25T17:24:57.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.134+0000 7fca01bb3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc1087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:57.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.134+0000 7fca01bb3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9fc108d70 con 0x7fc9fc108380 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.134+0000 7fc9fb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc1087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.134+0000 7fc9fb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc1087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54450/0 (socket says 172.21.15.98:54450) 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.134+0000 7fc9fb7fe700 1 -- 172.21.15.98:0/2393749950 learned_addr learned my addr 172.21.15.98:0/2393749950 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.134+0000 7fc9fb7fe700 1 -- 172.21.15.98:0/2393749950 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9fc109530 con 0x7fc9fc108380 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.134+0000 7fc9fb7fe700 1 --2- 172.21.15.98:0/2393749950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc1087a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9f00098a0 tx=0x7fc9f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f045cfd22d8a514c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.134+0000 7fc9fa7fc700 1 -- 172.21.15.98:0/2393749950 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9f0004160 con 0x7fc9fc108380 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.134+0000 7fc9fa7fc700 1 -- 172.21.15.98:0/2393749950 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc9f0004300 con 0x7fc9fc108380 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.134+0000 7fc9fa7fc700 1 -- 172.21.15.98:0/2393749950 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9f0011af0 con 0x7fc9fc108380 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.135+0000 7fca01bb3700 1 -- 172.21.15.98:0/2393749950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 msgr2=0x7fc9fc1087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.135+0000 7fca01bb3700 1 --2- 172.21.15.98:0/2393749950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc1087a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9f00098a0 tx=0x7fc9f0009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.135+0000 7fca01bb3700 1 -- 172.21.15.98:0/2393749950 shutdown_connections 2024-04-25T17:24:57.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.135+0000 7fca01bb3700 1 --2- 172.21.15.98:0/2393749950 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc1087a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.135+0000 7fca01bb3700 1 -- 172.21.15.98:0/2393749950 >> 172.21.15.98:0/2393749950 conn(0x7fc9fc103920 msgr2=0x7fc9fc105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.135+0000 7fca01bb3700 1 -- 172.21.15.98:0/2393749950 shutdown_connections 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.135+0000 7fca01bb3700 1 -- 172.21.15.98:0/2393749950 wait complete. 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.136+0000 7fca01bb3700 1 Processor -- start 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.136+0000 7fca01bb3700 1 -- start start 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.136+0000 7fca01bb3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc19c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.136+0000 7fca01bb3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9fc10c8e0 con 0x7fc9fc108380 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.136+0000 7fc9fb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc19c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.136+0000 7fc9fb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc19c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54466/0 (socket says 172.21.15.98:54466) 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.136+0000 7fc9fb7fe700 1 -- 172.21.15.98:0/1386458760 learned_addr learned my addr 172.21.15.98:0/1386458760 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.136+0000 7fc9fb7fe700 1 -- 172.21.15.98:0/1386458760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9fc19c710 con 0x7fc9fc108380 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.137+0000 7fc9fb7fe700 1 --2- 172.21.15.98:0/1386458760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc19c1d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0003f40 tx=0x7fc9f0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.137+0000 7fc9f8ff9700 1 -- 172.21.15.98:0/1386458760 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9f00038b0 con 0x7fc9fc108380 2024-04-25T17:24:57.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.137+0000 7fc9f8ff9700 1 -- 172.21.15.98:0/1386458760 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc9f0003a50 con 0x7fc9fc108380 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.137+0000 7fca01bb3700 1 -- 172.21.15.98:0/1386458760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9fc19c9a0 con 0x7fc9fc108380 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.137+0000 7fc9f8ff9700 1 -- 172.21.15.98:0/1386458760 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9f001a5c0 con 0x7fc9fc108380 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.137+0000 7fca01bb3700 1 -- 172.21.15.98:0/1386458760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9fc19f690 con 0x7fc9fc108380 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.138+0000 7fc9f8ff9700 1 -- 172.21.15.98:0/1386458760 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc9f001a8c0 con 0x7fc9fc108380 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.138+0000 7fca01bb3700 1 -- 172.21.15.98:0/1386458760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9fc0640a0 con 0x7fc9fc108380 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.138+0000 7fc9f8ff9700 1 --2- 172.21.15.98:0/1386458760 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fc9e4039620 0x7fc9e403bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.138+0000 7fc9f8ff9700 1 -- 172.21.15.98:0/1386458760 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc9f004db10 con 0x7fc9fc108380 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.138+0000 7fc9faffd700 1 --2- 172.21.15.98:0/1386458760 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fc9e4039620 0x7fc9e403bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.138+0000 7fc9faffd700 1 --2- 172.21.15.98:0/1386458760 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fc9e4039620 0x7fc9e403bae0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec006fd0 tx=0x7fc9ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.143+0000 7fc9f8ff9700 1 -- 172.21.15.98:0/1386458760 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9f0019c70 con 0x7fc9fc108380 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.361+0000 7fca01bb3700 1 -- 172.21.15.98:0/1386458760 --> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc9fc104ad0 con 0x7fc9e4039620 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.365+0000 7fc9f8ff9700 1 -- 172.21.15.98:0/1386458760 <== mgr.14120 v2:172.21.15.98:6800/3170662380 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc9fc104ad0 con 0x7fc9e4039620 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.367+0000 7fca01bb3700 1 -- 172.21.15.98:0/1386458760 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fc9e4039620 msgr2=0x7fc9e403bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.367+0000 7fca01bb3700 1 --2- 172.21.15.98:0/1386458760 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fc9e4039620 0x7fc9e403bae0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec006fd0 tx=0x7fc9ec006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:57.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.367+0000 7fca01bb3700 1 -- 172.21.15.98:0/1386458760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 msgr2=0x7fc9fc19c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:57.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.367+0000 7fca01bb3700 1 --2- 172.21.15.98:0/1386458760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc19c1d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0003f40 tx=0x7fc9f0011890 comp rx=0 tx=0).stop 2024-04-25T17:24:57.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.368+0000 7fca01bb3700 1 -- 172.21.15.98:0/1386458760 shutdown_connections 2024-04-25T17:24:57.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.368+0000 7fca01bb3700 1 --2- 172.21.15.98:0/1386458760 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fc9e4039620 0x7fc9e403bae0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:57.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.368+0000 7fca01bb3700 1 --2- 172.21.15.98:0/1386458760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9fc108380 0x7fc9fc19c1d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:57.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.368+0000 7fca01bb3700 1 -- 172.21.15.98:0/1386458760 >> 172.21.15.98:0/1386458760 conn(0x7fc9fc103920 msgr2=0x7fc9fc1043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:57.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.368+0000 7fca01bb3700 1 -- 172.21.15.98:0/1386458760 shutdown_connections 2024-04-25T17:24:57.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:56.368+0000 7fca01bb3700 1 -- 172.21.15.98:0/1386458760 wait complete. 2024-04-25T17:24:57.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:57 smithi098 ceph-mon[29192]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:24:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:57 smithi098 ceph-mon[29192]: Saving service mgr spec with placement count:2 2024-04-25T17:24:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:57 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:57 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:58.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:58 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:24:58.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:58 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2193177845' entity='client.admin' 2024-04-25T17:24:59.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.872+0000 7f01b4b28700 1 Processor -- start 2024-04-25T17:24:59.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.872+0000 7f01b4b28700 1 -- start start 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.872+0000 7f01b4b28700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b0108770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.873+0000 7f01b4b28700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01b0108d40 con 0x7f01b0108350 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.873+0000 7f01ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b0108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.873+0000 7f01ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b0108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54472/0 (socket says 172.21.15.98:54472) 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.873+0000 7f01ae59c700 1 -- 172.21.15.98:0/635812350 learned_addr learned my addr 172.21.15.98:0/635812350 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.873+0000 7f01ae59c700 1 -- 172.21.15.98:0/635812350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b0109550 con 0x7f01b0108350 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.873+0000 7f01ae59c700 1 --2- 172.21.15.98:0/635812350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b0108770 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f01a40098a0 tx=0x7f01a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f29e4e6523f75ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.873+0000 7f01ad59a700 1 -- 172.21.15.98:0/635812350 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01a4004160 con 0x7f01b0108350 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.873+0000 7f01ad59a700 1 -- 172.21.15.98:0/635812350 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01a4004300 con 0x7f01b0108350 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.873+0000 7f01ad59a700 1 -- 172.21.15.98:0/635812350 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01a4011a20 con 0x7f01b0108350 2024-04-25T17:24:59.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.874+0000 7f01b4b28700 1 -- 172.21.15.98:0/635812350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 msgr2=0x7f01b0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.874+0000 7f01b4b28700 1 --2- 172.21.15.98:0/635812350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b0108770 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f01a40098a0 tx=0x7f01a4009bb0 comp rx=0 tx=0).stop 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.874+0000 7f01b4b28700 1 -- 172.21.15.98:0/635812350 shutdown_connections 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.874+0000 7f01b4b28700 1 --2- 172.21.15.98:0/635812350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b0108770 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.874+0000 7f01b4b28700 1 -- 172.21.15.98:0/635812350 >> 172.21.15.98:0/635812350 conn(0x7f01b01039d0 msgr2=0x7f01b0105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.874+0000 7f01b4b28700 1 -- 172.21.15.98:0/635812350 shutdown_connections 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.874+0000 7f01b4b28700 1 -- 172.21.15.98:0/635812350 wait complete. 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.875+0000 7f01b4b28700 1 Processor -- start 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.875+0000 7f01b4b28700 1 -- start start 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.875+0000 7f01b4b28700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b019c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.875+0000 7f01b4b28700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01b010c900 con 0x7f01b0108350 2024-04-25T17:24:59.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.875+0000 7f01ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b019c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.875+0000 7f01ae59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b019c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54488/0 (socket says 172.21.15.98:54488) 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.875+0000 7f01ae59c700 1 -- 172.21.15.98:0/2193177845 learned_addr learned my addr 172.21.15.98:0/2193177845 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.876+0000 7f01ae59c700 1 -- 172.21.15.98:0/2193177845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b019c6d0 con 0x7f01b0108350 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.876+0000 7f01ae59c700 1 --2- 172.21.15.98:0/2193177845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b019c190 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f01a40116e0 tx=0x7f01a4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.876+0000 7f019f7fe700 1 -- 172.21.15.98:0/2193177845 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01a40039e0 con 0x7f01b0108350 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.876+0000 7f01b4b28700 1 -- 172.21.15.98:0/2193177845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01b019c960 con 0x7f01b0108350 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.876+0000 7f019f7fe700 1 -- 172.21.15.98:0/2193177845 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01a401b8f0 con 0x7f01b0108350 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.876+0000 7f019f7fe700 1 -- 172.21.15.98:0/2193177845 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01a4023760 con 0x7f01b0108350 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.876+0000 7f01b4b28700 1 -- 172.21.15.98:0/2193177845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01b019f650 con 0x7f01b0108350 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.877+0000 7f01b4b28700 1 -- 172.21.15.98:0/2193177845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01b00640a0 con 0x7f01b0108350 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.877+0000 7f019f7fe700 1 -- 172.21.15.98:0/2193177845 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f01a40239e0 con 0x7f01b0108350 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.877+0000 7f019f7fe700 1 --2- 172.21.15.98:0/2193177845 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f0198039670 0x7f019803bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.877+0000 7f019f7fe700 1 -- 172.21.15.98:0/2193177845 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f01a404deb0 con 0x7f01b0108350 2024-04-25T17:24:59.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.878+0000 7f01add9b700 1 --2- 172.21.15.98:0/2193177845 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f0198039670 0x7f019803bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.878+0000 7f01add9b700 1 --2- 172.21.15.98:0/2193177845 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f0198039670 0x7f019803bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f01a0006fd0 tx=0x7f01a0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:57.882+0000 7f019f7fe700 1 -- 172.21.15.98:0/2193177845 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f01a401f030 con 0x7f01b0108350 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.100+0000 7f01b4b28700 1 -- 172.21.15.98:0/2193177845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f01b019fc20 con 0x7f01b0108350 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.103+0000 7f019f7fe700 1 -- 172.21.15.98:0/2193177845 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f01a4018e30 con 0x7f01b0108350 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.106+0000 7f01b4b28700 1 -- 172.21.15.98:0/2193177845 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f0198039670 msgr2=0x7f019803bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.106+0000 7f01b4b28700 1 --2- 172.21.15.98:0/2193177845 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f0198039670 0x7f019803bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f01a0006fd0 tx=0x7f01a0006e40 comp rx=0 tx=0).stop 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.106+0000 7f01b4b28700 1 -- 172.21.15.98:0/2193177845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 msgr2=0x7f01b019c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.106+0000 7f01b4b28700 1 --2- 172.21.15.98:0/2193177845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b019c190 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f01a40116e0 tx=0x7f01a4011aa0 comp rx=0 tx=0).stop 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.106+0000 7f01b4b28700 1 -- 172.21.15.98:0/2193177845 shutdown_connections 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.106+0000 7f01b4b28700 1 --2- 172.21.15.98:0/2193177845 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f0198039670 0x7f019803bb30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.106+0000 7f01b4b28700 1 --2- 172.21.15.98:0/2193177845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01b0108350 0x7f01b019c190 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.106+0000 7f01b4b28700 1 -- 172.21.15.98:0/2193177845 >> 172.21.15.98:0/2193177845 conn(0x7f01b01039d0 msgr2=0x7f01b0104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:24:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.106+0000 7f01b4b28700 1 -- 172.21.15.98:0/2193177845 shutdown_connections 2024-04-25T17:24:59.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:58.106+0000 7f01b4b28700 1 -- 172.21.15.98:0/2193177845 wait complete. 2024-04-25T17:24:59.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:24:59 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:25:00.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.817+0000 7f85c6d28700 1 Processor -- start 2024-04-25T17:25:00.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.817+0000 7f85c6d28700 1 -- start start 2024-04-25T17:25:00.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.817+0000 7f85c6d28700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c0108c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:00.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.817+0000 7f85c6d28700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85c0109220 con 0x7f85c0108830 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.817+0000 7f85c4ac4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c0108c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.817+0000 7f85c4ac4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c0108c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54496/0 (socket says 172.21.15.98:54496) 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.817+0000 7f85c4ac4700 1 -- 172.21.15.98:0/4203958329 learned_addr learned my addr 172.21.15.98:0/4203958329 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.818+0000 7f85c4ac4700 1 -- 172.21.15.98:0/4203958329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c01074e0 con 0x7f85c0108830 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.818+0000 7f85c4ac4700 1 --2- 172.21.15.98:0/4203958329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c0108c50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f85b40098a0 tx=0x7f85b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c372605008816c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.818+0000 7f85bf7fe700 1 -- 172.21.15.98:0/4203958329 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85b4004160 con 0x7f85c0108830 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.818+0000 7f85bf7fe700 1 -- 172.21.15.98:0/4203958329 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85b4004300 con 0x7f85c0108830 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.818+0000 7f85bf7fe700 1 -- 172.21.15.98:0/4203958329 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85b4011a20 con 0x7f85c0108830 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.819+0000 7f85c6d28700 1 -- 172.21.15.98:0/4203958329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 msgr2=0x7f85c0108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.819+0000 7f85c6d28700 1 --2- 172.21.15.98:0/4203958329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c0108c50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f85b40098a0 tx=0x7f85b4009bb0 comp rx=0 tx=0).stop 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.819+0000 7f85c6d28700 1 -- 172.21.15.98:0/4203958329 shutdown_connections 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.819+0000 7f85c6d28700 1 --2- 172.21.15.98:0/4203958329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c0108c50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:00.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.819+0000 7f85c6d28700 1 -- 172.21.15.98:0/4203958329 >> 172.21.15.98:0/4203958329 conn(0x7f85c0101fb0 msgr2=0x7f85c01043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.819+0000 7f85c6d28700 1 -- 172.21.15.98:0/4203958329 shutdown_connections 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.819+0000 7f85c6d28700 1 -- 172.21.15.98:0/4203958329 wait complete. 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.819+0000 7f85c6d28700 1 Processor -- start 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.819+0000 7f85c6d28700 1 -- start start 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85c6d28700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c019b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85c6d28700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85c010b8a0 con 0x7f85c0108830 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85c4ac4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c019b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85c4ac4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c019b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54512/0 (socket says 172.21.15.98:54512) 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85c4ac4700 1 -- 172.21.15.98:0/2983501706 learned_addr learned my addr 172.21.15.98:0/2983501706 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85c4ac4700 1 -- 172.21.15.98:0/2983501706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c019b5d0 con 0x7f85c0108830 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85c4ac4700 1 --2- 172.21.15.98:0/2983501706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c019b090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85b40116e0 tx=0x7f85b4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85bdffb700 1 -- 172.21.15.98:0/2983501706 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85b40039e0 con 0x7f85c0108830 2024-04-25T17:25:00.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85bdffb700 1 -- 172.21.15.98:0/2983501706 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85b401b8f0 con 0x7f85c0108830 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85c6d28700 1 -- 172.21.15.98:0/2983501706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c019b860 con 0x7f85c0108830 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85bdffb700 1 -- 172.21.15.98:0/2983501706 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85b4023760 con 0x7f85c0108830 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.820+0000 7f85c6d28700 1 -- 172.21.15.98:0/2983501706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c019e550 con 0x7f85c0108830 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.821+0000 7f85bdffb700 1 -- 172.21.15.98:0/2983501706 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f85b4023900 con 0x7f85c0108830 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.821+0000 7f85c6d28700 1 -- 172.21.15.98:0/2983501706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85c00640a0 con 0x7f85c0108830 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.822+0000 7f85bdffb700 1 --2- 172.21.15.98:0/2983501706 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f85b0039670 0x7f85b003bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.822+0000 7f85bdffb700 1 -- 172.21.15.98:0/2983501706 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f85b404e250 con 0x7f85c0108830 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.822+0000 7f85bffff700 1 --2- 172.21.15.98:0/2983501706 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f85b0039670 0x7f85b003bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.822+0000 7f85bffff700 1 --2- 172.21.15.98:0/2983501706 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f85b0039670 0x7f85b003bb30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f85ac006fd0 tx=0x7f85ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:24:59.827+0000 7f85bdffb700 1 -- 172.21.15.98:0/2983501706 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f85b401f030 con 0x7f85c0108830 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.036+0000 7f85c6d28700 1 -- 172.21.15.98:0/2983501706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f85c019eb70 con 0x7f85c0108830 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.039+0000 7f85bdffb700 1 -- 172.21.15.98:0/2983501706 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f85b401bd50 con 0x7f85c0108830 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.041+0000 7f85c6d28700 1 -- 172.21.15.98:0/2983501706 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f85b0039670 msgr2=0x7f85b003bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:00.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.041+0000 7f85c6d28700 1 --2- 172.21.15.98:0/2983501706 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f85b0039670 0x7f85b003bb30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f85ac006fd0 tx=0x7f85ac006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:00.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.041+0000 7f85c6d28700 1 -- 172.21.15.98:0/2983501706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 msgr2=0x7f85c019b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:00.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.041+0000 7f85c6d28700 1 --2- 172.21.15.98:0/2983501706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c019b090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85b40116e0 tx=0x7f85b4011aa0 comp rx=0 tx=0).stop 2024-04-25T17:25:00.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.041+0000 7f85c6d28700 1 -- 172.21.15.98:0/2983501706 shutdown_connections 2024-04-25T17:25:00.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.041+0000 7f85c6d28700 1 --2- 172.21.15.98:0/2983501706 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7f85b0039670 0x7f85b003bb30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:00.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.041+0000 7f85c6d28700 1 --2- 172.21.15.98:0/2983501706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85c0108830 0x7f85c019b090 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:00.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.041+0000 7f85c6d28700 1 -- 172.21.15.98:0/2983501706 >> 172.21.15.98:0/2983501706 conn(0x7f85c0101fb0 msgr2=0x7f85c0102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:00.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.041+0000 7f85c6d28700 1 -- 172.21.15.98:0/2983501706 shutdown_connections 2024-04-25T17:25:00.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:00.041+0000 7f85c6d28700 1 -- 172.21.15.98:0/2983501706 wait complete. 2024-04-25T17:25:00.604 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling the dashboard module... 2024-04-25T17:25:01.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:01 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2983501706' entity='client.admin' 2024-04-25T17:25:01.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:01 smithi098 ceph-mon[29192]: from='mgr.14120 172.21.15.98:0/3898727255' entity='mgr.x' 2024-04-25T17:25:02.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:02 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2561513481' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T17:25:02.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:02 smithi098 ceph-mon[29192]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:02.410 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: ignoring --setuser ceph since I am not root 2024-04-25T17:25:02.410 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: ignoring --setgroup ceph since I am not root 2024-04-25T17:25:02.410 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:02.224+0000 7f0e0d082200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T17:25:02.410 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:02.407+0000 7f0e0d082200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:25:02.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:02.511+0000 7f0e0d082200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T17:25:02.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.035+0000 7feb417c5700 1 Processor -- start 2024-04-25T17:25:02.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.035+0000 7feb417c5700 1 -- start start 2024-04-25T17:25:02.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.035+0000 7feb417c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c07c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:02.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.035+0000 7feb417c5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c07c8e0 con 0x7feb3c07de60 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.035+0000 7feb3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c07c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.035+0000 7feb3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c07c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54518/0 (socket says 172.21.15.98:54518) 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.035+0000 7feb3affd700 1 -- 172.21.15.98:0/314284913 learned_addr learned my addr 172.21.15.98:0/314284913 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.035+0000 7feb3affd700 1 -- 172.21.15.98:0/314284913 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3c07ca60 con 0x7feb3c07de60 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.035+0000 7feb3affd700 1 --2- 172.21.15.98:0/314284913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c07c310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0098a0 tx=0x7feb2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc49672a9d05f3d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.036+0000 7feb39ffb700 1 -- 172.21.15.98:0/314284913 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb2c004160 con 0x7feb3c07de60 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.036+0000 7feb39ffb700 1 -- 172.21.15.98:0/314284913 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb2c004300 con 0x7feb3c07de60 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.036+0000 7feb39ffb700 1 -- 172.21.15.98:0/314284913 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb2c011a20 con 0x7feb3c07de60 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.036+0000 7feb417c5700 1 -- 172.21.15.98:0/314284913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 msgr2=0x7feb3c07c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.036+0000 7feb417c5700 1 --2- 172.21.15.98:0/314284913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c07c310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0098a0 tx=0x7feb2c009bb0 comp rx=0 tx=0).stop 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.036+0000 7feb417c5700 1 -- 172.21.15.98:0/314284913 shutdown_connections 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.036+0000 7feb417c5700 1 --2- 172.21.15.98:0/314284913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c07c310 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.036+0000 7feb417c5700 1 -- 172.21.15.98:0/314284913 >> 172.21.15.98:0/314284913 conn(0x7feb3c1039b0 msgr2=0x7feb3c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:02.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.036+0000 7feb417c5700 1 -- 172.21.15.98:0/314284913 shutdown_connections 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.036+0000 7feb417c5700 1 -- 172.21.15.98:0/314284913 wait complete. 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.037+0000 7feb417c5700 1 Processor -- start 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.037+0000 7feb417c5700 1 -- start start 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.037+0000 7feb417c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c19e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.037+0000 7feb417c5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c10e930 con 0x7feb3c07de60 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.037+0000 7feb3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c19e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.037+0000 7feb3affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c19e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54524/0 (socket says 172.21.15.98:54524) 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.037+0000 7feb3affd700 1 -- 172.21.15.98:0/2561513481 learned_addr learned my addr 172.21.15.98:0/2561513481 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.038+0000 7feb3affd700 1 -- 172.21.15.98:0/2561513481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3c19e8b0 con 0x7feb3c07de60 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.038+0000 7feb3affd700 1 --2- 172.21.15.98:0/2561513481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c19e370 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb2c000c00 tx=0x7feb2c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.042+0000 7feb23fff700 1 -- 172.21.15.98:0/2561513481 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb2c003aa0 con 0x7feb3c07de60 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.042+0000 7feb23fff700 1 -- 172.21.15.98:0/2561513481 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb2c01b8f0 con 0x7feb3c07de60 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.042+0000 7feb23fff700 1 -- 172.21.15.98:0/2561513481 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb2c023870 con 0x7feb3c07de60 2024-04-25T17:25:02.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.042+0000 7feb417c5700 1 -- 172.21.15.98:0/2561513481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb3c19eb40 con 0x7feb3c07de60 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.042+0000 7feb417c5700 1 -- 172.21.15.98:0/2561513481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb3c1a1830 con 0x7feb3c07de60 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.043+0000 7feb23fff700 1 -- 172.21.15.98:0/2561513481 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7feb2c023b20 con 0x7feb3c07de60 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.043+0000 7feb417c5700 1 -- 172.21.15.98:0/2561513481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb3c0640a0 con 0x7feb3c07de60 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.043+0000 7feb23fff700 1 --2- 172.21.15.98:0/2561513481 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7feb240396c0 0x7feb2403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.043+0000 7feb23fff700 1 -- 172.21.15.98:0/2561513481 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7feb2c04de30 con 0x7feb3c07de60 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.043+0000 7feb3a7fc700 1 --2- 172.21.15.98:0/2561513481 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7feb240396c0 0x7feb2403bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.044+0000 7feb3a7fc700 1 --2- 172.21.15.98:0/2561513481 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7feb240396c0 0x7feb2403bb80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feb30006fd0 tx=0x7feb30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.048+0000 7feb23fff700 1 -- 172.21.15.98:0/2561513481 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb2c01f030 con 0x7feb3c07de60 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:01.324+0000 7feb417c5700 1 -- 172.21.15.98:0/2561513481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7feb3c1a1e20 con 0x7feb3c07de60 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.049+0000 7feb23fff700 1 -- 172.21.15.98:0/2561513481 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7feb2c01baa0 con 0x7feb3c07de60 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.051+0000 7feb23fff700 1 -- 172.21.15.98:0/2561513481 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7feb2c02c720 con 0x7feb3c07de60 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.053+0000 7feb417c5700 1 -- 172.21.15.98:0/2561513481 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7feb240396c0 msgr2=0x7feb2403bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.053+0000 7feb417c5700 1 --2- 172.21.15.98:0/2561513481 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7feb240396c0 0x7feb2403bb80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feb30006fd0 tx=0x7feb30006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:02.733 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.053+0000 7feb417c5700 1 -- 172.21.15.98:0/2561513481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 msgr2=0x7feb3c19e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:02.734 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.053+0000 7feb417c5700 1 --2- 172.21.15.98:0/2561513481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c19e370 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb2c000c00 tx=0x7feb2c011aa0 comp rx=0 tx=0).stop 2024-04-25T17:25:02.734 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.053+0000 7feb417c5700 1 -- 172.21.15.98:0/2561513481 shutdown_connections 2024-04-25T17:25:02.734 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.053+0000 7feb417c5700 1 --2- 172.21.15.98:0/2561513481 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7feb240396c0 0x7feb2403bb80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:02.734 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.053+0000 7feb417c5700 1 --2- 172.21.15.98:0/2561513481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb3c07de60 0x7feb3c19e370 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:02.734 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.053+0000 7feb417c5700 1 -- 172.21.15.98:0/2561513481 >> 172.21.15.98:0/2561513481 conn(0x7feb3c1039b0 msgr2=0x7feb3c1043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:02.734 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.053+0000 7feb417c5700 1 -- 172.21.15.98:0/2561513481 shutdown_connections 2024-04-25T17:25:02.734 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:02.053+0000 7feb417c5700 1 -- 172.21.15.98:0/2561513481 wait complete. 2024-04-25T17:25:02.997 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:02.826+0000 7f0e0d082200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T17:25:02.998 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:02.913+0000 7f0e0d082200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T17:25:03.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:03 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2561513481' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T17:25:03.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:03 smithi098 ceph-mon[29192]: mgrmap e9: x(active, since 20s) 2024-04-25T17:25:04.056 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T17:25:04.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T17:25:04.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T17:25:04.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T17:25:04.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.193+0000 7fdca0b4f700 1 Processor -- start 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.193+0000 7fdca0b4f700 1 -- start start 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.193+0000 7fdca0b4f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c108790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.193+0000 7fdca0b4f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9c108d60 con 0x7fdc9c108370 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.193+0000 7fdc9a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c108790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.193+0000 7fdc9a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c108790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54554/0 (socket says 172.21.15.98:54554) 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.193+0000 7fdc9a59c700 1 -- 172.21.15.98:0/3083369262 learned_addr learned my addr 172.21.15.98:0/3083369262 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.194+0000 7fdc9a59c700 1 -- 172.21.15.98:0/3083369262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9c109520 con 0x7fdc9c108370 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.194+0000 7fdc9a59c700 1 --2- 172.21.15.98:0/3083369262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c108790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0098a0 tx=0x7fdc8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239d0c12a8939cad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.194+0000 7fdc9959a700 1 -- 172.21.15.98:0/3083369262 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c004160 con 0x7fdc9c108370 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.194+0000 7fdc9959a700 1 -- 172.21.15.98:0/3083369262 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc8c004300 con 0x7fdc9c108370 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.194+0000 7fdc9959a700 1 -- 172.21.15.98:0/3083369262 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c011a20 con 0x7fdc9c108370 2024-04-25T17:25:04.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.195+0000 7fdca0b4f700 1 -- 172.21.15.98:0/3083369262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 msgr2=0x7fdc9c108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.195+0000 7fdca0b4f700 1 --2- 172.21.15.98:0/3083369262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c108790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0098a0 tx=0x7fdc8c009bb0 comp rx=0 tx=0).stop 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.195+0000 7fdca0b4f700 1 -- 172.21.15.98:0/3083369262 shutdown_connections 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.195+0000 7fdca0b4f700 1 --2- 172.21.15.98:0/3083369262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c108790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.195+0000 7fdca0b4f700 1 -- 172.21.15.98:0/3083369262 >> 172.21.15.98:0/3083369262 conn(0x7fdc9c1038f0 msgr2=0x7fdc9c105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.195+0000 7fdca0b4f700 1 -- 172.21.15.98:0/3083369262 shutdown_connections 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.195+0000 7fdca0b4f700 1 -- 172.21.15.98:0/3083369262 wait complete. 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.196+0000 7fdca0b4f700 1 Processor -- start 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.196+0000 7fdca0b4f700 1 -- start start 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.196+0000 7fdca0b4f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c19c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.196+0000 7fdca0b4f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9c10c8d0 con 0x7fdc9c108370 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.196+0000 7fdc9a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c19c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.196+0000 7fdc9a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c19c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54560/0 (socket says 172.21.15.98:54560) 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.196+0000 7fdc9a59c700 1 -- 172.21.15.98:0/2231315589 learned_addr learned my addr 172.21.15.98:0/2231315589 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.196+0000 7fdc9a59c700 1 -- 172.21.15.98:0/2231315589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9c19c6c0 con 0x7fdc9c108370 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.196+0000 7fdc9a59c700 1 --2- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c19c180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0116e0 tx=0x7fdc8c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.197+0000 7fdc8b7fe700 1 -- 172.21.15.98:0/2231315589 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c0039e0 con 0x7fdc9c108370 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.197+0000 7fdc8b7fe700 1 -- 172.21.15.98:0/2231315589 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc8c01b8f0 con 0x7fdc9c108370 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.197+0000 7fdc8b7fe700 1 -- 172.21.15.98:0/2231315589 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c023760 con 0x7fdc9c108370 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.197+0000 7fdca0b4f700 1 -- 172.21.15.98:0/2231315589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc9c19c950 con 0x7fdc9c108370 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.197+0000 7fdca0b4f700 1 -- 172.21.15.98:0/2231315589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc9c19f640 con 0x7fdc9c108370 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.198+0000 7fdc8b7fe700 1 -- 172.21.15.98:0/2231315589 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fdc8c023920 con 0x7fdc9c108370 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.198+0000 7fdca0b4f700 1 -- 172.21.15.98:0/2231315589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc9c0640a0 con 0x7fdc9c108370 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.198+0000 7fdc8b7fe700 1 --2- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fdc840396d0 0x7fdc8403bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.198+0000 7fdc8b7fe700 1 -- 172.21.15.98:0/2231315589 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdc8c04e080 con 0x7fdc9c108370 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.198+0000 7fdc99d9b700 1 -- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fdc840396d0 msgr2=0x7fdc8403bb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3170662380 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.198+0000 7fdc99d9b700 1 --2- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fdc840396d0 0x7fdc8403bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.203+0000 7fdc8b7fe700 1 -- 172.21.15.98:0/2231315589 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc8c01f030 con 0x7fdc9c108370 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.399+0000 7fdc99d9b700 1 -- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fdc840396d0 msgr2=0x7fdc8403bb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3170662380 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.399+0000 7fdc99d9b700 1 --2- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fdc840396d0 0x7fdc8403bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T17:25:04.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.500+0000 7fdca0b4f700 1 -- 172.21.15.98:0/2231315589 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fdc9c19fbf0 con 0x7fdc9c108370 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.500+0000 7fdc8b7fe700 1 -- 172.21.15.98:0/2231315589 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fdc8c01bd50 con 0x7fdc9c108370 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.502+0000 7fdca0b4f700 1 -- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fdc840396d0 msgr2=0x7fdc8403bb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.502+0000 7fdca0b4f700 1 --2- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fdc840396d0 0x7fdc8403bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.502+0000 7fdca0b4f700 1 -- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 msgr2=0x7fdc9c19c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.502+0000 7fdca0b4f700 1 --2- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c19c180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0116e0 tx=0x7fdc8c011aa0 comp rx=0 tx=0).stop 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.502+0000 7fdca0b4f700 1 -- 172.21.15.98:0/2231315589 shutdown_connections 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.502+0000 7fdca0b4f700 1 --2- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7fdc840396d0 0x7fdc8403bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.502+0000 7fdca0b4f700 1 --2- 172.21.15.98:0/2231315589 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc9c108370 0x7fdc9c19c180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.502+0000 7fdca0b4f700 1 -- 172.21.15.98:0/2231315589 >> 172.21.15.98:0/2231315589 conn(0x7fdc9c1038f0 msgr2=0x7fdc9c104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.503+0000 7fdca0b4f700 1 -- 172.21.15.98:0/2231315589 shutdown_connections 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:03.503+0000 7fdca0b4f700 1 -- 172.21.15.98:0/2231315589 wait complete. 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-04-25T17:25:04.067 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 9... 2024-04-25T17:25:04.132 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:04 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2231315589' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T17:25:04.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:04 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:04.130+0000 7f0e0d082200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T17:25:04.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:04 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:04.304+0000 7f0e0d082200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T17:25:04.770 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:04 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:04.479+0000 7f0e0d082200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T17:25:04.771 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:04 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:04.769+0000 7f0e0d082200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T17:25:05.064 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:04 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:04.863+0000 7f0e0d082200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T17:25:05.064 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:05.063+0000 7f0e0d082200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T17:25:05.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:05.256+0000 7f0e0d082200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:25:06.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:05.848+0000 7f0e0d082200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T17:25:06.570 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:06 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:06.271+0000 7f0e0d082200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T17:25:06.570 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:06 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:06.367+0000 7f0e0d082200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T17:25:06.570 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:06 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:06.569+0000 7f0e0d082200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T17:25:07.425 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:07.088+0000 7f0e0d082200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T17:25:07.425 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:07.181+0000 7f0e0d082200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T17:25:07.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:07.423+0000 7f0e0d082200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T17:25:07.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:07.555+0000 7f0e0d082200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T17:25:07.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:07.651+0000 7f0e0d082200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T17:25:08.065 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:07.767+0000 7f0e0d082200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T17:25:08.065 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:08.064+0000 7f0e0d082200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T17:25:08.938 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:08.665+0000 7f0e0d082200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T17:25:08.938 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:25:08.880+0000 7f0e0d082200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: Active manager daemon x restarted 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: Activating manager daemon x 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: mgrmap e10: x(active, starting, since 0.00332372s) 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: Manager daemon x is now available 2024-04-25T17:25:09.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:25:10.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T17:25:10.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T17:25:10.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:09 smithi098 ceph-mon[29192]: mgrmap e11: x(active, since 1.00673s) 2024-04-25T17:25:10.513 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T17:25:10.513 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.474+0000 7faa4f059700 1 Processor -- start 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.474+0000 7faa4f059700 1 -- start start 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.474+0000 7faa4f059700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48106a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.474+0000 7faa4f059700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa48107040 con 0x7faa48106650 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.475+0000 7faa4cdf5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48106a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.475+0000 7faa4cdf5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48106a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38448/0 (socket says 172.21.15.98:38448) 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.475+0000 7faa4cdf5700 1 -- 172.21.15.98:0/2481303476 learned_addr learned my addr 172.21.15.98:0/2481303476 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.475+0000 7faa4cdf5700 1 -- 172.21.15.98:0/2481303476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa481095e0 con 0x7faa48106650 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.475+0000 7faa4cdf5700 1 --2- 172.21.15.98:0/2481303476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48106a70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0098a0 tx=0x7faa3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8dd285a1b9682d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.475+0000 7faa477fe700 1 -- 172.21.15.98:0/2481303476 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa3c004160 con 0x7faa48106650 2024-04-25T17:25:10.514 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.475+0000 7faa477fe700 1 -- 172.21.15.98:0/2481303476 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa3c004300 con 0x7faa48106650 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.475+0000 7faa477fe700 1 -- 172.21.15.98:0/2481303476 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa3c011a20 con 0x7faa48106650 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.476+0000 7faa4f059700 1 -- 172.21.15.98:0/2481303476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 msgr2=0x7faa48106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.476+0000 7faa4f059700 1 --2- 172.21.15.98:0/2481303476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48106a70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0098a0 tx=0x7faa3c009bb0 comp rx=0 tx=0).stop 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.476+0000 7faa4f059700 1 -- 172.21.15.98:0/2481303476 shutdown_connections 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.476+0000 7faa4f059700 1 --2- 172.21.15.98:0/2481303476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48106a70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.476+0000 7faa4f059700 1 -- 172.21.15.98:0/2481303476 >> 172.21.15.98:0/2481303476 conn(0x7faa48101fe0 msgr2=0x7faa48104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.476+0000 7faa4f059700 1 -- 172.21.15.98:0/2481303476 shutdown_connections 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.476+0000 7faa4f059700 1 -- 172.21.15.98:0/2481303476 wait complete. 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.477+0000 7faa4f059700 1 Processor -- start 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.477+0000 7faa4f059700 1 -- start start 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.477+0000 7faa4f059700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48108e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.477+0000 7faa4f059700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa4810c990 con 0x7faa48106650 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.477+0000 7faa4cdf5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48108e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:10.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.477+0000 7faa4cdf5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48108e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38464/0 (socket says 172.21.15.98:38464) 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.477+0000 7faa4cdf5700 1 -- 172.21.15.98:0/210934810 learned_addr learned my addr 172.21.15.98:0/210934810 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.477+0000 7faa4cdf5700 1 -- 172.21.15.98:0/210934810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa481093a0 con 0x7faa48106650 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.477+0000 7faa4cdf5700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48108e60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009520 tx=0x7faa3c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.477+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa3c0039e0 con 0x7faa48106650 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.478+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa3c01b8f0 con 0x7faa48106650 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.478+0000 7faa4f059700 1 -- 172.21.15.98:0/210934810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa48107590 con 0x7faa48106650 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.478+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa3c019770 con 0x7faa48106650 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.478+0000 7faa4f059700 1 -- 172.21.15.98:0/210934810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa48107ad0 con 0x7faa48106650 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.479+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7faa3c019a70 con 0x7faa48106650 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.479+0000 7faa45ffb700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 0x7faa3803bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.479+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 --> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7faa3803c290 con 0x7faa380396c0 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.479+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faa3c04df60 con 0x7faa48106650 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.479+0000 7faa47fff700 1 -- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 msgr2=0x7faa3803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3170662380 2024-04-25T17:25:10.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.479+0000 7faa47fff700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 0x7faa3803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.680+0000 7faa47fff700 1 -- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 msgr2=0x7faa3803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3170662380 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:04.680+0000 7faa47fff700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 0x7faa3803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:05.081+0000 7faa47fff700 1 -- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 msgr2=0x7faa3803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3170662380 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:05.081+0000 7faa47fff700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 0x7faa3803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:05.882+0000 7faa47fff700 1 -- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 msgr2=0x7faa3803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3170662380 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:05.882+0000 7faa47fff700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 0x7faa3803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:07.483+0000 7faa47fff700 1 -- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 msgr2=0x7faa3803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/3170662380 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:07.483+0000 7faa47fff700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 0x7faa3803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:08.884+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7faa3c01a8f0 con 0x7faa48106650 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:08.884+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 msgr2=0x7faa3803bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:08.884+0000 7faa45ffb700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 0x7faa3803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.886+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7faa3c04fff0 con 0x7faa48106650 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.886+0000 7faa45ffb700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa3803cfb0 0x7faa3803f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.886+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7faa3c024e70 con 0x7faa3803cfb0 2024-04-25T17:25:10.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.886+0000 7faa47fff700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa3803cfb0 0x7faa3803f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.888+0000 7faa47fff700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa3803cfb0 0x7faa3803f3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faa34003a10 tx=0x7faa340092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.889+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7faa3c024e70 con 0x7faa3803cfb0 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.895+0000 7faa4f059700 1 -- 172.21.15.98:0/210934810 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7faa48069b50 con 0x7faa3803cfb0 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.895+0000 7faa45ffb700 1 -- 172.21.15.98:0/210934810 <== mgr.14152 v2:172.21.15.98:6800/4145487482 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7faa48069b50 con 0x7faa3803cfb0 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 -- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa3803cfb0 msgr2=0x7faa3803f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa3803cfb0 0x7faa3803f3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faa34003a10 tx=0x7faa340092b0 comp rx=0 tx=0).stop 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 -- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 msgr2=0x7faa48108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48108e60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009520 tx=0x7faa3c011aa0 comp rx=0 tx=0).stop 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 -- 172.21.15.98:0/210934810 shutdown_connections 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa3803cfb0 0x7faa3803f3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa48106650 0x7faa48108e60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 --2- 172.21.15.98:0/210934810 >> [v2:172.21.15.98:6800/3170662380,v1:172.21.15.98:6801/3170662380] conn(0x7faa380396c0 0x7faa3803bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:10.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 -- 172.21.15.98:0/210934810 >> 172.21.15.98:0/210934810 conn(0x7faa48101fe0 msgr2=0x7faa48102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:10.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 -- 172.21.15.98:0/210934810 shutdown_connections 2024-04-25T17:25:10.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:09.896+0000 7faa4f059700 1 -- 172.21.15.98:0/210934810 wait complete. 2024-04-25T17:25:10.519 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 9 is available 2024-04-25T17:25:10.519 INFO:teuthology.orchestra.run.smithi098.stdout:Generating a dashboard self-signed certificate... 2024-04-25T17:25:11.200 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:10 smithi098 ceph-mon[29192]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T17:25:11.200 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:10 smithi098 ceph-mon[29192]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T17:25:11.200 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:11 smithi098 ceph-mon[29192]: [25/Apr/2024:17:25:10] ENGINE Bus STARTING 2024-04-25T17:25:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:11 smithi098 ceph-mon[29192]: [25/Apr/2024:17:25:10] ENGINE Serving on https://172.21.15.98:7150 2024-04-25T17:25:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:11 smithi098 ceph-mon[29192]: [25/Apr/2024:17:25:10] ENGINE Serving on http://172.21.15.98:8765 2024-04-25T17:25:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:11 smithi098 ceph-mon[29192]: [25/Apr/2024:17:25:10] ENGINE Bus STARTED 2024-04-25T17:25:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:11 smithi098 ceph-mon[29192]: mgrmap e12: x(active, since 2s) 2024-04-25T17:25:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:12.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T17:25:12.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.300+0000 7faba56ab700 1 Processor -- start 2024-04-25T17:25:12.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.300+0000 7faba56ab700 1 -- start start 2024-04-25T17:25:12.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.300+0000 7faba56ab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba0107660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:12.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.300+0000 7faba56ab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faba0107c30 con 0x7faba0107240 2024-04-25T17:25:12.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.300+0000 7fab9effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba0107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:12.321 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.300+0000 7fab9effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba0107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38568/0 (socket says 172.21.15.98:38568) 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.300+0000 7fab9effd700 1 -- 172.21.15.98:0/1313357259 learned_addr learned my addr 172.21.15.98:0/1313357259 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.301+0000 7fab9effd700 1 -- 172.21.15.98:0/1313357259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba0108440 con 0x7faba0107240 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.301+0000 7fab9effd700 1 --2- 172.21.15.98:0/1313357259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba0107660 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fab900098a0 tx=0x7fab90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8927e3831059d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.301+0000 7fab9dffb700 1 -- 172.21.15.98:0/1313357259 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab90004160 con 0x7faba0107240 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.301+0000 7fab9dffb700 1 -- 172.21.15.98:0/1313357259 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab90004300 con 0x7faba0107240 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.301+0000 7fab9dffb700 1 -- 172.21.15.98:0/1313357259 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab90011a20 con 0x7faba0107240 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.302+0000 7faba56ab700 1 -- 172.21.15.98:0/1313357259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 msgr2=0x7faba0107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.302+0000 7faba56ab700 1 --2- 172.21.15.98:0/1313357259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba0107660 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fab900098a0 tx=0x7fab90009bb0 comp rx=0 tx=0).stop 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.302+0000 7faba56ab700 1 -- 172.21.15.98:0/1313357259 shutdown_connections 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.302+0000 7faba56ab700 1 --2- 172.21.15.98:0/1313357259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba0107660 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.302+0000 7faba56ab700 1 -- 172.21.15.98:0/1313357259 >> 172.21.15.98:0/1313357259 conn(0x7faba0102820 msgr2=0x7faba0104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.302+0000 7faba56ab700 1 -- 172.21.15.98:0/1313357259 shutdown_connections 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.302+0000 7faba56ab700 1 -- 172.21.15.98:0/1313357259 wait complete. 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.302+0000 7faba56ab700 1 Processor -- start 2024-04-25T17:25:12.322 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7faba56ab700 1 -- start start 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7faba56ab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba01923b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7faba56ab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faba010b7f0 con 0x7faba0107240 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7fab9effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba01923b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7fab9effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba01923b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38582/0 (socket says 172.21.15.98:38582) 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7fab9effd700 1 -- 172.21.15.98:0/618104810 learned_addr learned my addr 172.21.15.98:0/618104810 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7fab9effd700 1 -- 172.21.15.98:0/618104810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba01928f0 con 0x7faba0107240 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7fab9effd700 1 --2- 172.21.15.98:0/618104810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba01923b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab900116e0 tx=0x7fab90011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7fab87fff700 1 -- 172.21.15.98:0/618104810 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab900039e0 con 0x7faba0107240 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7fab87fff700 1 -- 172.21.15.98:0/618104810 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab9001b8f0 con 0x7faba0107240 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.303+0000 7faba56ab700 1 -- 172.21.15.98:0/618104810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba0190ae0 con 0x7faba0107240 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.304+0000 7fab87fff700 1 -- 172.21.15.98:0/618104810 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab90023760 con 0x7faba0107240 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.304+0000 7faba56ab700 1 -- 172.21.15.98:0/618104810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba0191020 con 0x7faba0107240 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.304+0000 7fab87fff700 1 -- 172.21.15.98:0/618104810 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fab900239a0 con 0x7faba0107240 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.305+0000 7fab87fff700 1 --2- 172.21.15.98:0/618104810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab880396d0 0x7fab8803bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.305+0000 7fab87fff700 1 -- 172.21.15.98:0/618104810 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fab9001bd50 con 0x7faba0107240 2024-04-25T17:25:12.323 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.305+0000 7faba56ab700 1 -- 172.21.15.98:0/618104810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faba00640a0 con 0x7faba0107240 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.305+0000 7fab9e7fc700 1 --2- 172.21.15.98:0/618104810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab880396d0 0x7fab8803bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.305+0000 7fab9e7fc700 1 --2- 172.21.15.98:0/618104810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab880396d0 0x7fab8803bb90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab94006fd0 tx=0x7fab94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.310+0000 7fab87fff700 1 -- 172.21.15.98:0/618104810 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fab9004a020 con 0x7faba0107240 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.525+0000 7faba56ab700 1 -- 172.21.15.98:0/618104810 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7faba0196000 con 0x7fab880396d0 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.705+0000 7fab87fff700 1 -- 172.21.15.98:0/618104810 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7faba0196000 con 0x7fab880396d0 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.707+0000 7faba56ab700 1 -- 172.21.15.98:0/618104810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab880396d0 msgr2=0x7fab8803bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.707+0000 7faba56ab700 1 --2- 172.21.15.98:0/618104810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab880396d0 0x7fab8803bb90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab94006fd0 tx=0x7fab94006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.707+0000 7faba56ab700 1 -- 172.21.15.98:0/618104810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 msgr2=0x7faba01923b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.707+0000 7faba56ab700 1 --2- 172.21.15.98:0/618104810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba01923b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fab900116e0 tx=0x7fab90011aa0 comp rx=0 tx=0).stop 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.707+0000 7faba56ab700 1 -- 172.21.15.98:0/618104810 shutdown_connections 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.707+0000 7faba56ab700 1 --2- 172.21.15.98:0/618104810 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab880396d0 0x7fab8803bb90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.707+0000 7faba56ab700 1 --2- 172.21.15.98:0/618104810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faba0107240 0x7faba01923b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.707+0000 7faba56ab700 1 -- 172.21.15.98:0/618104810 >> 172.21.15.98:0/618104810 conn(0x7faba0102820 msgr2=0x7faba01035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.707+0000 7faba56ab700 1 -- 172.21.15.98:0/618104810 shutdown_connections 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:11.707+0000 7faba56ab700 1 -- 172.21.15.98:0/618104810 wait complete. 2024-04-25T17:25:12.324 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial admin user... 2024-04-25T17:25:13.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:12 smithi098 ceph-mon[29192]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:25:13.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:14.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$7fL0Tk7I.yUMTezlpKPwjOMg6G/qbblTTstNz77/t.dPcAB5QLHpu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714065913, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T17:25:14.192 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.779+0000 7f20231da700 1 Processor -- start 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.779+0000 7f20231da700 1 -- start start 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.780+0000 7f20231da700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c10b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.780+0000 7f20231da700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f201c07be60 con 0x7f201c108df0 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.780+0000 7f2020f76700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c10b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.780+0000 7f2020f76700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c10b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38594/0 (socket says 172.21.15.98:38594) 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.780+0000 7f2020f76700 1 -- 172.21.15.98:0/3717243973 learned_addr learned my addr 172.21.15.98:0/3717243973 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.780+0000 7f2020f76700 1 -- 172.21.15.98:0/3717243973 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c10b720 con 0x7f201c108df0 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.780+0000 7f2020f76700 1 --2- 172.21.15.98:0/3717243973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c10b1e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f20100098a0 tx=0x7f2010009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd538bc0fb1c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.781+0000 7f201b7fe700 1 -- 172.21.15.98:0/3717243973 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2010004160 con 0x7f201c108df0 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.781+0000 7f201b7fe700 1 -- 172.21.15.98:0/3717243973 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2010004300 con 0x7f201c108df0 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.781+0000 7f201b7fe700 1 -- 172.21.15.98:0/3717243973 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20100119d0 con 0x7f201c108df0 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.781+0000 7f20231da700 1 -- 172.21.15.98:0/3717243973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 msgr2=0x7f201c10b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.781+0000 7f20231da700 1 --2- 172.21.15.98:0/3717243973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c10b1e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f20100098a0 tx=0x7f2010009bb0 comp rx=0 tx=0).stop 2024-04-25T17:25:14.193 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.782+0000 7f20231da700 1 -- 172.21.15.98:0/3717243973 shutdown_connections 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.782+0000 7f20231da700 1 --2- 172.21.15.98:0/3717243973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c10b1e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.782+0000 7f20231da700 1 -- 172.21.15.98:0/3717243973 >> 172.21.15.98:0/3717243973 conn(0x7f201c102820 msgr2=0x7f201c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.782+0000 7f20231da700 1 -- 172.21.15.98:0/3717243973 shutdown_connections 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.782+0000 7f20231da700 1 -- 172.21.15.98:0/3717243973 wait complete. 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.782+0000 7f20231da700 1 Processor -- start 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.782+0000 7f20231da700 1 -- start start 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.782+0000 7f20231da700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c108670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.782+0000 7f20231da700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f201c10c720 con 0x7f201c108df0 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.782+0000 7f2020f76700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c108670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.783+0000 7f2020f76700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c108670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38598/0 (socket says 172.21.15.98:38598) 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.783+0000 7f2020f76700 1 -- 172.21.15.98:0/1731999776 learned_addr learned my addr 172.21.15.98:0/1731999776 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.783+0000 7f2020f76700 1 -- 172.21.15.98:0/1731999776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c108bb0 con 0x7f201c108df0 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.783+0000 7f2020f76700 1 --2- 172.21.15.98:0/1731999776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c108670 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2010000c00 tx=0x7f2010011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:14.194 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.783+0000 7f2019ffb700 1 -- 172.21.15.98:0/1731999776 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2010003af0 con 0x7f201c108df0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.783+0000 7f2019ffb700 1 -- 172.21.15.98:0/1731999776 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f201001e070 con 0x7f201c108df0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.783+0000 7f20231da700 1 -- 172.21.15.98:0/1731999776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201c106da0 con 0x7f201c108df0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.783+0000 7f2019ffb700 1 -- 172.21.15.98:0/1731999776 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f201001a6b0 con 0x7f201c108df0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.783+0000 7f20231da700 1 -- 172.21.15.98:0/1731999776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201c1072e0 con 0x7f201c108df0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.784+0000 7f2019ffb700 1 -- 172.21.15.98:0/1731999776 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f201001a9d0 con 0x7f201c108df0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.784+0000 7f20231da700 1 -- 172.21.15.98:0/1731999776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201c0640a0 con 0x7f201c108df0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.785+0000 7f2019ffb700 1 --2- 172.21.15.98:0/1731999776 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f200c0396c0 0x7f200c03bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.785+0000 7f2019ffb700 1 -- 172.21.15.98:0/1731999776 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f201004de90 con 0x7f201c108df0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.785+0000 7f201bfff700 1 --2- 172.21.15.98:0/1731999776 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f200c0396c0 0x7f200c03bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.785+0000 7f201bfff700 1 --2- 172.21.15.98:0/1731999776 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f200c0396c0 0x7f200c03bb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2008006fd0 tx=0x7f2008006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:12.790+0000 7f2019ffb700 1 -- 172.21.15.98:0/1731999776 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f201001adc0 con 0x7f201c108df0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.012+0000 7f20231da700 1 -- 172.21.15.98:0/1731999776 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f201c107620 con 0x7f200c0396c0 2024-04-25T17:25:14.195 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.236+0000 7f2019ffb700 1 -- 172.21.15.98:0/1731999776 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f201c107620 con 0x7f200c0396c0 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.238+0000 7f20231da700 1 -- 172.21.15.98:0/1731999776 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f200c0396c0 msgr2=0x7f200c03bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.238+0000 7f20231da700 1 --2- 172.21.15.98:0/1731999776 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f200c0396c0 0x7f200c03bb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2008006fd0 tx=0x7f2008006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.238+0000 7f20231da700 1 -- 172.21.15.98:0/1731999776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 msgr2=0x7f201c108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.238+0000 7f20231da700 1 --2- 172.21.15.98:0/1731999776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c108670 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2010000c00 tx=0x7f2010011aa0 comp rx=0 tx=0).stop 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.238+0000 7f20231da700 1 -- 172.21.15.98:0/1731999776 shutdown_connections 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.238+0000 7f20231da700 1 --2- 172.21.15.98:0/1731999776 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f200c0396c0 0x7f200c03bb80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.238+0000 7f20231da700 1 --2- 172.21.15.98:0/1731999776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c108df0 0x7f201c108670 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.238+0000 7f20231da700 1 -- 172.21.15.98:0/1731999776 >> 172.21.15.98:0/1731999776 conn(0x7f201c102820 msgr2=0x7f201c103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.238+0000 7f20231da700 1 -- 172.21.15.98:0/1731999776 shutdown_connections 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:13.238+0000 7f20231da700 1 -- 172.21.15.98:0/1731999776 wait complete. 2024-04-25T17:25:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:Fetching dashboard port number... 2024-04-25T17:25:14.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:13 smithi098 ceph-mon[29192]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:25:14.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:15.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:14 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3813456208' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T17:25:15.536 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T17:25:15.536 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.611+0000 7f03bb257700 1 Processor -- start 2024-04-25T17:25:15.536 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.611+0000 7f03bb257700 1 -- start start 2024-04-25T17:25:15.536 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.612+0000 7f03bb257700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b4108760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:15.536 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.612+0000 7f03bb257700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03b4108d30 con 0x7f03b4108340 2024-04-25T17:25:15.536 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.612+0000 7f03b8ff3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b4108760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:15.536 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.612+0000 7f03b8ff3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b4108760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46692/0 (socket says 172.21.15.98:46692) 2024-04-25T17:25:15.536 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.612+0000 7f03b8ff3700 1 -- 172.21.15.98:0/397309889 learned_addr learned my addr 172.21.15.98:0/397309889 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.612+0000 7f03b8ff3700 1 -- 172.21.15.98:0/397309889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b4109540 con 0x7f03b4108340 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.612+0000 7f03b8ff3700 1 --2- 172.21.15.98:0/397309889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b4108760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f03b0009e80 tx=0x7f03b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2f653bc4825f94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.612+0000 7f03ab7fe700 1 -- 172.21.15.98:0/397309889 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03b0017070 con 0x7f03b4108340 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.612+0000 7f03ab7fe700 1 -- 172.21.15.98:0/397309889 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03b000b8c0 con 0x7f03b4108340 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.612+0000 7f03ab7fe700 1 -- 172.21.15.98:0/397309889 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03b0004490 con 0x7f03b4108340 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.613+0000 7f03bb257700 1 -- 172.21.15.98:0/397309889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 msgr2=0x7f03b4108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.613+0000 7f03bb257700 1 --2- 172.21.15.98:0/397309889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b4108760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f03b0009e80 tx=0x7f03b000b330 comp rx=0 tx=0).stop 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.613+0000 7f03bb257700 1 -- 172.21.15.98:0/397309889 shutdown_connections 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.613+0000 7f03bb257700 1 --2- 172.21.15.98:0/397309889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b4108760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.613+0000 7f03bb257700 1 -- 172.21.15.98:0/397309889 >> 172.21.15.98:0/397309889 conn(0x7f03b41038c0 msgr2=0x7f03b4105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.613+0000 7f03bb257700 1 -- 172.21.15.98:0/397309889 shutdown_connections 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.613+0000 7f03bb257700 1 -- 172.21.15.98:0/397309889 wait complete. 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03bb257700 1 Processor -- start 2024-04-25T17:25:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03bb257700 1 -- start start 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03bb257700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b419c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03bb257700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f03b410c8f0 con 0x7f03b4108340 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03b8ff3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b419c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03b8ff3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b419c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46700/0 (socket says 172.21.15.98:46700) 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03b8ff3700 1 -- 172.21.15.98:0/3813456208 learned_addr learned my addr 172.21.15.98:0/3813456208 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03b8ff3700 1 -- 172.21.15.98:0/3813456208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b419c710 con 0x7f03b4108340 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03b8ff3700 1 --2- 172.21.15.98:0/3813456208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b419c1d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f03b000b7b0 tx=0x7f03b000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03a9ffb700 1 -- 172.21.15.98:0/3813456208 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03b0020020 con 0x7f03b4108340 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03a9ffb700 1 -- 172.21.15.98:0/3813456208 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03b0003b50 con 0x7f03b4108340 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03a9ffb700 1 -- 172.21.15.98:0/3813456208 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03b0017040 con 0x7f03b4108340 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.614+0000 7f03bb257700 1 -- 172.21.15.98:0/3813456208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b419c9a0 con 0x7f03b4108340 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.615+0000 7f03bb257700 1 -- 172.21.15.98:0/3813456208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b419f690 con 0x7f03b4108340 2024-04-25T17:25:15.538 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.615+0000 7f03a9ffb700 1 -- 172.21.15.98:0/3813456208 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f03b0023020 con 0x7f03b4108340 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.615+0000 7f03bb257700 1 -- 172.21.15.98:0/3813456208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b40640a0 con 0x7f03b4108340 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.615+0000 7f03a9ffb700 1 --2- 172.21.15.98:0/3813456208 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f039c039710 0x7f039c03bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.616+0000 7f03a9ffb700 1 -- 172.21.15.98:0/3813456208 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f03b000f050 con 0x7f03b4108340 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.616+0000 7f03abfff700 1 --2- 172.21.15.98:0/3813456208 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f039c039710 0x7f039c03bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.616+0000 7f03abfff700 1 --2- 172.21.15.98:0/3813456208 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f039c039710 0x7f039c03bbd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f03a4006fd0 tx=0x7f03a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.620+0000 7f03a9ffb700 1 -- 172.21.15.98:0/3813456208 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03b002a020 con 0x7f03b4108340 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.822+0000 7f03bb257700 1 -- 172.21.15.98:0/3813456208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f03b419f860 con 0x7f03b4108340 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.822+0000 7f03a9ffb700 1 -- 172.21.15.98:0/3813456208 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f03b419f860 con 0x7f03b4108340 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.825+0000 7f03bb257700 1 -- 172.21.15.98:0/3813456208 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f039c039710 msgr2=0x7f039c03bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.825+0000 7f03bb257700 1 --2- 172.21.15.98:0/3813456208 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f039c039710 0x7f039c03bbd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f03a4006fd0 tx=0x7f03a4006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.825+0000 7f03bb257700 1 -- 172.21.15.98:0/3813456208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 msgr2=0x7f03b419c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.825+0000 7f03bb257700 1 --2- 172.21.15.98:0/3813456208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b419c1d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f03b000b7b0 tx=0x7f03b000bd30 comp rx=0 tx=0).stop 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.825+0000 7f03bb257700 1 -- 172.21.15.98:0/3813456208 shutdown_connections 2024-04-25T17:25:15.539 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.825+0000 7f03bb257700 1 --2- 172.21.15.98:0/3813456208 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f039c039710 0x7f039c03bbd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:15.540 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.825+0000 7f03bb257700 1 --2- 172.21.15.98:0/3813456208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03b4108340 0x7f03b419c1d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:15.540 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.825+0000 7f03bb257700 1 -- 172.21.15.98:0/3813456208 >> 172.21.15.98:0/3813456208 conn(0x7f03b41038c0 msgr2=0x7f03b41043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:15.540 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.825+0000 7f03bb257700 1 -- 172.21.15.98:0/3813456208 shutdown_connections 2024-04-25T17:25:15.540 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:14.825+0000 7f03bb257700 1 -- 172.21.15.98:0/3813456208 wait complete. 2024-04-25T17:25:15.542 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T17:25:15.542 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-25T17:25:15.549 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T17:25:15.549 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T17:25:15.549 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-25T17:25:15.549 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T17:25:15.550 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph Dashboard is now available at: 2024-04-25T17:25:15.550 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:15.550 INFO:teuthology.orchestra.run.smithi098.stdout: URL: https://smithi098:8443/ 2024-04-25T17:25:15.550 INFO:teuthology.orchestra.run.smithi098.stdout: User: admin 2024-04-25T17:25:15.551 INFO:teuthology.orchestra.run.smithi098.stdout: Password: 90h3picn7v 2024-04-25T17:25:15.551 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:15.551 INFO:teuthology.orchestra.run.smithi098.stdout:Saving cluster configuration to /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config directory 2024-04-25T17:25:15.551 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling autotune for osd_memory_target 2024-04-25T17:25:16.165 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:15 smithi098 ceph-mon[29192]: mgrmap e13: x(active, since 6s) 2024-04-25T17:25:16.742 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.957+0000 7fb3cf8db700 1 Processor -- start 2024-04-25T17:25:16.742 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.957+0000 7fb3cf8db700 1 -- start start 2024-04-25T17:25:16.742 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.957+0000 7fb3cf8db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c8108740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:16.742 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.957+0000 7fb3cf8db700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c8108d10 con 0x7fb3c8108320 2024-04-25T17:25:16.742 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.957+0000 7fb3cd677700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c8108740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:16.742 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.957+0000 7fb3cd677700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c8108740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46712/0 (socket says 172.21.15.98:46712) 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.957+0000 7fb3cd677700 1 -- 172.21.15.98:0/3757324794 learned_addr learned my addr 172.21.15.98:0/3757324794 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.958+0000 7fb3cd677700 1 -- 172.21.15.98:0/3757324794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c8109520 con 0x7fb3c8108320 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.958+0000 7fb3cd677700 1 --2- 172.21.15.98:0/3757324794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c8108740 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb3c40098a0 tx=0x7fb3c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f06dd1f3b23cd8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.958+0000 7fb3bffff700 1 -- 172.21.15.98:0/3757324794 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3c4004160 con 0x7fb3c8108320 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.958+0000 7fb3bffff700 1 -- 172.21.15.98:0/3757324794 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c4004300 con 0x7fb3c8108320 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.958+0000 7fb3bffff700 1 -- 172.21.15.98:0/3757324794 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3c4011a20 con 0x7fb3c8108320 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.958+0000 7fb3cf8db700 1 -- 172.21.15.98:0/3757324794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 msgr2=0x7fb3c8108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.958+0000 7fb3cf8db700 1 --2- 172.21.15.98:0/3757324794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c8108740 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb3c40098a0 tx=0x7fb3c4009bb0 comp rx=0 tx=0).stop 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.958+0000 7fb3cf8db700 1 -- 172.21.15.98:0/3757324794 shutdown_connections 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.958+0000 7fb3cf8db700 1 --2- 172.21.15.98:0/3757324794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c8108740 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.958+0000 7fb3cf8db700 1 -- 172.21.15.98:0/3757324794 >> 172.21.15.98:0/3757324794 conn(0x7fb3c81038e0 msgr2=0x7fb3c8105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.959+0000 7fb3cf8db700 1 -- 172.21.15.98:0/3757324794 shutdown_connections 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.959+0000 7fb3cf8db700 1 -- 172.21.15.98:0/3757324794 wait complete. 2024-04-25T17:25:16.743 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.959+0000 7fb3cf8db700 1 Processor -- start 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.959+0000 7fb3cf8db700 1 -- start start 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.959+0000 7fb3cf8db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c819c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.959+0000 7fb3cf8db700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c810c8d0 con 0x7fb3c8108320 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.959+0000 7fb3cd677700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c819c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.959+0000 7fb3cd677700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c819c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46720/0 (socket says 172.21.15.98:46720) 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.959+0000 7fb3cd677700 1 -- 172.21.15.98:0/2732715020 learned_addr learned my addr 172.21.15.98:0/2732715020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.960+0000 7fb3cd677700 1 -- 172.21.15.98:0/2732715020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c819c6c0 con 0x7fb3c8108320 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.960+0000 7fb3cd677700 1 --2- 172.21.15.98:0/2732715020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c819c180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009520 tx=0x7fb3c4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.960+0000 7fb3be7fc700 1 -- 172.21.15.98:0/2732715020 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3c40039e0 con 0x7fb3c8108320 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.960+0000 7fb3be7fc700 1 -- 172.21.15.98:0/2732715020 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c401b8f0 con 0x7fb3c8108320 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.960+0000 7fb3cf8db700 1 -- 172.21.15.98:0/2732715020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c819c950 con 0x7fb3c8108320 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.960+0000 7fb3be7fc700 1 -- 172.21.15.98:0/2732715020 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3c4019770 con 0x7fb3c8108320 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.960+0000 7fb3cf8db700 1 -- 172.21.15.98:0/2732715020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3c819f640 con 0x7fb3c8108320 2024-04-25T17:25:16.744 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.961+0000 7fb3be7fc700 1 -- 172.21.15.98:0/2732715020 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb3c4019a40 con 0x7fb3c8108320 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.961+0000 7fb3be7fc700 1 --2- 172.21.15.98:0/2732715020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb3b40397b0 0x7fb3b403bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.961+0000 7fb3be7fc700 1 -- 172.21.15.98:0/2732715020 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb3c404e0a0 con 0x7fb3c8108320 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.961+0000 7fb3cf8db700 1 -- 172.21.15.98:0/2732715020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3c80640a0 con 0x7fb3c8108320 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.961+0000 7fb3cce76700 1 --2- 172.21.15.98:0/2732715020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb3b40397b0 0x7fb3b403bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.962+0000 7fb3cce76700 1 --2- 172.21.15.98:0/2732715020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb3b40397b0 0x7fb3b403bc70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8006fd0 tx=0x7fb3b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:15.966+0000 7fb3be7fc700 1 -- 172.21.15.98:0/2732715020 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3c401f030 con 0x7fb3c8108320 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.164+0000 7fb3cf8db700 1 -- 172.21.15.98:0/2732715020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fb3c819f810 con 0x7fb3c8108320 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.165+0000 7fb3be7fc700 1 -- 172.21.15.98:0/2732715020 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fb3c4018b80 con 0x7fb3c8108320 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.168+0000 7fb3cf8db700 1 -- 172.21.15.98:0/2732715020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb3b40397b0 msgr2=0x7fb3b403bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.168+0000 7fb3cf8db700 1 --2- 172.21.15.98:0/2732715020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb3b40397b0 0x7fb3b403bc70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8006fd0 tx=0x7fb3b8006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.168+0000 7fb3cf8db700 1 -- 172.21.15.98:0/2732715020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 msgr2=0x7fb3c819c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.168+0000 7fb3cf8db700 1 --2- 172.21.15.98:0/2732715020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c819c180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009520 tx=0x7fb3c4011aa0 comp rx=0 tx=0).stop 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.168+0000 7fb3cf8db700 1 -- 172.21.15.98:0/2732715020 shutdown_connections 2024-04-25T17:25:16.745 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.168+0000 7fb3cf8db700 1 --2- 172.21.15.98:0/2732715020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb3b40397b0 0x7fb3b403bc70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:16.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.168+0000 7fb3cf8db700 1 --2- 172.21.15.98:0/2732715020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3c8108320 0x7fb3c819c180 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:16.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.168+0000 7fb3cf8db700 1 -- 172.21.15.98:0/2732715020 >> 172.21.15.98:0/2732715020 conn(0x7fb3c81038e0 msgr2=0x7fb3c8104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:16.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.168+0000 7fb3cf8db700 1 -- 172.21.15.98:0/2732715020 shutdown_connections 2024-04-25T17:25:16.746 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:16.168+0000 7fb3cf8db700 1 -- 172.21.15.98:0/2732715020 wait complete. 2024-04-25T17:25:18.132 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.222+0000 7f082849c700 1 Processor -- start 2024-04-25T17:25:18.132 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.222+0000 7f082849c700 1 -- start start 2024-04-25T17:25:18.132 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.222+0000 7f082849c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082010b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.222+0000 7f082849c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f082007be20 con 0x7f0820108e30 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.223+0000 7f0826238700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082010b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.223+0000 7f0826238700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082010b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46728/0 (socket says 172.21.15.98:46728) 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.223+0000 7f0826238700 1 -- 172.21.15.98:0/493646820 learned_addr learned my addr 172.21.15.98:0/493646820 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.223+0000 7f0826238700 1 -- 172.21.15.98:0/493646820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082010b760 con 0x7f0820108e30 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.223+0000 7f0826238700 1 --2- 172.21.15.98:0/493646820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082010b220 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f08100098a0 tx=0x7f0810009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8a25e1bb51f948 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.223+0000 7f0825236700 1 -- 172.21.15.98:0/493646820 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0810004160 con 0x7f0820108e30 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.223+0000 7f0825236700 1 -- 172.21.15.98:0/493646820 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0810004300 con 0x7f0820108e30 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.223+0000 7f0825236700 1 -- 172.21.15.98:0/493646820 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0810011a20 con 0x7f0820108e30 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.224+0000 7f082849c700 1 -- 172.21.15.98:0/493646820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 msgr2=0x7f082010b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.224+0000 7f082849c700 1 --2- 172.21.15.98:0/493646820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082010b220 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f08100098a0 tx=0x7f0810009bb0 comp rx=0 tx=0).stop 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.224+0000 7f082849c700 1 -- 172.21.15.98:0/493646820 shutdown_connections 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.224+0000 7f082849c700 1 --2- 172.21.15.98:0/493646820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082010b220 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:18.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.224+0000 7f082849c700 1 -- 172.21.15.98:0/493646820 >> 172.21.15.98:0/493646820 conn(0x7f0820102820 msgr2=0x7f0820104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.224+0000 7f082849c700 1 -- 172.21.15.98:0/493646820 shutdown_connections 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.224+0000 7f082849c700 1 -- 172.21.15.98:0/493646820 wait complete. 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.225+0000 7f082849c700 1 Processor -- start 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.225+0000 7f082849c700 1 -- start start 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.225+0000 7f082849c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082019c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.225+0000 7f082849c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f082010c700 con 0x7f0820108e30 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.225+0000 7f0826238700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082019c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.225+0000 7f0826238700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082019c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46744/0 (socket says 172.21.15.98:46744) 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.225+0000 7f0826238700 1 -- 172.21.15.98:0/1976974201 learned_addr learned my addr 172.21.15.98:0/1976974201 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.225+0000 7f0826238700 1 -- 172.21.15.98:0/1976974201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082019c6b0 con 0x7f0820108e30 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.226+0000 7f0826238700 1 --2- 172.21.15.98:0/1976974201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082019c170 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0810000c00 tx=0x7f0810011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.226+0000 7f08177fe700 1 -- 172.21.15.98:0/1976974201 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0810003aa0 con 0x7f0820108e30 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.226+0000 7f08177fe700 1 -- 172.21.15.98:0/1976974201 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f081001b8f0 con 0x7f0820108e30 2024-04-25T17:25:18.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.226+0000 7f082849c700 1 -- 172.21.15.98:0/1976974201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f082019c940 con 0x7f0820108e30 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.226+0000 7f08177fe700 1 -- 172.21.15.98:0/1976974201 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0810023870 con 0x7f0820108e30 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.226+0000 7f082849c700 1 -- 172.21.15.98:0/1976974201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f082019f630 con 0x7f0820108e30 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.227+0000 7f08177fe700 1 -- 172.21.15.98:0/1976974201 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0810023b20 con 0x7f0820108e30 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.227+0000 7f08177fe700 1 --2- 172.21.15.98:0/1976974201 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f080c0397c0 0x7f080c03bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.227+0000 7f08177fe700 1 -- 172.21.15.98:0/1976974201 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f081004e080 con 0x7f0820108e30 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.227+0000 7f0825a37700 1 --2- 172.21.15.98:0/1976974201 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f080c0397c0 0x7f080c03bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.227+0000 7f082849c700 1 -- 172.21.15.98:0/1976974201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08200640a0 con 0x7f0820108e30 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.227+0000 7f0825a37700 1 --2- 172.21.15.98:0/1976974201 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f080c0397c0 0x7f080c03bc80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f081c006fd0 tx=0x7f081c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.232+0000 7f08177fe700 1 -- 172.21.15.98:0/1976974201 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f081001f030 con 0x7f0820108e30 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.560+0000 7f082849c700 1 -- 172.21.15.98:0/1976974201 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f082019f800 con 0x7f0820108e30 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.562+0000 7f08177fe700 1 -- 172.21.15.98:0/1976974201 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f0810018c10 con 0x7f0820108e30 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.565+0000 7f082849c700 1 -- 172.21.15.98:0/1976974201 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f080c0397c0 msgr2=0x7f080c03bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.565+0000 7f082849c700 1 --2- 172.21.15.98:0/1976974201 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f080c0397c0 0x7f080c03bc80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f081c006fd0 tx=0x7f081c006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.565+0000 7f082849c700 1 -- 172.21.15.98:0/1976974201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 msgr2=0x7f082019c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:18.135 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.565+0000 7f082849c700 1 --2- 172.21.15.98:0/1976974201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082019c170 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0810000c00 tx=0x7f0810011aa0 comp rx=0 tx=0).stop 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.565+0000 7f082849c700 1 -- 172.21.15.98:0/1976974201 shutdown_connections 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.565+0000 7f082849c700 1 --2- 172.21.15.98:0/1976974201 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f080c0397c0 0x7f080c03bc80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.565+0000 7f082849c700 1 --2- 172.21.15.98:0/1976974201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0820108e30 0x7f082019c170 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.565+0000 7f082849c700 1 -- 172.21.15.98:0/1976974201 >> 172.21.15.98:0/1976974201 conn(0x7f0820102820 msgr2=0x7f0820103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.565+0000 7f082849c700 1 -- 172.21.15.98:0/1976974201 shutdown_connections 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T17:25:17.565+0000 7f082849c700 1 -- 172.21.15.98:0/1976974201 wait complete. 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T17:25:18.136 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:18.137 INFO:teuthology.orchestra.run.smithi098.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T17:25:18.137 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:18.137 INFO:teuthology.orchestra.run.smithi098.stdout: ceph telemetry on 2024-04-25T17:25:18.137 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:18.137 INFO:teuthology.orchestra.run.smithi098.stdout:For more information see: 2024-04-25T17:25:18.137 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:18.137 INFO:teuthology.orchestra.run.smithi098.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T17:25:18.137 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:18.137 INFO:teuthology.orchestra.run.smithi098.stdout:Bootstrap complete. 2024-04-25T17:25:18.171 INFO:tasks.cephadm:Fetching config... 2024-04-25T17:25:18.171 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:25:18.171 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T17:25:18.186 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T17:25:18.186 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:25:18.186 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T17:25:18.243 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T17:25:18.243 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:25:18.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T17:25:18.307 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T17:25:18.307 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:25:18.307 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T17:25:18.362 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T17:25:18.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDitixWwEzjIanHWKHEIREwnhwxWLgffJkEk79ZMEaBm106qwF+2ZHM21Y8SfEpW6emkfVuz79bion9coU/R7Xai6Tf6fVErXSQOTrnWpvS89xFOrRGvy4H4ZhWbMrno3syHiLkgdp8Jf7A9tx6T6SdqvbOnmqhG7gRnNYLRLroRlot09Jwgr81L0RrGKNMYm+LgD1Ysirbrj4/cCVyQGGJBIys2sRY93noP+IFDU98RJ+021mXa24haoYRbygvxZ42ntVvGMJ1uR+IPfqTs2aZbVwCGXPiXNlTCbIGtb2C9DpWzujuf4UQqOAKKX88OvApdssgxwMSCfDBJLKvjuH2NrVGCd27h2wHGByPeRpdf6yPEqPT90l6VwbcNjtHwqcM51Zys4FmV60wBBF56kOmuKBGpulRB2+ar5MTLj2QFOp5pc+oC7OLlJWM/wrUxVMJ4dKS0KGKsqL/GpuhFpW3fmyAhxmIiJczyqfKrENU6+3836Rylu4ZNdV1dQ6FKb8= ceph-79754014-0328-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T17:25:18.436 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDitixWwEzjIanHWKHEIREwnhwxWLgffJkEk79ZMEaBm106qwF+2ZHM21Y8SfEpW6emkfVuz79bion9coU/R7Xai6Tf6fVErXSQOTrnWpvS89xFOrRGvy4H4ZhWbMrno3syHiLkgdp8Jf7A9tx6T6SdqvbOnmqhG7gRnNYLRLroRlot09Jwgr81L0RrGKNMYm+LgD1Ysirbrj4/cCVyQGGJBIys2sRY93noP+IFDU98RJ+021mXa24haoYRbygvxZ42ntVvGMJ1uR+IPfqTs2aZbVwCGXPiXNlTCbIGtb2C9DpWzujuf4UQqOAKKX88OvApdssgxwMSCfDBJLKvjuH2NrVGCd27h2wHGByPeRpdf6yPEqPT90l6VwbcNjtHwqcM51Zys4FmV60wBBF56kOmuKBGpulRB2+ar5MTLj2QFOp5pc+oC7OLlJWM/wrUxVMJ4dKS0KGKsqL/GpuhFpW3fmyAhxmIiJczyqfKrENU6+3836Rylu4ZNdV1dQ6FKb8= ceph-79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:25:18.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDitixWwEzjIanHWKHEIREwnhwxWLgffJkEk79ZMEaBm106qwF+2ZHM21Y8SfEpW6emkfVuz79bion9coU/R7Xai6Tf6fVErXSQOTrnWpvS89xFOrRGvy4H4ZhWbMrno3syHiLkgdp8Jf7A9tx6T6SdqvbOnmqhG7gRnNYLRLroRlot09Jwgr81L0RrGKNMYm+LgD1Ysirbrj4/cCVyQGGJBIys2sRY93noP+IFDU98RJ+021mXa24haoYRbygvxZ42ntVvGMJ1uR+IPfqTs2aZbVwCGXPiXNlTCbIGtb2C9DpWzujuf4UQqOAKKX88OvApdssgxwMSCfDBJLKvjuH2NrVGCd27h2wHGByPeRpdf6yPEqPT90l6VwbcNjtHwqcM51Zys4FmV60wBBF56kOmuKBGpulRB2+ar5MTLj2QFOp5pc+oC7OLlJWM/wrUxVMJ4dKS0KGKsqL/GpuhFpW3fmyAhxmIiJczyqfKrENU6+3836Rylu4ZNdV1dQ6FKb8= ceph-79754014-0328-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T17:25:18.487 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDitixWwEzjIanHWKHEIREwnhwxWLgffJkEk79ZMEaBm106qwF+2ZHM21Y8SfEpW6emkfVuz79bion9coU/R7Xai6Tf6fVErXSQOTrnWpvS89xFOrRGvy4H4ZhWbMrno3syHiLkgdp8Jf7A9tx6T6SdqvbOnmqhG7gRnNYLRLroRlot09Jwgr81L0RrGKNMYm+LgD1Ysirbrj4/cCVyQGGJBIys2sRY93noP+IFDU98RJ+021mXa24haoYRbygvxZ42ntVvGMJ1uR+IPfqTs2aZbVwCGXPiXNlTCbIGtb2C9DpWzujuf4UQqOAKKX88OvApdssgxwMSCfDBJLKvjuH2NrVGCd27h2wHGByPeRpdf6yPEqPT90l6VwbcNjtHwqcM51Zys4FmV60wBBF56kOmuKBGpulRB2+ar5MTLj2QFOp5pc+oC7OLlJWM/wrUxVMJ4dKS0KGKsqL/GpuhFpW3fmyAhxmIiJczyqfKrENU6+3836Rylu4ZNdV1dQ6FKb8= ceph-79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:25:18.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDitixWwEzjIanHWKHEIREwnhwxWLgffJkEk79ZMEaBm106qwF+2ZHM21Y8SfEpW6emkfVuz79bion9coU/R7Xai6Tf6fVErXSQOTrnWpvS89xFOrRGvy4H4ZhWbMrno3syHiLkgdp8Jf7A9tx6T6SdqvbOnmqhG7gRnNYLRLroRlot09Jwgr81L0RrGKNMYm+LgD1Ysirbrj4/cCVyQGGJBIys2sRY93noP+IFDU98RJ+021mXa24haoYRbygvxZ42ntVvGMJ1uR+IPfqTs2aZbVwCGXPiXNlTCbIGtb2C9DpWzujuf4UQqOAKKX88OvApdssgxwMSCfDBJLKvjuH2NrVGCd27h2wHGByPeRpdf6yPEqPT90l6VwbcNjtHwqcM51Zys4FmV60wBBF56kOmuKBGpulRB2+ar5MTLj2QFOp5pc+oC7OLlJWM/wrUxVMJ4dKS0KGKsqL/GpuhFpW3fmyAhxmIiJczyqfKrENU6+3836Rylu4ZNdV1dQ6FKb8= ceph-79754014-0328-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T17:25:18.538 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDitixWwEzjIanHWKHEIREwnhwxWLgffJkEk79ZMEaBm106qwF+2ZHM21Y8SfEpW6emkfVuz79bion9coU/R7Xai6Tf6fVErXSQOTrnWpvS89xFOrRGvy4H4ZhWbMrno3syHiLkgdp8Jf7A9tx6T6SdqvbOnmqhG7gRnNYLRLroRlot09Jwgr81L0RrGKNMYm+LgD1Ysirbrj4/cCVyQGGJBIys2sRY93noP+IFDU98RJ+021mXa24haoYRbygvxZ42ntVvGMJ1uR+IPfqTs2aZbVwCGXPiXNlTCbIGtb2C9DpWzujuf4UQqOAKKX88OvApdssgxwMSCfDBJLKvjuH2NrVGCd27h2wHGByPeRpdf6yPEqPT90l6VwbcNjtHwqcM51Zys4FmV60wBBF56kOmuKBGpulRB2+ar5MTLj2QFOp5pc+oC7OLlJWM/wrUxVMJ4dKS0KGKsqL/GpuhFpW3fmyAhxmIiJczyqfKrENU6+3836Rylu4ZNdV1dQ6FKb8= ceph-79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:25:18.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T17:25:18.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:18 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1976974201' entity='client.admin' 2024-04-25T17:25:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:25:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 -- 172.21.15.98:0/807947514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90ac100770 msgr2=0x7f90ac100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 --2- 172.21.15.98:0/807947514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90ac100770 0x7f90ac100b90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009910 tx=0x7f90a8009c20 comp rx=0 tx=0).stop 2024-04-25T17:25:21.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 -- 172.21.15.98:0/807947514 shutdown_connections 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 --2- 172.21.15.98:0/807947514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90ac100770 0x7f90ac100b90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 -- 172.21.15.98:0/807947514 >> 172.21.15.98:0/807947514 conn(0x7f90ac0fc0f0 msgr2=0x7f90ac0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 -- 172.21.15.98:0/807947514 shutdown_connections 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 -- 172.21.15.98:0/807947514 wait complete. 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 Processor -- start 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 -- start start 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90ac100770 0x7f90ac197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.273+0000 7f90b4398700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f90ac1083f0 con 0x7f90ac100770 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.274+0000 7f90b2134700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90ac100770 0x7f90ac197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.274+0000 7f90b2134700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90ac100770 0x7f90ac197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46756/0 (socket says 172.21.15.98:46756) 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.274+0000 7f90b2134700 1 -- 172.21.15.98:0/116332657 learned_addr learned my addr 172.21.15.98:0/116332657 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.274+0000 7f90b2134700 1 -- 172.21.15.98:0/116332657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90ac198040 con 0x7f90ac100770 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.274+0000 7f90b2134700 1 --2- 172.21.15.98:0/116332657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90ac100770 0x7f90ac197b00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f90a8005800 tx=0x7f90a8009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.274+0000 7f90a37fe700 1 -- 172.21.15.98:0/116332657 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90a8027070 con 0x7f90ac100770 2024-04-25T17:25:21.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.274+0000 7f90a37fe700 1 -- 172.21.15.98:0/116332657 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90a800be10 con 0x7f90ac100770 2024-04-25T17:25:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.274+0000 7f90b4398700 1 -- 172.21.15.98:0/116332657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90ac1982d0 con 0x7f90ac100770 2024-04-25T17:25:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.274+0000 7f90a37fe700 1 -- 172.21.15.98:0/116332657 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90a800f460 con 0x7f90ac100770 2024-04-25T17:25:21.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.274+0000 7f90b4398700 1 -- 172.21.15.98:0/116332657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90ac19afc0 con 0x7f90ac100770 2024-04-25T17:25:21.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.275+0000 7f90a37fe700 1 -- 172.21.15.98:0/116332657 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f90a802a070 con 0x7f90ac100770 2024-04-25T17:25:21.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.275+0000 7f90b4398700 1 -- 172.21.15.98:0/116332657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90ac108b70 con 0x7f90ac100770 2024-04-25T17:25:21.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.275+0000 7f90a37fe700 1 --2- 172.21.15.98:0/116332657 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9098039650 0x7f909803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:21.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.275+0000 7f90a37fe700 1 -- 172.21.15.98:0/116332657 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f90a801e030 con 0x7f90ac100770 2024-04-25T17:25:21.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.275+0000 7f90b1933700 1 --2- 172.21.15.98:0/116332657 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9098039650 0x7f909803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:21.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.276+0000 7f90b1933700 1 --2- 172.21.15.98:0/116332657 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9098039650 0x7f909803bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f909c006fd0 tx=0x7f909c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:21.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.280+0000 7f90a37fe700 1 -- 172.21.15.98:0/116332657 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90a8013070 con 0x7f90ac100770 2024-04-25T17:25:21.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.481+0000 7f90b4398700 1 -- 172.21.15.98:0/116332657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f90ac02e090 con 0x7f90ac100770 2024-04-25T17:25:21.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.484+0000 7f90a37fe700 1 -- 172.21.15.98:0/116332657 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f90a8031070 con 0x7f90ac100770 2024-04-25T17:25:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.486+0000 7f90b4398700 1 -- 172.21.15.98:0/116332657 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9098039650 msgr2=0x7f909803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:21.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.486+0000 7f90b4398700 1 --2- 172.21.15.98:0/116332657 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9098039650 0x7f909803bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f909c006fd0 tx=0x7f909c006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.486+0000 7f90b4398700 1 -- 172.21.15.98:0/116332657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90ac100770 msgr2=0x7f90ac197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.486+0000 7f90b4398700 1 --2- 172.21.15.98:0/116332657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90ac100770 0x7f90ac197b00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f90a8005800 tx=0x7f90a8009f60 comp rx=0 tx=0).stop 2024-04-25T17:25:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.486+0000 7f90b4398700 1 -- 172.21.15.98:0/116332657 shutdown_connections 2024-04-25T17:25:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.486+0000 7f90b4398700 1 --2- 172.21.15.98:0/116332657 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9098039650 0x7f909803bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.486+0000 7f90b4398700 1 --2- 172.21.15.98:0/116332657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90ac100770 0x7f90ac197b00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.486+0000 7f90b4398700 1 -- 172.21.15.98:0/116332657 >> 172.21.15.98:0/116332657 conn(0x7f90ac0fc0f0 msgr2=0x7f90ac0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.486+0000 7f90b4398700 1 -- 172.21.15.98:0/116332657 shutdown_connections 2024-04-25T17:25:21.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:21.486+0000 7f90b4398700 1 -- 172.21.15.98:0/116332657 wait complete. 2024-04-25T17:25:22.132 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T17:25:22.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T17:25:22.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:22 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/116332657' entity='client.admin' 2024-04-25T17:25:22.818 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:25:24.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:24.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:24.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:25:24.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:24.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:25:24.053 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.293+0000 7f15239ec700 1 -- 172.21.15.98:0/3022142660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f151c103ad0 msgr2=0x7f151c103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.293+0000 7f15239ec700 1 --2- 172.21.15.98:0/3022142660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f151c103ad0 0x7f151c103ef0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1518009910 tx=0x7f1518009c20 comp rx=0 tx=0).stop 2024-04-25T17:25:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.293+0000 7f15239ec700 1 -- 172.21.15.98:0/3022142660 shutdown_connections 2024-04-25T17:25:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.293+0000 7f15239ec700 1 --2- 172.21.15.98:0/3022142660 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f151c103ad0 0x7f151c103ef0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.293+0000 7f15239ec700 1 -- 172.21.15.98:0/3022142660 >> 172.21.15.98:0/3022142660 conn(0x7f151c0fc0f0 msgr2=0x7f151c0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.293+0000 7f15239ec700 1 -- 172.21.15.98:0/3022142660 shutdown_connections 2024-04-25T17:25:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.293+0000 7f15239ec700 1 -- 172.21.15.98:0/3022142660 wait complete. 2024-04-25T17:25:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.294+0000 7f15239ec700 1 Processor -- start 2024-04-25T17:25:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.294+0000 7f15239ec700 1 -- start start 2024-04-25T17:25:24.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.294+0000 7f15239ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f151c103ad0 0x7f151c1030f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.294+0000 7f15239ec700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f151c1083d0 con 0x7f151c103ad0 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.294+0000 7f1521788700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f151c103ad0 0x7f151c1030f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.294+0000 7f1521788700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f151c103ad0 0x7f151c1030f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33002/0 (socket says 172.21.15.98:33002) 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.294+0000 7f1521788700 1 -- 172.21.15.98:0/1797575206 learned_addr learned my addr 172.21.15.98:0/1797575206 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.294+0000 7f1521788700 1 -- 172.21.15.98:0/1797575206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151c103630 con 0x7f151c103ad0 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.295+0000 7f1521788700 1 --2- 172.21.15.98:0/1797575206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f151c103ad0 0x7f151c1030f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1518005800 tx=0x7f1518009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.295+0000 7f15127fc700 1 -- 172.21.15.98:0/1797575206 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1518027070 con 0x7f151c103ad0 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.295+0000 7f15239ec700 1 -- 172.21.15.98:0/1797575206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151c101820 con 0x7f151c103ad0 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.295+0000 7f15127fc700 1 -- 172.21.15.98:0/1797575206 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f151800be10 con 0x7f151c103ad0 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.295+0000 7f15127fc700 1 -- 172.21.15.98:0/1797575206 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f151800f460 con 0x7f151c103ad0 2024-04-25T17:25:24.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.295+0000 7f15239ec700 1 -- 172.21.15.98:0/1797575206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f151c101d60 con 0x7f151c103ad0 2024-04-25T17:25:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.295+0000 7f15127fc700 1 -- 172.21.15.98:0/1797575206 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f151802a070 con 0x7f151c103ad0 2024-04-25T17:25:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.296+0000 7f15127fc700 1 --2- 172.21.15.98:0/1797575206 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1508039650 0x7f150803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.296+0000 7f15239ec700 1 -- 172.21.15.98:0/1797575206 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f151c108b50 con 0x7f151c103ad0 2024-04-25T17:25:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.296+0000 7f15127fc700 1 -- 172.21.15.98:0/1797575206 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f151801e030 con 0x7f151c103ad0 2024-04-25T17:25:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.296+0000 7f1520f87700 1 --2- 172.21.15.98:0/1797575206 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1508039650 0x7f150803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:24.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.296+0000 7f1520f87700 1 --2- 172.21.15.98:0/1797575206 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1508039650 0x7f150803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f150c006fd0 tx=0x7f150c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:24.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.301+0000 7f15127fc700 1 -- 172.21.15.98:0/1797575206 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1518013070 con 0x7f151c103ad0 2024-04-25T17:25:24.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.501+0000 7f15239ec700 1 -- 172.21.15.98:0/1797575206 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f151c190f50 con 0x7f1508039650 2024-04-25T17:25:24.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.504+0000 7f15127fc700 1 -- 172.21.15.98:0/1797575206 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f151c190f50 con 0x7f1508039650 2024-04-25T17:25:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.506+0000 7f15239ec700 1 -- 172.21.15.98:0/1797575206 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1508039650 msgr2=0x7f150803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.506+0000 7f15239ec700 1 --2- 172.21.15.98:0/1797575206 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1508039650 0x7f150803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f150c006fd0 tx=0x7f150c006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.506+0000 7f15239ec700 1 -- 172.21.15.98:0/1797575206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f151c103ad0 msgr2=0x7f151c1030f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.506+0000 7f15239ec700 1 --2- 172.21.15.98:0/1797575206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f151c103ad0 0x7f151c1030f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1518005800 tx=0x7f1518009f60 comp rx=0 tx=0).stop 2024-04-25T17:25:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.506+0000 7f15239ec700 1 -- 172.21.15.98:0/1797575206 shutdown_connections 2024-04-25T17:25:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.506+0000 7f15239ec700 1 --2- 172.21.15.98:0/1797575206 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1508039650 0x7f150803bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.506+0000 7f15239ec700 1 --2- 172.21.15.98:0/1797575206 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f151c103ad0 0x7f151c1030f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.506+0000 7f15239ec700 1 -- 172.21.15.98:0/1797575206 >> 172.21.15.98:0/1797575206 conn(0x7f151c0fc0f0 msgr2=0x7f151c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.507+0000 7f15239ec700 1 -- 172.21.15.98:0/1797575206 shutdown_connections 2024-04-25T17:25:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:24.507+0000 7f15239ec700 1 -- 172.21.15.98:0/1797575206 wait complete. 2024-04-25T17:25:25.233 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-04-25T17:25:25.233 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:25:25.233 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-04-25T17:25:25.252 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:25:25.252 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:25:25.310 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-04-25T17:25:25.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch host add smithi173 2024-04-25T17:25:25.555 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:25:25.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:25:25.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:25.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:25:25.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:25:25.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:25:25.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T17:25:25.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:25:25.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:25:25.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:25.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:25.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:26.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:26 smithi098 ceph-mon[29192]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:25:26.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.966+0000 7f2023861700 1 -- 172.21.15.98:0/3257516093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c103610 msgr2=0x7f201c1039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.966+0000 7f2023861700 1 --2- 172.21.15.98:0/3257516093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c103610 0x7f201c1039f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2018009910 tx=0x7f2018009c20 comp rx=0 tx=0).stop 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.966+0000 7f2023861700 1 -- 172.21.15.98:0/3257516093 shutdown_connections 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.966+0000 7f2023861700 1 --2- 172.21.15.98:0/3257516093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c103610 0x7f201c1039f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.966+0000 7f2023861700 1 -- 172.21.15.98:0/3257516093 >> 172.21.15.98:0/3257516093 conn(0x7f201c0fce40 msgr2=0x7f201c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.966+0000 7f2023861700 1 -- 172.21.15.98:0/3257516093 shutdown_connections 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.966+0000 7f2023861700 1 -- 172.21.15.98:0/3257516093 wait complete. 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.966+0000 7f2023861700 1 Processor -- start 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.966+0000 7f2023861700 1 -- start start 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f2023861700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c103610 0x7f201c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f2023861700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f201c10c9b0 con 0x7f201c103610 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f20215fd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c103610 0x7f201c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f20215fd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c103610 0x7f201c197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33026/0 (socket says 172.21.15.98:33026) 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f20215fd700 1 -- 172.21.15.98:0/3592076561 learned_addr learned my addr 172.21.15.98:0/3592076561 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f20215fd700 1 -- 172.21.15.98:0/3592076561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c198460 con 0x7f201c103610 2024-04-25T17:25:26.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f20215fd700 1 --2- 172.21.15.98:0/3592076561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c103610 0x7f201c197f20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f20180056d0 tx=0x7f20180057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:26.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f20127fc700 1 -- 172.21.15.98:0/3592076561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2018026070 con 0x7f201c103610 2024-04-25T17:25:26.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f20127fc700 1 -- 172.21.15.98:0/3592076561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2018004370 con 0x7f201c103610 2024-04-25T17:25:26.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f2023861700 1 -- 172.21.15.98:0/3592076561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201c1986f0 con 0x7f201c103610 2024-04-25T17:25:26.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f20127fc700 1 -- 172.21.15.98:0/3592076561 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20180039b0 con 0x7f201c103610 2024-04-25T17:25:26.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.967+0000 7f2023861700 1 -- 172.21.15.98:0/3592076561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201c198b50 con 0x7f201c103610 2024-04-25T17:25:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.968+0000 7f20127fc700 1 -- 172.21.15.98:0/3592076561 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2018029070 con 0x7f201c103610 2024-04-25T17:25:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.968+0000 7f2023861700 1 -- 172.21.15.98:0/3592076561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201c06b190 con 0x7f201c103610 2024-04-25T17:25:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.968+0000 7f20127fc700 1 --2- 172.21.15.98:0/3592076561 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f20080396b0 0x7f200803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.968+0000 7f20127fc700 1 -- 172.21.15.98:0/3592076561 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f201801e070 con 0x7f201c103610 2024-04-25T17:25:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.968+0000 7f2020dfc700 1 --2- 172.21.15.98:0/3592076561 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f20080396b0 0x7f200803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:26.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.969+0000 7f2020dfc700 1 --2- 172.21.15.98:0/3592076561 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f20080396b0 0x7f200803bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f200c006fd0 tx=0x7f200c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:26.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:26.973+0000 7f20127fc700 1 -- 172.21.15.98:0/3592076561 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2018013070 con 0x7f201c103610 2024-04-25T17:25:27.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:27.177+0000 7f2023861700 1 -- 172.21.15.98:0/3592076561 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) -- 0x7f201c199230 con 0x7f20080396b0 2024-04-25T17:25:28.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:28 smithi098 ceph-mon[29192]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:25:28.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:28 smithi098 ceph-mon[29192]: Deploying cephadm binary to smithi173 2024-04-25T17:25:29.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:29 smithi098 ceph-mon[29192]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:31.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.326+0000 7f20127fc700 1 -- 172.21.15.98:0/3592076561 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f201c199230 con 0x7f20080396b0 2024-04-25T17:25:31.328 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-04-25T17:25:31.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.329+0000 7f2023861700 1 -- 172.21.15.98:0/3592076561 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f20080396b0 msgr2=0x7f200803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:31.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.329+0000 7f2023861700 1 --2- 172.21.15.98:0/3592076561 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f20080396b0 0x7f200803bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f200c006fd0 tx=0x7f200c006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:31.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.329+0000 7f2023861700 1 -- 172.21.15.98:0/3592076561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c103610 msgr2=0x7f201c197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:31.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.329+0000 7f2023861700 1 --2- 172.21.15.98:0/3592076561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c103610 0x7f201c197f20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f20180056d0 tx=0x7f20180057b0 comp rx=0 tx=0).stop 2024-04-25T17:25:31.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.329+0000 7f2023861700 1 -- 172.21.15.98:0/3592076561 shutdown_connections 2024-04-25T17:25:31.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.329+0000 7f2023861700 1 --2- 172.21.15.98:0/3592076561 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f20080396b0 0x7f200803bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:31.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.329+0000 7f2023861700 1 --2- 172.21.15.98:0/3592076561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f201c103610 0x7f201c197f20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:31.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.329+0000 7f2023861700 1 -- 172.21.15.98:0/3592076561 >> 172.21.15.98:0/3592076561 conn(0x7f201c0fce40 msgr2=0x7f201c1043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:31.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.329+0000 7f2023861700 1 -- 172.21.15.98:0/3592076561 shutdown_connections 2024-04-25T17:25:31.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:31.329+0000 7f2023861700 1 -- 172.21.15.98:0/3592076561 wait complete. 2024-04-25T17:25:31.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T17:25:32.170 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:25:32.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:31 smithi098 ceph-mon[29192]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:32.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:32.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:25:32.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:33.149 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:32 smithi098 ceph-mon[29192]: Added host smithi173 2024-04-25T17:25:33.149 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:32 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.745+0000 7f49913a4700 1 -- 172.21.15.98:0/568610900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f498c1036d0 msgr2=0x7f498c103ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.745+0000 7f49913a4700 1 --2- 172.21.15.98:0/568610900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f498c1036d0 0x7f498c103ab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4974009910 tx=0x7f4974009c20 comp rx=0 tx=0).stop 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.745+0000 7f49913a4700 1 -- 172.21.15.98:0/568610900 shutdown_connections 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.745+0000 7f49913a4700 1 --2- 172.21.15.98:0/568610900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f498c1036d0 0x7f498c103ab0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.745+0000 7f49913a4700 1 -- 172.21.15.98:0/568610900 >> 172.21.15.98:0/568610900 conn(0x7f498c0fce40 msgr2=0x7f498c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.745+0000 7f49913a4700 1 -- 172.21.15.98:0/568610900 shutdown_connections 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.745+0000 7f49913a4700 1 -- 172.21.15.98:0/568610900 wait complete. 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.746+0000 7f49913a4700 1 Processor -- start 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.746+0000 7f49913a4700 1 -- start start 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.746+0000 7f49913a4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f498c1036d0 0x7f498c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.746+0000 7f49913a4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f498c10c990 con 0x7f498c1036d0 2024-04-25T17:25:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.746+0000 7f498affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f498c1036d0 0x7f498c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.746+0000 7f498affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f498c1036d0 0x7f498c197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34164/0 (socket says 172.21.15.98:34164) 2024-04-25T17:25:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.746+0000 7f498affd700 1 -- 172.21.15.98:0/3741033762 learned_addr learned my addr 172.21.15.98:0/3741033762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.747+0000 7f498affd700 1 -- 172.21.15.98:0/3741033762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498c1983f0 con 0x7f498c1036d0 2024-04-25T17:25:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.747+0000 7f498affd700 1 --2- 172.21.15.98:0/3741033762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f498c1036d0 0x7f498c197eb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f49740056d0 tx=0x7f49740057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.747+0000 7f4983fff700 1 -- 172.21.15.98:0/3741033762 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4974026070 con 0x7f498c1036d0 2024-04-25T17:25:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.747+0000 7f4983fff700 1 -- 172.21.15.98:0/3741033762 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4974004370 con 0x7f498c1036d0 2024-04-25T17:25:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.747+0000 7f49913a4700 1 -- 172.21.15.98:0/3741033762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498c198680 con 0x7f498c1036d0 2024-04-25T17:25:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.747+0000 7f4983fff700 1 -- 172.21.15.98:0/3741033762 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49740039b0 con 0x7f498c1036d0 2024-04-25T17:25:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.747+0000 7f49913a4700 1 -- 172.21.15.98:0/3741033762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f498c198ae0 con 0x7f498c1036d0 2024-04-25T17:25:33.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.748+0000 7f4983fff700 1 -- 172.21.15.98:0/3741033762 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4974029070 con 0x7f498c1036d0 2024-04-25T17:25:33.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.748+0000 7f4983fff700 1 --2- 172.21.15.98:0/3741033762 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f49780396b0 0x7f497803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:33.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.748+0000 7f49913a4700 1 -- 172.21.15.98:0/3741033762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f498c06b070 con 0x7f498c1036d0 2024-04-25T17:25:33.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.748+0000 7f4983fff700 1 -- 172.21.15.98:0/3741033762 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f497401e070 con 0x7f498c1036d0 2024-04-25T17:25:33.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.748+0000 7f498a7fc700 1 --2- 172.21.15.98:0/3741033762 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f49780396b0 0x7f497803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:33.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.748+0000 7f498a7fc700 1 --2- 172.21.15.98:0/3741033762 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f49780396b0 0x7f497803bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f497c006fd0 tx=0x7f497c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.753+0000 7f4983fff700 1 -- 172.21.15.98:0/3741033762 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4974013070 con 0x7f498c1036d0 2024-04-25T17:25:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.957+0000 7f49913a4700 1 -- 172.21.15.98:0/3741033762 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f498c199210 con 0x7f49780396b0 2024-04-25T17:25:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.958+0000 7f4983fff700 1 -- 172.21.15.98:0/3741033762 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f498c199210 con 0x7f49780396b0 2024-04-25T17:25:33.959 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:33.959 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-04-25T17:25:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.960+0000 7f49913a4700 1 -- 172.21.15.98:0/3741033762 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f49780396b0 msgr2=0x7f497803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.960+0000 7f49913a4700 1 --2- 172.21.15.98:0/3741033762 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f49780396b0 0x7f497803bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f497c006fd0 tx=0x7f497c006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.960+0000 7f49913a4700 1 -- 172.21.15.98:0/3741033762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f498c1036d0 msgr2=0x7f498c197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.960+0000 7f49913a4700 1 --2- 172.21.15.98:0/3741033762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f498c1036d0 0x7f498c197eb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f49740056d0 tx=0x7f49740057b0 comp rx=0 tx=0).stop 2024-04-25T17:25:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.960+0000 7f49913a4700 1 -- 172.21.15.98:0/3741033762 shutdown_connections 2024-04-25T17:25:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.960+0000 7f49913a4700 1 --2- 172.21.15.98:0/3741033762 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f49780396b0 0x7f497803bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.960+0000 7f49913a4700 1 --2- 172.21.15.98:0/3741033762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f498c1036d0 0x7f498c197eb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.960+0000 7f49913a4700 1 -- 172.21.15.98:0/3741033762 >> 172.21.15.98:0/3741033762 conn(0x7f498c0fce40 msgr2=0x7f498c104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.960+0000 7f49913a4700 1 -- 172.21.15.98:0/3741033762 shutdown_connections 2024-04-25T17:25:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:33.960+0000 7f49913a4700 1 -- 172.21.15.98:0/3741033762 wait complete. 2024-04-25T17:25:34.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:33 smithi098 ceph-mon[29192]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:34.541 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-04-25T17:25:34.541 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:25:34.541 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-04-25T17:25:34.561 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:25:34.561 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:25:34.618 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-04-25T17:25:34.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch host add smithi203 2024-04-25T17:25:34.850 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:25:35.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:34 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:35.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:34 smithi098 ceph-mon[29192]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:25:36.133 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:35 smithi098 ceph-mon[29192]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.342+0000 7f5fc6104700 1 -- 172.21.15.98:0/1470711411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fc0103610 msgr2=0x7f5fc01039f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.342+0000 7f5fc6104700 1 --2- 172.21.15.98:0/1470711411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fc0103610 0x7f5fc01039f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0009910 tx=0x7f5fb0009c20 comp rx=0 tx=0).stop 2024-04-25T17:25:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.342+0000 7f5fc6104700 1 -- 172.21.15.98:0/1470711411 shutdown_connections 2024-04-25T17:25:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.342+0000 7f5fc6104700 1 --2- 172.21.15.98:0/1470711411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fc0103610 0x7f5fc01039f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.342+0000 7f5fc6104700 1 -- 172.21.15.98:0/1470711411 >> 172.21.15.98:0/1470711411 conn(0x7f5fc00fce40 msgr2=0x7f5fc00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.342+0000 7f5fc6104700 1 -- 172.21.15.98:0/1470711411 shutdown_connections 2024-04-25T17:25:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.342+0000 7f5fc6104700 1 -- 172.21.15.98:0/1470711411 wait complete. 2024-04-25T17:25:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.342+0000 7f5fc6104700 1 Processor -- start 2024-04-25T17:25:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.343+0000 7f5fc6104700 1 -- start start 2024-04-25T17:25:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.343+0000 7f5fc6104700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fc0103610 0x7f5fc0197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.343+0000 7f5fc6104700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc010c9b0 con 0x7f5fc0103610 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.343+0000 7f5fbf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fc0103610 0x7f5fc0197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.343+0000 7f5fbf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fc0103610 0x7f5fc0197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34194/0 (socket says 172.21.15.98:34194) 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.343+0000 7f5fbf7fe700 1 -- 172.21.15.98:0/401775138 learned_addr learned my addr 172.21.15.98:0/401775138 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.343+0000 7f5fbf7fe700 1 -- 172.21.15.98:0/401775138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc0198440 con 0x7f5fc0103610 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.343+0000 7f5fbf7fe700 1 --2- 172.21.15.98:0/401775138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fc0103610 0x7f5fc0197f00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00056d0 tx=0x7f5fb00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.343+0000 7f5fbcff9700 1 -- 172.21.15.98:0/401775138 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0026070 con 0x7f5fc0103610 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.344+0000 7f5fbcff9700 1 -- 172.21.15.98:0/401775138 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fb0004370 con 0x7f5fc0103610 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.344+0000 7f5fc6104700 1 -- 172.21.15.98:0/401775138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc01986d0 con 0x7f5fc0103610 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.344+0000 7f5fbcff9700 1 -- 172.21.15.98:0/401775138 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb00039b0 con 0x7f5fc0103610 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.344+0000 7f5fc6104700 1 -- 172.21.15.98:0/401775138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc0198b30 con 0x7f5fc0103610 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.344+0000 7f5fbcff9700 1 -- 172.21.15.98:0/401775138 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5fb0029070 con 0x7f5fc0103610 2024-04-25T17:25:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.344+0000 7f5fc6104700 1 -- 172.21.15.98:0/401775138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fc006b210 con 0x7f5fc0103610 2024-04-25T17:25:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.345+0000 7f5fbcff9700 1 --2- 172.21.15.98:0/401775138 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5fa80396b0 0x7f5fa803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.345+0000 7f5fbcff9700 1 -- 172.21.15.98:0/401775138 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5fb001e070 con 0x7f5fc0103610 2024-04-25T17:25:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.345+0000 7f5fbeffd700 1 --2- 172.21.15.98:0/401775138 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5fa80396b0 0x7f5fa803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.345+0000 7f5fbeffd700 1 --2- 172.21.15.98:0/401775138 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5fa80396b0 0x7f5fa803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4006fd0 tx=0x7f5fb4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:36.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.350+0000 7f5fbcff9700 1 -- 172.21.15.98:0/401775138 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5fb0013070 con 0x7f5fc0103610 2024-04-25T17:25:36.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:36.555+0000 7f5fc6104700 1 -- 172.21.15.98:0/401775138 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}) -- 0x7f5fc01992d0 con 0x7f5fa80396b0 2024-04-25T17:25:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:37 smithi098 ceph-mon[29192]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:25:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:37 smithi098 ceph-mon[29192]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:25:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:25:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:25:39.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:38 smithi098 ceph-mon[29192]: Deploying cephadm binary to smithi203 2024-04-25T17:25:39.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:38 smithi098 ceph-mon[29192]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T17:25:39.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:38 smithi098 ceph-mon[29192]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:25:39.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:38 smithi098 ceph-mon[29192]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:25:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:39 smithi098 ceph-mon[29192]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:40.685 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi203' with addr '172.21.15.203' 2024-04-25T17:25:40.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.683+0000 7f5fbcff9700 1 -- 172.21.15.98:0/401775138 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f5fc01992d0 con 0x7f5fa80396b0 2024-04-25T17:25:40.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.686+0000 7f5fc6104700 1 -- 172.21.15.98:0/401775138 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5fa80396b0 msgr2=0x7f5fa803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:40.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.686+0000 7f5fc6104700 1 --2- 172.21.15.98:0/401775138 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5fa80396b0 0x7f5fa803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4006fd0 tx=0x7f5fb4006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:40.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.686+0000 7f5fc6104700 1 -- 172.21.15.98:0/401775138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fc0103610 msgr2=0x7f5fc0197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:40.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.686+0000 7f5fc6104700 1 --2- 172.21.15.98:0/401775138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fc0103610 0x7f5fc0197f00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00056d0 tx=0x7f5fb00057b0 comp rx=0 tx=0).stop 2024-04-25T17:25:40.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.686+0000 7f5fc6104700 1 -- 172.21.15.98:0/401775138 shutdown_connections 2024-04-25T17:25:40.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.686+0000 7f5fc6104700 1 --2- 172.21.15.98:0/401775138 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5fa80396b0 0x7f5fa803bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:40.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.686+0000 7f5fc6104700 1 --2- 172.21.15.98:0/401775138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fc0103610 0x7f5fc0197f00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:40.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.686+0000 7f5fc6104700 1 -- 172.21.15.98:0/401775138 >> 172.21.15.98:0/401775138 conn(0x7f5fc00fce40 msgr2=0x7f5fc01043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:40.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.686+0000 7f5fc6104700 1 -- 172.21.15.98:0/401775138 shutdown_connections 2024-04-25T17:25:40.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:40.686+0000 7f5fc6104700 1 -- 172.21.15.98:0/401775138 wait complete. 2024-04-25T17:25:41.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T17:25:41.553 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:25:41.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:41 smithi098 ceph-mon[29192]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:25:41.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:41.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:41.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:41.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:41.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:41 smithi098 ceph-mon[29192]: Added host smithi203 2024-04-25T17:25:41.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:25:41.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:41 smithi098 ceph-mon[29192]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:41.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:43.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.121+0000 7f1b60fe3700 1 -- 172.21.15.98:0/1851611205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c101590 msgr2=0x7f1b5c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:43.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.121+0000 7f1b60fe3700 1 --2- 172.21.15.98:0/1851611205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c101590 0x7f1b5c101970 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1b50009910 tx=0x7f1b50009c20 comp rx=0 tx=0).stop 2024-04-25T17:25:43.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.121+0000 7f1b60fe3700 1 -- 172.21.15.98:0/1851611205 shutdown_connections 2024-04-25T17:25:43.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.121+0000 7f1b60fe3700 1 --2- 172.21.15.98:0/1851611205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c101590 0x7f1b5c101970 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:43.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.121+0000 7f1b60fe3700 1 -- 172.21.15.98:0/1851611205 >> 172.21.15.98:0/1851611205 conn(0x7f1b5c0fce40 msgr2=0x7f1b5c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:43.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.121+0000 7f1b60fe3700 1 -- 172.21.15.98:0/1851611205 shutdown_connections 2024-04-25T17:25:43.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.122+0000 7f1b60fe3700 1 -- 172.21.15.98:0/1851611205 wait complete. 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.122+0000 7f1b60fe3700 1 Processor -- start 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.122+0000 7f1b60fe3700 1 -- start start 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.122+0000 7f1b60fe3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c101590 0x7f1b5c197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.122+0000 7f1b60fe3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c10b810 con 0x7f1b5c101590 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.122+0000 7f1b5a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c101590 0x7f1b5c197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.123+0000 7f1b5a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c101590 0x7f1b5c197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34220/0 (socket says 172.21.15.98:34220) 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.123+0000 7f1b5a59c700 1 -- 172.21.15.98:0/542592474 learned_addr learned my addr 172.21.15.98:0/542592474 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.123+0000 7f1b5a59c700 1 -- 172.21.15.98:0/542592474 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c1983c0 con 0x7f1b5c101590 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.123+0000 7f1b5a59c700 1 --2- 172.21.15.98:0/542592474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c101590 0x7f1b5c197e80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1b500056d0 tx=0x7f1b500057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.123+0000 7f1b4b7fe700 1 -- 172.21.15.98:0/542592474 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b50026070 con 0x7f1b5c101590 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.123+0000 7f1b60fe3700 1 -- 172.21.15.98:0/542592474 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5c1986b0 con 0x7f1b5c101590 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.123+0000 7f1b4b7fe700 1 -- 172.21.15.98:0/542592474 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b5000bba0 con 0x7f1b5c101590 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.123+0000 7f1b4b7fe700 1 -- 172.21.15.98:0/542592474 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b50003920 con 0x7f1b5c101590 2024-04-25T17:25:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.123+0000 7f1b60fe3700 1 -- 172.21.15.98:0/542592474 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5c19c750 con 0x7f1b5c101590 2024-04-25T17:25:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.124+0000 7f1b4b7fe700 1 -- 172.21.15.98:0/542592474 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1b50029070 con 0x7f1b5c101590 2024-04-25T17:25:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.124+0000 7f1b60fe3700 1 -- 172.21.15.98:0/542592474 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5c102a70 con 0x7f1b5c101590 2024-04-25T17:25:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.124+0000 7f1b4b7fe700 1 --2- 172.21.15.98:0/542592474 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1b44039650 0x7f1b4403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.124+0000 7f1b4b7fe700 1 -- 172.21.15.98:0/542592474 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1b5001e070 con 0x7f1b5c101590 2024-04-25T17:25:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.124+0000 7f1b59d9b700 1 --2- 172.21.15.98:0/542592474 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1b44039650 0x7f1b4403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.125+0000 7f1b59d9b700 1 --2- 172.21.15.98:0/542592474 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1b44039650 0x7f1b4403bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c006fd0 tx=0x7f1b4c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:43.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.129+0000 7f1b4b7fe700 1 -- 172.21.15.98:0/542592474 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b50013070 con 0x7f1b5c101590 2024-04-25T17:25:43.132 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:43.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.335+0000 7f1b60fe3700 1 -- 172.21.15.98:0/542592474 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1b5c1991e0 con 0x7f1b44039650 2024-04-25T17:25:43.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.336+0000 7f1b4b7fe700 1 -- 172.21.15.98:0/542592474 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f1b5c1991e0 con 0x7f1b44039650 2024-04-25T17:25:43.337 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:25:43.337 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}, {"addr": "172.21.15.203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-04-25T17:25:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.338+0000 7f1b60fe3700 1 -- 172.21.15.98:0/542592474 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1b44039650 msgr2=0x7f1b4403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.338+0000 7f1b60fe3700 1 --2- 172.21.15.98:0/542592474 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1b44039650 0x7f1b4403bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c006fd0 tx=0x7f1b4c006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.338+0000 7f1b60fe3700 1 -- 172.21.15.98:0/542592474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c101590 msgr2=0x7f1b5c197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.338+0000 7f1b60fe3700 1 --2- 172.21.15.98:0/542592474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c101590 0x7f1b5c197e80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1b500056d0 tx=0x7f1b500057b0 comp rx=0 tx=0).stop 2024-04-25T17:25:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.339+0000 7f1b60fe3700 1 -- 172.21.15.98:0/542592474 shutdown_connections 2024-04-25T17:25:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.339+0000 7f1b60fe3700 1 --2- 172.21.15.98:0/542592474 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1b44039650 0x7f1b4403bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.339+0000 7f1b60fe3700 1 --2- 172.21.15.98:0/542592474 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b5c101590 0x7f1b5c197e80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.339+0000 7f1b60fe3700 1 -- 172.21.15.98:0/542592474 >> 172.21.15.98:0/542592474 conn(0x7f1b5c0fce40 msgr2=0x7f1b5c06a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.339+0000 7f1b60fe3700 1 -- 172.21.15.98:0/542592474 shutdown_connections 2024-04-25T17:25:43.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:43.339+0000 7f1b60fe3700 1 -- 172.21.15.98:0/542592474 wait complete. 2024-04-25T17:25:43.963 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T17:25:43.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T17:25:44.213 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:25:44.335 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:43 smithi098 ceph-mon[29192]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:44.335 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:45.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:45 smithi098 ceph-mon[29192]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:25:45.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.728+0000 7f11bdd81700 1 -- 172.21.15.98:0/3667480730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8106ac0 msgr2=0x7f11b8106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:45.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.728+0000 7f11bdd81700 1 --2- 172.21.15.98:0/3667480730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8106ac0 0x7f11b8106ea0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f11a8009910 tx=0x7f11a8009c20 comp rx=0 tx=0).stop 2024-04-25T17:25:45.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.728+0000 7f11bdd81700 1 -- 172.21.15.98:0/3667480730 shutdown_connections 2024-04-25T17:25:45.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.728+0000 7f11bdd81700 1 --2- 172.21.15.98:0/3667480730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8106ac0 0x7f11b8106ea0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:45.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.728+0000 7f11bdd81700 1 -- 172.21.15.98:0/3667480730 >> 172.21.15.98:0/3667480730 conn(0x7f11b8077ac0 msgr2=0x7f11b8077ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:45.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.728+0000 7f11bdd81700 1 -- 172.21.15.98:0/3667480730 shutdown_connections 2024-04-25T17:25:45.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.728+0000 7f11bdd81700 1 -- 172.21.15.98:0/3667480730 wait complete. 2024-04-25T17:25:45.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.729+0000 7f11bdd81700 1 Processor -- start 2024-04-25T17:25:45.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.729+0000 7f11bdd81700 1 -- start start 2024-04-25T17:25:45.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.729+0000 7f11bdd81700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8106ac0 0x7f11b8196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:45.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.729+0000 7f11bdd81700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11b810afb0 con 0x7f11b8106ac0 2024-04-25T17:25:45.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.729+0000 7f11b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8106ac0 0x7f11b8196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:45.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.729+0000 7f11b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8106ac0 0x7f11b8196dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50452/0 (socket says 172.21.15.98:50452) 2024-04-25T17:25:45.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.729+0000 7f11b77fe700 1 -- 172.21.15.98:0/647779492 learned_addr learned my addr 172.21.15.98:0/647779492 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:25:45.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.729+0000 7f11b77fe700 1 -- 172.21.15.98:0/647779492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b8197300 con 0x7f11b8106ac0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.729+0000 7f11b77fe700 1 --2- 172.21.15.98:0/647779492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8106ac0 0x7f11b8196dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f11a80056d0 tx=0x7f11a80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.730+0000 7f11b4ff9700 1 -- 172.21.15.98:0/647779492 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11a8026070 con 0x7f11b8106ac0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.730+0000 7f11b4ff9700 1 -- 172.21.15.98:0/647779492 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11a800bba0 con 0x7f11b8106ac0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.730+0000 7f11bdd81700 1 -- 172.21.15.98:0/647779492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b81975f0 con 0x7f11b8106ac0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.730+0000 7f11b4ff9700 1 -- 172.21.15.98:0/647779492 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11a8003920 con 0x7f11b8106ac0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.730+0000 7f11bdd81700 1 -- 172.21.15.98:0/647779492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b819b690 con 0x7f11b8106ac0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.730+0000 7f11b4ff9700 1 -- 172.21.15.98:0/647779492 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f11a8029070 con 0x7f11b8106ac0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.731+0000 7f11bdd81700 1 -- 172.21.15.98:0/647779492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b8102200 con 0x7f11b8106ac0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.731+0000 7f11b4ff9700 1 --2- 172.21.15.98:0/647779492 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11a00396b0 0x7f11a003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.731+0000 7f11b4ff9700 1 -- 172.21.15.98:0/647779492 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f11a801e070 con 0x7f11b8106ac0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.731+0000 7f11b6ffd700 1 --2- 172.21.15.98:0/647779492 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11a00396b0 0x7f11a003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:45.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.731+0000 7f11b6ffd700 1 --2- 172.21.15.98:0/647779492 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11a00396b0 0x7f11a003bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11ac006fd0 tx=0x7f11ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:45.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.736+0000 7f11b4ff9700 1 -- 172.21.15.98:0/647779492 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f11a8013070 con 0x7f11b8106ac0 2024-04-25T17:25:45.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:45.936+0000 7f11bdd81700 1 -- 172.21.15.98:0/647779492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f11b8198120 con 0x7f11b8106ac0 2024-04-25T17:25:46.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.008+0000 7f11b4ff9700 1 -- 172.21.15.98:0/647779492 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f11a8030090 con 0x7f11b8106ac0 2024-04-25T17:25:46.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.010+0000 7f11bdd81700 1 -- 172.21.15.98:0/647779492 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11a00396b0 msgr2=0x7f11a003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:46.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.010+0000 7f11bdd81700 1 --2- 172.21.15.98:0/647779492 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11a00396b0 0x7f11a003bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11ac006fd0 tx=0x7f11ac006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:46.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.010+0000 7f11bdd81700 1 -- 172.21.15.98:0/647779492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8106ac0 msgr2=0x7f11b8196dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:46.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.010+0000 7f11bdd81700 1 --2- 172.21.15.98:0/647779492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8106ac0 0x7f11b8196dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f11a80056d0 tx=0x7f11a80057b0 comp rx=0 tx=0).stop 2024-04-25T17:25:46.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.010+0000 7f11bdd81700 1 -- 172.21.15.98:0/647779492 shutdown_connections 2024-04-25T17:25:46.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.010+0000 7f11bdd81700 1 --2- 172.21.15.98:0/647779492 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11a00396b0 0x7f11a003bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:46.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.010+0000 7f11bdd81700 1 --2- 172.21.15.98:0/647779492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11b8106ac0 0x7f11b8196dc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:46.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.010+0000 7f11bdd81700 1 -- 172.21.15.98:0/647779492 >> 172.21.15.98:0/647779492 conn(0x7f11b8077ac0 msgr2=0x7f11b80ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:46.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.010+0000 7f11bdd81700 1 -- 172.21.15.98:0/647779492 shutdown_connections 2024-04-25T17:25:46.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:25:46.010+0000 7f11bdd81700 1 -- 172.21.15.98:0/647779492 wait complete. 2024-04-25T17:25:46.012 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-04-25T17:25:46.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:46 smithi098 ceph-mon[29192]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:46.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:46 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/647779492' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T17:25:46.609 INFO:tasks.cephadm:Adding mon.a on smithi098 2024-04-25T17:25:46.610 INFO:tasks.cephadm:Adding mon.b on smithi173 2024-04-25T17:25:46.610 INFO:tasks.cephadm:Adding mon.c on smithi203 2024-04-25T17:25:46.610 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c' 2024-04-25T17:25:46.845 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-25T17:25:46.893 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:25:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:47 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/647779492' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T17:25:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:47 smithi098 ceph-mon[29192]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T17:25:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:25:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:25:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:25:48.380 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:48 smithi098 ceph-mon[29192]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T17:25:48.380 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:48 smithi098 ceph-mon[29192]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:25:48.381 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:48 smithi098 ceph-mon[29192]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:25:48.381 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:48 smithi098 ceph-mon[29192]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:48.381 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:48 smithi098 ceph-mon[29192]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:25:48.381 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:48 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:48.381 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:48 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:48.381 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:48 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:48.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.441+0000 7f5033d4d700 1 -- 172.21.15.203:0/3150935792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c1015b0 msgr2=0x7f502c101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:48.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.441+0000 7f5033d4d700 1 --2- 172.21.15.203:0/3150935792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c1015b0 0x7f502c101990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f501c009910 tx=0x7f501c009c20 comp rx=0 tx=0).stop 2024-04-25T17:25:48.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.441+0000 7f5033d4d700 1 -- 172.21.15.203:0/3150935792 shutdown_connections 2024-04-25T17:25:48.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.441+0000 7f5033d4d700 1 --2- 172.21.15.203:0/3150935792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c1015b0 0x7f502c101990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:48.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.441+0000 7f5033d4d700 1 -- 172.21.15.203:0/3150935792 >> 172.21.15.203:0/3150935792 conn(0x7f502c0fce60 msgr2=0x7f502c0ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:48.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.441+0000 7f5033d4d700 1 -- 172.21.15.203:0/3150935792 shutdown_connections 2024-04-25T17:25:48.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.441+0000 7f5033d4d700 1 -- 172.21.15.203:0/3150935792 wait complete. 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.441+0000 7f5033d4d700 1 Processor -- start 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.442+0000 7f5033d4d700 1 -- start start 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.442+0000 7f5033d4d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c1015b0 0x7f502c196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.442+0000 7f5033d4d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f502c10b830 con 0x7f502c1015b0 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.442+0000 7f5031ae9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c1015b0 0x7f502c196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.442+0000 7f5031ae9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c1015b0 0x7f502c196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.203:47864/0 (socket says 172.21.15.203:47864) 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.442+0000 7f5031ae9700 1 -- 172.21.15.203:0/3809569377 learned_addr learned my addr 172.21.15.203:0/3809569377 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.442+0000 7f5031ae9700 1 -- 172.21.15.203:0/3809569377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502c06b6f0 con 0x7f502c1015b0 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.442+0000 7f5031ae9700 1 --2- 172.21.15.203:0/3809569377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c1015b0 0x7f502c196af0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f501c0056d0 tx=0x7f501c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.443+0000 7f5022ffd700 1 -- 172.21.15.203:0/3809569377 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f501c026070 con 0x7f502c1015b0 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.443+0000 7f5033d4d700 1 -- 172.21.15.203:0/3809569377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f502c197030 con 0x7f502c1015b0 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.443+0000 7f5022ffd700 1 -- 172.21.15.203:0/3809569377 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f501c00bba0 con 0x7f502c1015b0 2024-04-25T17:25:48.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.443+0000 7f5022ffd700 1 -- 172.21.15.203:0/3809569377 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f501c003920 con 0x7f502c1015b0 2024-04-25T17:25:48.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.443+0000 7f5033d4d700 1 -- 172.21.15.203:0/3809569377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f502c1974f0 con 0x7f502c1015b0 2024-04-25T17:25:48.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.444+0000 7f5022ffd700 1 -- 172.21.15.203:0/3809569377 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f501c029070 con 0x7f502c1015b0 2024-04-25T17:25:48.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.444+0000 7f5033d4d700 1 -- 172.21.15.203:0/3809569377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f502c102a90 con 0x7f502c1015b0 2024-04-25T17:25:48.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.444+0000 7f5022ffd700 1 --2- 172.21.15.203:0/3809569377 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5018039650 0x7f501803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:48.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.444+0000 7f5022ffd700 1 -- 172.21.15.203:0/3809569377 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f501c01e070 con 0x7f502c1015b0 2024-04-25T17:25:48.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.444+0000 7f50312e8700 1 --2- 172.21.15.203:0/3809569377 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5018039650 0x7f501803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:48.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.445+0000 7f50312e8700 1 --2- 172.21.15.203:0/3809569377 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5018039650 0x7f501803bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5028006fd0 tx=0x7f5028006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:48.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.453+0000 7f5022ffd700 1 -- 172.21.15.203:0/3809569377 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f501c013070 con 0x7f502c1015b0 2024-04-25T17:25:48.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.661+0000 7f5033d4d700 1 -- 172.21.15.203:0/3809569377 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}) -- 0x7f502c06aee0 con 0x7f5018039650 2024-04-25T17:25:48.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.667+0000 7f5022ffd700 1 -- 172.21.15.203:0/3809569377 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f502c06aee0 con 0x7f5018039650 2024-04-25T17:25:48.669 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-04-25T17:25:48.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.669+0000 7f5033d4d700 1 -- 172.21.15.203:0/3809569377 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5018039650 msgr2=0x7f501803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:48.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.669+0000 7f5033d4d700 1 --2- 172.21.15.203:0/3809569377 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5018039650 0x7f501803bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5028006fd0 tx=0x7f5028006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:48.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.669+0000 7f5033d4d700 1 -- 172.21.15.203:0/3809569377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c1015b0 msgr2=0x7f502c196af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:48.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.669+0000 7f5033d4d700 1 --2- 172.21.15.203:0/3809569377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c1015b0 0x7f502c196af0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f501c0056d0 tx=0x7f501c0057b0 comp rx=0 tx=0).stop 2024-04-25T17:25:48.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.669+0000 7f5033d4d700 1 -- 172.21.15.203:0/3809569377 shutdown_connections 2024-04-25T17:25:48.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.669+0000 7f5033d4d700 1 --2- 172.21.15.203:0/3809569377 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5018039650 0x7f501803bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:48.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.669+0000 7f5033d4d700 1 --2- 172.21.15.203:0/3809569377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f502c1015b0 0x7f502c196af0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:48.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.669+0000 7f5033d4d700 1 -- 172.21.15.203:0/3809569377 >> 172.21.15.203:0/3809569377 conn(0x7f502c0fce60 msgr2=0x7f502c102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:48.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.669+0000 7f5033d4d700 1 -- 172.21.15.203:0/3809569377 shutdown_connections 2024-04-25T17:25:48.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:48.669+0000 7f5033d4d700 1 -- 172.21.15.203:0/3809569377 wait complete. 2024-04-25T17:25:49.253 DEBUG:teuthology.orchestra.run.smithi173:mon.b> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T17:25:49.261 DEBUG:teuthology.orchestra.run.smithi203:mon.c> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T17:25:49.263 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T17:25:49.263 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T17:25:49.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:25:49.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c;count:3 2024-04-25T17:25:49.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:49.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:25:49.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:25:49.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:25:49.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:49.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:25:49.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:25:49.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: Deploying daemon mon.c on smithi203 2024-04-25T17:25:49.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:49 smithi098 ceph-mon[29192]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:50.763 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:25:52.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:51 smithi098 ceph-mon[29192]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:52.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.805+0000 7ff36266e700 1 -- 172.21.15.203:0/129607188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff35c1084e0 msgr2=0x7ff35c1088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:52.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.805+0000 7ff36266e700 1 --2- 172.21.15.203:0/129607188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff35c1084e0 0x7ff35c1088c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009910 tx=0x7ff34c009c20 comp rx=0 tx=0).stop 2024-04-25T17:25:52.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.805+0000 7ff36266e700 1 -- 172.21.15.203:0/129607188 shutdown_connections 2024-04-25T17:25:52.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.805+0000 7ff36266e700 1 --2- 172.21.15.203:0/129607188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff35c1084e0 0x7ff35c1088c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:52.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.805+0000 7ff36266e700 1 -- 172.21.15.203:0/129607188 >> 172.21.15.203:0/129607188 conn(0x7ff35c076ea0 msgr2=0x7ff35c0772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:52.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.805+0000 7ff36266e700 1 -- 172.21.15.203:0/129607188 shutdown_connections 2024-04-25T17:25:52.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.805+0000 7ff36266e700 1 -- 172.21.15.203:0/129607188 wait complete. 2024-04-25T17:25:52.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.805+0000 7ff36266e700 1 Processor -- start 2024-04-25T17:25:52.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.805+0000 7ff36266e700 1 -- start start 2024-04-25T17:25:52.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.806+0000 7ff36266e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff35c1084e0 0x7ff35c101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:52.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.806+0000 7ff36266e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c10c920 con 0x7ff35c1084e0 2024-04-25T17:25:52.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.806+0000 7ff35bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff35c1084e0 0x7ff35c101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:52.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.806+0000 7ff35bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff35c1084e0 0x7ff35c101460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.203:32906/0 (socket says 172.21.15.203:32906) 2024-04-25T17:25:52.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.806+0000 7ff35bfff700 1 -- 172.21.15.203:0/530609591 learned_addr learned my addr 172.21.15.203:0/530609591 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T17:25:52.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.807+0000 7ff35bfff700 1 -- 172.21.15.203:0/530609591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35c103350 con 0x7ff35c1084e0 2024-04-25T17:25:52.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.807+0000 7ff35bfff700 1 --2- 172.21.15.203:0/530609591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff35c1084e0 0x7ff35c101460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0056d0 tx=0x7ff34c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:52.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.807+0000 7ff3597fa700 1 -- 172.21.15.203:0/530609591 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff34c026070 con 0x7ff35c1084e0 2024-04-25T17:25:52.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.807+0000 7ff3597fa700 1 -- 172.21.15.203:0/530609591 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff34c00bc00 con 0x7ff35c1084e0 2024-04-25T17:25:52.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.807+0000 7ff36266e700 1 -- 172.21.15.203:0/530609591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff35c101a00 con 0x7ff35c1084e0 2024-04-25T17:25:52.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.807+0000 7ff3597fa700 1 -- 172.21.15.203:0/530609591 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff34c003920 con 0x7ff35c1084e0 2024-04-25T17:25:52.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.807+0000 7ff36266e700 1 -- 172.21.15.203:0/530609591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff35c101ec0 con 0x7ff35c1084e0 2024-04-25T17:25:52.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.808+0000 7ff3597fa700 1 -- 172.21.15.203:0/530609591 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff34c029070 con 0x7ff35c1084e0 2024-04-25T17:25:52.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.808+0000 7ff36266e700 1 -- 172.21.15.203:0/530609591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff35c103b90 con 0x7ff35c1084e0 2024-04-25T17:25:52.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.808+0000 7ff3597fa700 1 --2- 172.21.15.203:0/530609591 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff344039650 0x7ff34403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:52.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.808+0000 7ff3597fa700 1 -- 172.21.15.203:0/530609591 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff34c01e070 con 0x7ff35c1084e0 2024-04-25T17:25:52.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.809+0000 7ff35b7fe700 1 --2- 172.21.15.203:0/530609591 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff344039650 0x7ff34403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:52.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.809+0000 7ff35b7fe700 1 --2- 172.21.15.203:0/530609591 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff344039650 0x7ff34403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff350006fd0 tx=0x7ff350006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:52.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:52.814+0000 7ff3597fa700 1 -- 172.21.15.203:0/530609591 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff34c013070 con 0x7ff35c1084e0 2024-04-25T17:25:53.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.124+0000 7ff36266e700 1 -- 172.21.15.203:0/530609591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff35c102990 con 0x7ff35c1084e0 2024-04-25T17:25:53.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.125+0000 7ff3597fa700 1 -- 172.21.15.203:0/530609591 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7ff34c030020 con 0x7ff35c1084e0 2024-04-25T17:25:53.126 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:25:53.127 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"79754014-0328-11ef-bc93-c7b262605968","modified":"2024-04-25T17:24:12.001663Z","created":"2024-04-25T17:24:12.001663Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T17:25:53.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.127+0000 7ff36266e700 1 -- 172.21.15.203:0/530609591 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff344039650 msgr2=0x7ff34403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:53.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.127+0000 7ff36266e700 1 --2- 172.21.15.203:0/530609591 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff344039650 0x7ff34403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff350006fd0 tx=0x7ff350006e40 comp rx=0 tx=0).stop 2024-04-25T17:25:53.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.127+0000 7ff36266e700 1 -- 172.21.15.203:0/530609591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff35c1084e0 msgr2=0x7ff35c101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:53.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.127+0000 7ff36266e700 1 --2- 172.21.15.203:0/530609591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff35c1084e0 0x7ff35c101460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0056d0 tx=0x7ff34c0057b0 comp rx=0 tx=0).stop 2024-04-25T17:25:53.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.127+0000 7ff36266e700 1 -- 172.21.15.203:0/530609591 shutdown_connections 2024-04-25T17:25:53.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.127+0000 7ff36266e700 1 --2- 172.21.15.203:0/530609591 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff344039650 0x7ff34403bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:53.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.127+0000 7ff36266e700 1 --2- 172.21.15.203:0/530609591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff35c1084e0 0x7ff35c101460 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:53.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.127+0000 7ff36266e700 1 -- 172.21.15.203:0/530609591 >> 172.21.15.203:0/530609591 conn(0x7ff35c076ea0 msgr2=0x7ff35c0fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:53.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.128+0000 7ff36266e700 1 -- 172.21.15.203:0/530609591 shutdown_connections 2024-04-25T17:25:53.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:53.128+0000 7ff36266e700 1 -- 172.21.15.203:0/530609591 wait complete. 2024-04-25T17:25:53.130 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-25T17:25:54.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:53 smithi098 ceph-mon[29192]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:54.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:53 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/530609591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T17:25:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:54 smithi203 systemd[1]: Started Ceph mon.c for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:25:54.702 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T17:25:54.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T17:25:54.938 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:25:59.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:25:59.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: mon.a calling monitor election 2024-04-25T17:25:59.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:59.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: mon.c calling monitor election 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: monmap epoch 2 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: last_changed 2024-04-25T17:25:54.087202+0000 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: created 2024-04-25T17:24:12.001663+0000 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: min_mon_release 19 (squid) 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: election_strategy: 1 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T17:25:59.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: fsmap 2024-04-25T17:25:59.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T17:25:59.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: mgrmap e13: x(active, since 50s) 2024-04-25T17:25:59.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: overall HEALTH_OK 2024-04-25T17:25:59.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:59.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:59.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:59.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:25:59.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:25:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:25:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:25:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: mon.a calling monitor election 2024-04-25T17:25:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: mon.c calling monitor election 2024-04-25T17:25:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: monmap epoch 2 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: last_changed 2024-04-25T17:25:54.087202+0000 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: created 2024-04-25T17:24:12.001663+0000 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: min_mon_release 19 (squid) 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: election_strategy: 1 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: fsmap 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: mgrmap e13: x(active, since 50s) 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: overall HEALTH_OK 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:59.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:59.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:25:59.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:25:59.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:25:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:25:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.508+0000 7f2c05ae5700 1 -- 172.21.15.203:0/2300409104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80035c0 msgr2=0x7f2be8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.508+0000 7f2c05ae5700 1 --2- 172.21.15.203:0/2300409104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80035c0 0x7f2be8005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2c00101380 tx=0x7f2bf000feb0 comp rx=0 tx=0).stop 2024-04-25T17:25:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.508+0000 7f2c05ae5700 1 -- 172.21.15.203:0/2300409104 shutdown_connections 2024-04-25T17:25:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.508+0000 7f2c05ae5700 1 --2- 172.21.15.203:0/2300409104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2be80035c0 0x7f2be8005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.508+0000 7f2c05ae5700 1 -- 172.21.15.203:0/2300409104 >> 172.21.15.203:0/2300409104 conn(0x7f2c000fcf80 msgr2=0x7f2c000ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.508+0000 7f2c05ae5700 1 -- 172.21.15.203:0/2300409104 shutdown_connections 2024-04-25T17:25:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.508+0000 7f2c05ae5700 1 -- 172.21.15.203:0/2300409104 wait complete. 2024-04-25T17:25:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.509+0000 7f2c05ae5700 1 Processor -- start 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.509+0000 7f2c05ae5700 1 -- start start 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.509+0000 7f2c05ae5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2be80035c0 0x7f2c001030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.509+0000 7f2c05ae5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c001035f0 0x7f2c00103a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.509+0000 7f2c05ae5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c001093a0 con 0x7f2c001035f0 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.509+0000 7f2c05ae5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2c00109220 con 0x7f2be80035c0 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bff7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2be80035c0 0x7f2c001030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bff7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2be80035c0 0x7f2c001030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:60728/0 (socket says 172.21.15.203:60728) 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bff7fe700 1 -- 172.21.15.203:0/1352556476 learned_addr learned my addr 172.21.15.203:0/1352556476 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bfeffd700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c001035f0 0x7f2c00103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bff7fe700 1 -- 172.21.15.203:0/1352556476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2be80035c0 msgr2=0x7f2c001030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bff7fe700 1 -- 172.21.15.203:0/1352556476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2be80035c0 msgr2=0x7f2c001030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bff7fe700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2be80035c0 0x7f2c001030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T17:25:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bff7fe700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2be80035c0 0x7f2c001030b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:25:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bfeffd700 1 -- 172.21.15.203:0/1352556476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2be80035c0 msgr2=0x7f2c001030b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:25:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bfeffd700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2be80035c0 0x7f2c001030b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bfeffd700 1 -- 172.21.15.203:0/1352556476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c00104f80 con 0x7f2c001035f0 2024-04-25T17:25:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bfeffd700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c001035f0 0x7f2c00103a70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400b4d0 tx=0x7f2bf400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bfcff9700 1 -- 172.21.15.203:0/1352556476 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2bf4011630 con 0x7f2c001035f0 2024-04-25T17:25:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2bfcff9700 1 -- 172.21.15.203:0/1352556476 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bf4011c80 con 0x7f2c001035f0 2024-04-25T17:25:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.510+0000 7f2c05ae5700 1 -- 172.21.15.203:0/1352556476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c001a4260 con 0x7f2c001035f0 2024-04-25T17:25:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.511+0000 7f2bfcff9700 1 -- 172.21.15.203:0/1352556476 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2bf4010da0 con 0x7f2c001035f0 2024-04-25T17:25:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.511+0000 7f2c05ae5700 1 -- 172.21.15.203:0/1352556476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c001a47c0 con 0x7f2c001035f0 2024-04-25T17:25:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.511+0000 7f2bfcff9700 1 -- 172.21.15.203:0/1352556476 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2bf4018020 con 0x7f2c001035f0 2024-04-25T17:25:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.512+0000 7f2bfcff9700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2bec0393f0 0x7f2bec03b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:25:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.512+0000 7f2c05ae5700 1 -- 172.21.15.203:0/1352556476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c00102b90 con 0x7f2c001035f0 2024-04-25T17:25:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.512+0000 7f2bfcff9700 1 -- 172.21.15.203:0/1352556476 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2bf404c2f0 con 0x7f2c001035f0 2024-04-25T17:25:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.512+0000 7f2bff7fe700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2bec0393f0 0x7f2bec03b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:25:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.512+0000 7f2bff7fe700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2bec0393f0 0x7f2bec03b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000fa60 tx=0x7f2bf0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:25:59.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.517+0000 7f2bfcff9700 1 -- 172.21.15.203:0/1352556476 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2bf401f020 con 0x7f2c001035f0 2024-04-25T17:25:59.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.802+0000 7f2c05ae5700 1 -- 172.21.15.203:0/1352556476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2c001a4ad0 con 0x7f2c001035f0 2024-04-25T17:25:59.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.803+0000 7f2bfcff9700 1 -- 172.21.15.203:0/1352556476 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f2bf4018310 con 0x7f2c001035f0 2024-04-25T17:25:59.803 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:25:59.804 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"79754014-0328-11ef-bc93-c7b262605968","modified":"2024-04-25T17:25:54.087202Z","created":"2024-04-25T17:24:12.001663Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 -- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2bec0393f0 msgr2=0x7f2bec03b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2bec0393f0 0x7f2bec03b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000fa60 tx=0x7f2bf0005960 comp rx=0 tx=0).stop 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 -- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c001035f0 msgr2=0x7f2c00103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c001035f0 0x7f2c00103a70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400b4d0 tx=0x7f2bf400b890 comp rx=0 tx=0).stop 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 -- 172.21.15.203:0/1352556476 shutdown_connections 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2bec0393f0 0x7f2bec03b8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2be80035c0 0x7f2c001030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 --2- 172.21.15.203:0/1352556476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c001035f0 0x7f2c00103a70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 -- 172.21.15.203:0/1352556476 >> 172.21.15.203:0/1352556476 conn(0x7f2c000fcf80 msgr2=0x7f2c00106760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 -- 172.21.15.203:0/1352556476 shutdown_connections 2024-04-25T17:25:59.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:25:59.805+0000 7f2c05ae5700 1 -- 172.21.15.203:0/1352556476 wait complete. 2024-04-25T17:25:59.807 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-04-25T17:26:00.344 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:00 smithi203 ceph-mon[32834]: Deploying daemon mon.b on smithi173 2024-04-25T17:26:00.344 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:00 smithi203 ceph-mon[32834]: from='client.? 172.21.15.203:0/1352556476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T17:26:00.344 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:00 smithi098 ceph-mon[29192]: Deploying daemon mon.b on smithi173 2024-04-25T17:26:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:00 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/1352556476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T17:26:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:01.346 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T17:26:01.346 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T17:26:01.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:01 smithi098 ceph-mon[29192]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:01.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:26:01 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:26:01.087+0000 7f0dca51f700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T17:26:01.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:01 smithi203 ceph-mon[32834]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:01.582 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:26:02.893 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(???) e0 preinit fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:26:02.893 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T17:26:02.893 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T17:26:02.893 INFO:journalctl@ceph.mon.b.smithi173.stdout: e1 2024-04-25T17:26:02.894 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T17:26:02.894 INFO:journalctl@ceph.mon.b.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T17:26:02.894 INFO:journalctl@ceph.mon.b.smithi173.stdout: legacy client fscid: -1 2024-04-25T17:26:02.894 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-04-25T17:26:02.894 INFO:journalctl@ceph.mon.b.smithi173.stdout: No filesystems configured 2024-04-25T17:26:02.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T17:26:02.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T17:26:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T17:26:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T17:26:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T17:26:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T17:26:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T17:26:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T17:26:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T17:26:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T17:26:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Added host smithi173 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Deploying cephadm binary to smithi203 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:26:03.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Added host smithi203 2024-04-25T17:26:03.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:03.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/647779492' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/647779492' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T17:26:03.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c;count:3 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:03.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Deploying daemon mon.c on smithi203 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/530609591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.a calling monitor election 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:03.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.c calling monitor election 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: monmap epoch 2 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: last_changed 2024-04-25T17:25:54.087202+0000 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: created 2024-04-25T17:24:12.001663+0000 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: min_mon_release 19 (squid) 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: election_strategy: 1 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: fsmap 2024-04-25T17:26:03.280 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mgrmap e13: x(active, since 50s) 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: overall HEALTH_OK 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: Deploying daemon mon.b on smithi173 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/1352556476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 ceph-mon[32856]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 bash[32837]: a6b8778b700b2ccf2eaac58c5a4ef4bb867e886b7e2d17b31a31b314111850b5 2024-04-25T17:26:03.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:02 smithi173 systemd[1]: Started Ceph mon.b for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:26:08.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:08.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:26:08.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: mon.a calling monitor election 2024-04-25T17:26:08.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: mon.c calling monitor election 2024-04-25T17:26:08.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:08.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: mon.b calling monitor election 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: monmap epoch 3 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: last_changed 2024-04-25T17:26:02.898513+0000 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: created 2024-04-25T17:24:12.001663+0000 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: min_mon_release 19 (squid) 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: election_strategy: 1 2024-04-25T17:26:08.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T17:26:08.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T17:26:08.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-04-25T17:26:08.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: fsmap 2024-04-25T17:26:08.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T17:26:08.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: mgrmap e13: x(active, since 59s) 2024-04-25T17:26:08.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: overall HEALTH_OK 2024-04-25T17:26:08.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:08.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:08.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:08.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:26:08.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: mon.a calling monitor election 2024-04-25T17:26:08.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: mon.c calling monitor election 2024-04-25T17:26:08.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: mon.b calling monitor election 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: monmap epoch 3 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: last_changed 2024-04-25T17:26:02.898513+0000 2024-04-25T17:26:08.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: created 2024-04-25T17:24:12.001663+0000 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: min_mon_release 19 (squid) 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: election_strategy: 1 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: fsmap 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: mgrmap e13: x(active, since 59s) 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: overall HEALTH_OK 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:08.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:26:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: mon.a calling monitor election 2024-04-25T17:26:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: mon.c calling monitor election 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: mon.b calling monitor election 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: monmap epoch 3 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: last_changed 2024-04-25T17:26:02.898513+0000 2024-04-25T17:26:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: created 2024-04-25T17:24:12.001663+0000 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: min_mon_release 19 (squid) 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: election_strategy: 1 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: fsmap 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: mgrmap e13: x(active, since 59s) 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: overall HEALTH_OK 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:08.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:09.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:08.998+0000 7f6389873700 1 -- 172.21.15.203:0/3792911719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f636c006230 msgr2=0x7f636c0086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:09.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:08.998+0000 7f6389873700 1 --2- 172.21.15.203:0/3792911719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f636c006230 0x7f636c0086a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6384075e20 tx=0x7f637400fe90 comp rx=0 tx=0).stop 2024-04-25T17:26:09.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:08.998+0000 7f6389873700 1 -- 172.21.15.203:0/3792911719 shutdown_connections 2024-04-25T17:26:09.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:08.998+0000 7f6389873700 1 --2- 172.21.15.203:0/3792911719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f636c006230 0x7f636c0086a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:09.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:08.998+0000 7f6389873700 1 -- 172.21.15.203:0/3792911719 >> 172.21.15.203:0/3792911719 conn(0x7f63840fe040 msgr2=0x7f6384100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:08.999+0000 7f6389873700 1 -- 172.21.15.203:0/3792911719 shutdown_connections 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:08.999+0000 7f6389873700 1 -- 172.21.15.203:0/3792911719 wait complete. 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:08.999+0000 7f6389873700 1 Processor -- start 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6389873700 1 -- start start 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6389873700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f636c006230 0x7f638419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6389873700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f638419a960 0x7f638419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6389873700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638419f230 0x7f638419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6389873700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f638410c6b0 con 0x7f638419f230 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6389873700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f638410c530 con 0x7f638419a960 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6389873700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f638410c830 con 0x7f636c006230 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6382ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f636c006230 0x7f638419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6382ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f636c006230 0x7f638419a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:33680/0 (socket says 172.21.15.203:33680) 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6382ffd700 1 -- 172.21.15.203:0/3156566983 learned_addr learned my addr 172.21.15.203:0/3156566983 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6382ffd700 1 -- 172.21.15.203:0/3156566983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f636c006230 msgr2=0x7f638419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-25T17:26:09.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63827fc700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f638419a960 0x7f638419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6382ffd700 1 -- 172.21.15.203:0/3156566983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f636c006230 msgr2=0x7f638419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6382ffd700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f636c006230 0x7f638419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f6382ffd700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f636c006230 0x7f638419a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63837fe700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638419f230 0x7f638419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63827fc700 1 -- 172.21.15.203:0/3156566983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f638419a960 msgr2=0x7f638419ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63827fc700 1 -- 172.21.15.203:0/3156566983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f638419a960 msgr2=0x7f638419ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63827fc700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f638419a960 0x7f638419ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63827fc700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f638419a960 0x7f638419ecf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63837fe700 1 -- 172.21.15.203:0/3156566983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f636c006230 msgr2=0x7f638419a420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63837fe700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f636c006230 0x7f638419a420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63837fe700 1 -- 172.21.15.203:0/3156566983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f638419a960 msgr2=0x7f638419ecf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63837fe700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f638419a960 0x7f638419ecf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.000+0000 7f63837fe700 1 -- 172.21.15.203:0/3156566983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63841a6fa0 con 0x7f638419f230 2024-04-25T17:26:09.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.001+0000 7f63837fe700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638419f230 0x7f638419f6e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f637c00bd50 tx=0x7f637c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:26:09.005 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.001+0000 7f6388871700 1 -- 172.21.15.203:0/3156566983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c019070 con 0x7f638419f230 2024-04-25T17:26:09.005 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.001+0000 7f6389873700 1 -- 172.21.15.203:0/3156566983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63841a71d0 con 0x7f638419f230 2024-04-25T17:26:09.005 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.001+0000 7f6388871700 1 -- 172.21.15.203:0/3156566983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f637c00cd10 con 0x7f638419f230 2024-04-25T17:26:09.005 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.001+0000 7f6388871700 1 -- 172.21.15.203:0/3156566983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637c012700 con 0x7f638419f230 2024-04-25T17:26:09.005 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.001+0000 7f6389873700 1 -- 172.21.15.203:0/3156566983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63841a76c0 con 0x7f638419f230 2024-04-25T17:26:09.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.002+0000 7f6388871700 1 -- 172.21.15.203:0/3156566983 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f637c01c070 con 0x7f638419f230 2024-04-25T17:26:09.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.002+0000 7f6389873700 1 -- 172.21.15.203:0/3156566983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63840640a0 con 0x7f638419f230 2024-04-25T17:26:09.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.002+0000 7f6388871700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6370039ae0 0x7f637003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:09.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.002+0000 7f6388871700 1 -- 172.21.15.203:0/3156566983 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f637c04c170 con 0x7f638419f230 2024-04-25T17:26:09.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.002+0000 7f6382ffd700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6370039ae0 0x7f637003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:09.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.003+0000 7f6382ffd700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6370039ae0 0x7f637003bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6374005b40 tx=0x7f6374005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:26:09.008 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.007+0000 7f6388871700 1 -- 172.21.15.203:0/3156566983 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f637c007290 con 0x7f638419f230 2024-04-25T17:26:09.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:09.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:09.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:09.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:09.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:09.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:09.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:09.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:09.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:09.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:09.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.304+0000 7f6389873700 1 -- 172.21.15.203:0/3156566983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f63841a79e0 con 0x7f638419f230 2024-04-25T17:26:09.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.305+0000 7f6388871700 1 -- 172.21.15.203:0/3156566983 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7f637c00e090 con 0x7f638419f230 2024-04-25T17:26:09.306 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T17:26:09.306 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":3,"fsid":"79754014-0328-11ef-bc93-c7b262605968","modified":"2024-04-25T17:26:02.898513Z","created":"2024-04-25T17:24:12.001663Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-25T17:26:09.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 -- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6370039ae0 msgr2=0x7f637003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:09.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6370039ae0 0x7f637003bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6374005b40 tx=0x7f6374005a90 comp rx=0 tx=0).stop 2024-04-25T17:26:09.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 -- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638419f230 msgr2=0x7f638419f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:09.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638419f230 0x7f638419f6e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f637c00bd50 tx=0x7f637c00be30 comp rx=0 tx=0).stop 2024-04-25T17:26:09.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 -- 172.21.15.203:0/3156566983 shutdown_connections 2024-04-25T17:26:09.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6370039ae0 0x7f637003bfa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:09.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f636c006230 0x7f638419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:09.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f638419f230 0x7f638419f6e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:09.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 --2- 172.21.15.203:0/3156566983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f638419a960 0x7f638419ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:09.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 -- 172.21.15.203:0/3156566983 >> 172.21.15.203:0/3156566983 conn(0x7f63840fe040 msgr2=0x7f6384106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:26:09.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 -- 172.21.15.203:0/3156566983 shutdown_connections 2024-04-25T17:26:09.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:09.307+0000 7f6389873700 1 -- 172.21.15.203:0/3156566983 wait complete. 2024-04-25T17:26:09.309 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 3 2024-04-25T17:26:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:26:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.115 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T17:26:10.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T17:26:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:26:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/3156566983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T17:26:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:10.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:10.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='client.? 172.21.15.203:0/3156566983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/3156566983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T17:26:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T17:26:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T17:26:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:11.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T17:26:11.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T17:26:11.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T17:26:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:12.435 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:12 smithi098 ceph-mon[29192]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T17:26:12.435 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:12 smithi098 ceph-mon[29192]: Reconfiguring daemon mon.a on smithi098 2024-04-25T17:26:12.435 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:12 smithi098 ceph-mon[29192]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:12 smithi203 ceph-mon[32834]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T17:26:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:12 smithi203 ceph-mon[32834]: Reconfiguring daemon mon.a on smithi098 2024-04-25T17:26:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:12 smithi203 ceph-mon[32834]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:12.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:12 smithi173 ceph-mon[32856]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T17:26:12.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:12 smithi173 ceph-mon[32856]: Reconfiguring daemon mon.a on smithi098 2024-04-25T17:26:12.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:12 smithi173 ceph-mon[32856]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.895+0000 7f23d9071700 1 -- 172.21.15.98:0/2493294501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23d4107920 msgr2=0x7f23d4107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.895+0000 7f23d9071700 1 --2- 172.21.15.98:0/2493294501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23d4107920 0x7f23d4107d00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009990 tx=0x7f23c8009ca0 comp rx=0 tx=0).stop 2024-04-25T17:26:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.895+0000 7f23d9071700 1 -- 172.21.15.98:0/2493294501 shutdown_connections 2024-04-25T17:26:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.895+0000 7f23d9071700 1 --2- 172.21.15.98:0/2493294501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23d4102330 0x7f23d410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.895+0000 7f23d9071700 1 --2- 172.21.15.98:0/2493294501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23d4107920 0x7f23d4107d00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.895+0000 7f23d9071700 1 --2- 172.21.15.98:0/2493294501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23d4101970 0x7f23d4101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.895+0000 7f23d9071700 1 -- 172.21.15.98:0/2493294501 >> 172.21.15.98:0/2493294501 conn(0x7f23d40779b0 msgr2=0x7f23d4077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.895+0000 7f23d9071700 1 -- 172.21.15.98:0/2493294501 shutdown_connections 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.895+0000 7f23d9071700 1 -- 172.21.15.98:0/2493294501 wait complete. 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d9071700 1 Processor -- start 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d9071700 1 -- start start 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d9071700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23d4101970 0x7f23d419a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d9071700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23d4102330 0x7f23d419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d9071700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23d4107920 0x7f23d41a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d9071700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f23d4111b60 con 0x7f23d4107920 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d9071700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f23d41119e0 con 0x7f23d4102330 2024-04-25T17:26:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d9071700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f23d4111ce0 con 0x7f23d4101970 2024-04-25T17:26:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23d4107920 0x7f23d41a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23d4107920 0x7f23d41a5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38178/0 (socket says 172.21.15.98:38178) 2024-04-25T17:26:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d359e700 1 -- 172.21.15.98:0/475834411 learned_addr learned my addr 172.21.15.98:0/475834411 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:26:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.896+0000 7f23d259c700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23d4102330 0x7f23d419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23d2d9d700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23d4101970 0x7f23d419a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23d359e700 1 -- 172.21.15.98:0/475834411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23d4101970 msgr2=0x7f23d419a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23d359e700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23d4101970 0x7f23d419a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23d359e700 1 -- 172.21.15.98:0/475834411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23d4102330 msgr2=0x7f23d419a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23d359e700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23d4102330 0x7f23d419a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23d359e700 1 -- 172.21.15.98:0/475834411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d41a55c0 con 0x7f23d4107920 2024-04-25T17:26:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23d359e700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23d4107920 0x7f23d41a5080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f23cc00ef30 tx=0x7f23cc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:26:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23bbfff700 1 -- 172.21.15.98:0/475834411 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc0092e0 con 0x7f23d4107920 2024-04-25T17:26:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23bbfff700 1 -- 172.21.15.98:0/475834411 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23cc004bb0 con 0x7f23d4107920 2024-04-25T17:26:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23d9071700 1 -- 172.21.15.98:0/475834411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23d41a5790 con 0x7f23d4107920 2024-04-25T17:26:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23bbfff700 1 -- 172.21.15.98:0/475834411 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc004020 con 0x7f23d4107920 2024-04-25T17:26:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.897+0000 7f23d9071700 1 -- 172.21.15.98:0/475834411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23d41a5c40 con 0x7f23d4107920 2024-04-25T17:26:12.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.898+0000 7f23bbfff700 1 -- 172.21.15.98:0/475834411 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f23cc01b030 con 0x7f23d4107920 2024-04-25T17:26:12.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.898+0000 7f23bbfff700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f23bc039890 0x7f23bc03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:12.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.898+0000 7f23bbfff700 1 -- 172.21.15.98:0/475834411 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f23cc04d390 con 0x7f23d4107920 2024-04-25T17:26:12.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.898+0000 7f23d2d9d700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f23bc039890 0x7f23bc03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:12.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.898+0000 7f23d9071700 1 -- 172.21.15.98:0/475834411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23d4109f20 con 0x7f23d4107920 2024-04-25T17:26:12.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.899+0000 7f23d2d9d700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f23bc039890 0x7f23bc03bd50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f23c8006010 tx=0x7f23c800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:26:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:12.904+0000 7f23bbfff700 1 -- 172.21.15.98:0/475834411 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f23cc022020 con 0x7f23d4107920 2024-04-25T17:26:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.102+0000 7f23d9071700 1 -- 172.21.15.98:0/475834411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f23d406b7f0 con 0x7f23d4107920 2024-04-25T17:26:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.102+0000 7f23bbfff700 1 -- 172.21.15.98:0/475834411 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f23cc010040 con 0x7f23d4107920 2024-04-25T17:26:13.103 INFO:teuthology.orchestra.run.smithi098.stdout:# minimal ceph.conf for 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:26:13.104 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-04-25T17:26:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:26:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: mon_host = [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-25T17:26:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 -- 172.21.15.98:0/475834411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f23bc039890 msgr2=0x7f23bc03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f23bc039890 0x7f23bc03bd50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f23c8006010 tx=0x7f23c800b280 comp rx=0 tx=0).stop 2024-04-25T17:26:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 -- 172.21.15.98:0/475834411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23d4107920 msgr2=0x7f23d41a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23d4107920 0x7f23d41a5080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f23cc00ef30 tx=0x7f23cc00c3a0 comp rx=0 tx=0).stop 2024-04-25T17:26:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 -- 172.21.15.98:0/475834411 shutdown_connections 2024-04-25T17:26:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f23bc039890 0x7f23bc03bd50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23d4101970 0x7f23d419a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23d4107920 0x7f23d41a5080 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 --2- 172.21.15.98:0/475834411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23d4102330 0x7f23d419a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 -- 172.21.15.98:0/475834411 >> 172.21.15.98:0/475834411 conn(0x7f23d40779b0 msgr2=0x7f23d40ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:26:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 -- 172.21.15.98:0/475834411 shutdown_connections 2024-04-25T17:26:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:13.104+0000 7f23d9071700 1 -- 172.21.15.98:0/475834411 wait complete. 2024-04-25T17:26:13.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:13.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:13.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:13 smithi098 ceph-mon[29192]: Reconfiguring mon.b (monmap changed)... 2024-04-25T17:26:13.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:13.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T17:26:13.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:13.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:13 smithi098 ceph-mon[29192]: Reconfiguring daemon mon.b on smithi173 2024-04-25T17:26:13.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:13 smithi098 ceph-mon[29192]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:13.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:13 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/475834411' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:13.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:13.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:13.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:13 smithi173 ceph-mon[32856]: Reconfiguring mon.b (monmap changed)... 2024-04-25T17:26:13.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:13.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T17:26:13.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:13.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:13 smithi173 ceph-mon[32856]: Reconfiguring daemon mon.b on smithi173 2024-04-25T17:26:13.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:13 smithi173 ceph-mon[32856]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:13.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:13 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/475834411' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:13.786 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T17:26:13.787 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:26:13.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T17:26:13.813 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:26:13.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:26:13.876 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:26:13.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T17:26:13.905 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:26:13.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:26:13.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:13.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:13.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:13 smithi203 ceph-mon[32834]: Reconfiguring mon.b (monmap changed)... 2024-04-25T17:26:13.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:13.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T17:26:13.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:13.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:13 smithi203 ceph-mon[32834]: Reconfiguring daemon mon.b on smithi173 2024-04-25T17:26:13.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:13 smithi203 ceph-mon[32834]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:13.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:13 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/475834411' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:13.970 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:26:13.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T17:26:14.001 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:26:14.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:26:14.068 INFO:tasks.cephadm:Adding mgr.x on smithi098 2024-04-25T17:26:14.068 INFO:tasks.cephadm:Adding mgr.y on smithi173 2024-04-25T17:26:14.069 INFO:tasks.cephadm:Adding mgr.z on smithi203 2024-04-25T17:26:14.069 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi098=x;smithi173=y;smithi203=z' 2024-04-25T17:26:14.350 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:26:15.352 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:15 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:15.352 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:15 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:15.353 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:15 smithi203 ceph-mon[32834]: Reconfiguring mon.c (monmap changed)... 2024-04-25T17:26:15.353 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:15 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:15.353 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:15 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T17:26:15.353 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:15 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:15.353 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:15 smithi203 ceph-mon[32834]: Reconfiguring daemon mon.c on smithi203 2024-04-25T17:26:15.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:15 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:15.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:15 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:15.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:15 smithi098 ceph-mon[29192]: Reconfiguring mon.c (monmap changed)... 2024-04-25T17:26:15.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:15 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:15.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:15 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T17:26:15.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:15 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:15.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:15 smithi098 ceph-mon[29192]: Reconfiguring daemon mon.c on smithi203 2024-04-25T17:26:15.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:15 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:15.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:15 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:15.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:15 smithi173 ceph-mon[32856]: Reconfiguring mon.c (monmap changed)... 2024-04-25T17:26:15.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:15 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T17:26:15.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:15 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T17:26:15.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:15 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:15.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:15 smithi173 ceph-mon[32856]: Reconfiguring daemon mon.c on smithi203 2024-04-25T17:26:16.357 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:16 smithi098 ceph-mon[29192]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:16.357 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:16 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:16.357 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:16 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:16.357 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:16 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:16.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:16 smithi203 ceph-mon[32834]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:16.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:16 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:16.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:16 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:16.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:16 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:16.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:16 smithi173 ceph-mon[32856]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:16.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:16 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:16.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:16 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:16.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:16 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:16.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 -- 172.21.15.203:0/602629294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe948102ba0 msgr2=0x7fe948103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:16.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 --2- 172.21.15.203:0/602629294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe948102ba0 0x7fe948103000 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe93c009990 tx=0x7fe93c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:26:16.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 -- 172.21.15.203:0/602629294 shutdown_connections 2024-04-25T17:26:16.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 --2- 172.21.15.203:0/602629294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe948103540 0x7fe948109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:16.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 --2- 172.21.15.203:0/602629294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9481019a0 0x7fe948101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:16.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 --2- 172.21.15.203:0/602629294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe948102ba0 0x7fe948103000 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:16.897 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 -- 172.21.15.203:0/602629294 >> 172.21.15.203:0/602629294 conn(0x7fe9480fcf40 msgr2=0x7fe9480ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:26:16.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 -- 172.21.15.203:0/602629294 shutdown_connections 2024-04-25T17:26:16.898 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 -- 172.21.15.203:0/602629294 wait complete. 2024-04-25T17:26:16.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 Processor -- start 2024-04-25T17:26:16.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.895+0000 7fe94ea25700 1 -- start start 2024-04-25T17:26:16.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe94ea25700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481019a0 0x7fe948198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe94ea25700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe948102ba0 0x7fe9481991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe94ea25700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe948103540 0x7fe9481a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe94ea25700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe94810d4b0 con 0x7fe948102ba0 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe94ea25700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe94810d330 con 0x7fe948103540 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe94ea25700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe94810d630 con 0x7fe9481019a0 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe947fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481019a0 0x7fe948198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe947fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481019a0 0x7fe948198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48056/0 (socket says 172.21.15.203:48056) 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe947fff700 1 -- 172.21.15.203:0/2220855039 learned_addr learned my addr 172.21.15.203:0/2220855039 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe9477fe700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe948102ba0 0x7fe9481991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe94cfc2700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe948103540 0x7fe9481a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe947fff700 1 -- 172.21.15.203:0/2220855039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe948103540 msgr2=0x7fe9481a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe947fff700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe948103540 0x7fe9481a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:16.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe947fff700 1 -- 172.21.15.203:0/2220855039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe948102ba0 msgr2=0x7fe9481991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe947fff700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe948102ba0 0x7fe9481991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe947fff700 1 -- 172.21.15.203:0/2220855039 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9481a08e0 con 0x7fe9481019a0 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.896+0000 7fe947fff700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481019a0 0x7fe948198c60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe93800c8f0 tx=0x7fe93800cc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.897+0000 7fe9457fa700 1 -- 172.21.15.203:0/2220855039 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938004220 con 0x7fe9481019a0 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.897+0000 7fe94ea25700 1 -- 172.21.15.203:0/2220855039 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9481a0bd0 con 0x7fe9481019a0 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.897+0000 7fe9457fa700 1 -- 172.21.15.203:0/2220855039 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9380043c0 con 0x7fe9481019a0 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.897+0000 7fe9457fa700 1 -- 172.21.15.203:0/2220855039 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe938003c00 con 0x7fe9481019a0 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.897+0000 7fe94ea25700 1 -- 172.21.15.203:0/2220855039 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9481a10c0 con 0x7fe9481019a0 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.898+0000 7fe9457fa700 1 -- 172.21.15.203:0/2220855039 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe93800f450 con 0x7fe9481019a0 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.898+0000 7fe94ea25700 1 -- 172.21.15.203:0/2220855039 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe948069760 con 0x7fe9481019a0 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.898+0000 7fe9457fa700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe930039890 0x7fe93003bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:16.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.898+0000 7fe9457fa700 1 -- 172.21.15.203:0/2220855039 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe93804c640 con 0x7fe9481019a0 2024-04-25T17:26:16.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.898+0000 7fe9477fe700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe930039890 0x7fe93003bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:16.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.898+0000 7fe9477fe700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe930039890 0x7fe93003bd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe94819a170 tx=0x7fe93c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:26:16.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:16.903+0000 7fe9457fa700 1 -- 172.21.15.203:0/2220855039 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe948069760 con 0x7fe9481019a0 2024-04-25T17:26:17.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.112+0000 7fe94ea25700 1 -- 172.21.15.203:0/2220855039 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}) -- 0x7fe9480ff140 con 0x7fe930039890 2024-04-25T17:26:17.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.117+0000 7fe9457fa700 1 -- 172.21.15.203:0/2220855039 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe9480ff140 con 0x7fe930039890 2024-04-25T17:26:17.119 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mgr update... 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 -- 172.21.15.203:0/2220855039 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe930039890 msgr2=0x7fe93003bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe930039890 0x7fe93003bd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe94819a170 tx=0x7fe93c005fb0 comp rx=0 tx=0).stop 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 -- 172.21.15.203:0/2220855039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481019a0 msgr2=0x7fe948198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481019a0 0x7fe948198c60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe93800c8f0 tx=0x7fe93800cc00 comp rx=0 tx=0).stop 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 -- 172.21.15.203:0/2220855039 shutdown_connections 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe930039890 0x7fe93003bd50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9481019a0 0x7fe948198c60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe948103540 0x7fe9481a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 --2- 172.21.15.203:0/2220855039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe948102ba0 0x7fe9481991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 -- 172.21.15.203:0/2220855039 >> 172.21.15.203:0/2220855039 conn(0x7fe9480fcf40 msgr2=0x7fe9480fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:26:17.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 -- 172.21.15.203:0/2220855039 shutdown_connections 2024-04-25T17:26:17.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:26:17.120+0000 7fe94ea25700 1 -- 172.21.15.203:0/2220855039 wait complete. 2024-04-25T17:26:17.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:17.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:17.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:17.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:17.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:17.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:17.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:17.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:17.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:17.683 DEBUG:teuthology.orchestra.run.smithi173:mgr.y> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T17:26:17.686 DEBUG:teuthology.orchestra.run.smithi203:mgr.z> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T17:26:17.688 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T17:26:17.688 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:26:17.688 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T17:26:17.705 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T17:26:17.705 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-04-25T17:26:17.761 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T17:26:17.762 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:17.762 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-25T17:26:17.762 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:17.762 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:17.762 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-25 17:25:21.869685299 +0000 2024-04-25T17:26:17.762 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-25 17:20:36.044014801 +0000 2024-04-25T17:26:17.762 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-25 17:20:36.044014801 +0000 2024-04-25T17:26:17.762 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-25 17:20:36.044014801 +0000 2024-04-25T17:26:17.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T17:26:17.825 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-25T17:26:17.825 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-25T17:26:17.825 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000214426 s, 2.4 MB/s 2024-04-25T17:26:17.827 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T17:26:17.882 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-04-25T17:26:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T17:26:17.939 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:17.939 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-25T17:26:17.939 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:17.939 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:17.939 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-25 17:25:21.902684900 +0000 2024-04-25T17:26:17.940 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-25 17:20:36.400010317 +0000 2024-04-25T17:26:17.940 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-25 17:20:36.400010317 +0000 2024-04-25T17:26:17.940 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-25 17:20:36.400010317 +0000 2024-04-25T17:26:17.940 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T17:26:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-25T17:26:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-25T17:26:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000208996 s, 2.4 MB/s 2024-04-25T17:26:18.003 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T17:26:18.059 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-04-25T17:26:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T17:26:18.114 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:18.114 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-25T17:26:18.114 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:18.114 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:18.114 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-25 17:25:21.934684513 +0000 2024-04-25T17:26:18.114 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-25 17:20:36.754005859 +0000 2024-04-25T17:26:18.115 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-25 17:20:36.754005859 +0000 2024-04-25T17:26:18.115 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-25 17:20:36.754005859 +0000 2024-04-25T17:26:18.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T17:26:18.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:18.176 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-25T17:26:18.176 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-25T17:26:18.176 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000199568 s, 2.6 MB/s 2024-04-25T17:26:18.177 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T17:26:18.233 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-04-25T17:26:18.287 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T17:26:18.287 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:18.287 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T17:26:18.288 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:18.288 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:18.288 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-25 17:25:21.966684126 +0000 2024-04-25T17:26:18.288 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-25 17:20:37.092001602 +0000 2024-04-25T17:26:18.288 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-25 17:20:37.092001602 +0000 2024-04-25T17:26:18.288 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-25 17:20:37.092001602 +0000 2024-04-25T17:26:18.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T17:26:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:18.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:18.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:18.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:18.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:18.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T17:26:18.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T17:26:18.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T17:26:18.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-25T17:26:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-25T17:26:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000255116 s, 2.0 MB/s 2024-04-25T17:26:18.351 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T17:26:18.406 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T17:26:18.406 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T17:26:18.425 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T17:26:18.425 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-04-25T17:26:18.482 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T17:26:18.482 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:18.482 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T17:26:18.482 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:18.483 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:18.483 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 17:26:02.334451300 +0000 2024-04-25T17:26:18.483 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 17:20:54.311779903 +0000 2024-04-25T17:26:18.483 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 17:20:54.311779903 +0000 2024-04-25T17:26:18.483 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 17:20:54.311779903 +0000 2024-04-25T17:26:18.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T17:26:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:18.549 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T17:26:18.549 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T17:26:18.549 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000303696 s, 1.7 MB/s 2024-04-25T17:26:18.551 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T17:26:18.609 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-04-25T17:26:18.667 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T17:26:18.667 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:18.667 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T17:26:18.667 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:18.667 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:18.667 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 17:26:02.335451278 +0000 2024-04-25T17:26:18.667 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 17:20:54.651771993 +0000 2024-04-25T17:26:18.668 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 17:20:54.651771993 +0000 2024-04-25T17:26:18.668 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 17:20:54.651771993 +0000 2024-04-25T17:26:18.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T17:26:18.732 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T17:26:18.732 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T17:26:18.732 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000205276 s, 2.5 MB/s 2024-04-25T17:26:18.734 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T17:26:18.791 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-04-25T17:26:18.847 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T17:26:18.847 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:18.847 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T17:26:18.847 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:18.847 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:18.847 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 17:26:02.335451278 +0000 2024-04-25T17:26:18.847 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 17:20:54.994764013 +0000 2024-04-25T17:26:18.847 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 17:20:54.994764013 +0000 2024-04-25T17:26:18.847 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 17:20:54.994764013 +0000 2024-04-25T17:26:18.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T17:26:18.911 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T17:26:18.911 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T17:26:18.911 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000256845 s, 2.0 MB/s 2024-04-25T17:26:18.912 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T17:26:18.969 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-04-25T17:26:19.025 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T17:26:19.025 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:19.025 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T17:26:19.025 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:19.025 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:19.025 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 17:26:02.335451278 +0000 2024-04-25T17:26:19.025 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 17:20:55.335756079 +0000 2024-04-25T17:26:19.025 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 17:20:55.335756079 +0000 2024-04-25T17:26:19.025 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 17:20:55.335756079 +0000 2024-04-25T17:26:19.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T17:26:19.087 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T17:26:19.087 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T17:26:19.087 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000194771 s, 2.6 MB/s 2024-04-25T17:26:19.089 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T17:26:19.145 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T17:26:19.146 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T17:26:19.161 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T17:26:19.162 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-04-25T17:26:19.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:19 smithi203 ceph-mon[32834]: Saving service mgr spec with placement smithi098=x;smithi173=y;smithi203=z;count:3 2024-04-25T17:26:19.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:19 smithi203 ceph-mon[32834]: Deploying daemon mgr.z on smithi203 2024-04-25T17:26:19.216 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T17:26:19.216 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:19.216 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-25T17:26:19.217 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:19.217 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:19.217 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 17:26:19.160898362 +0000 2024-04-25T17:26:19.217 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 17:21:03.866737715 +0000 2024-04-25T17:26:19.217 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 17:21:03.866737715 +0000 2024-04-25T17:26:19.217 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-25 17:21:03.866737715 +0000 2024-04-25T17:26:19.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T17:26:19.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:19 smithi173 ceph-mon[32856]: Saving service mgr spec with placement smithi098=x;smithi173=y;smithi203=z;count:3 2024-04-25T17:26:19.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:19 smithi173 ceph-mon[32856]: Deploying daemon mgr.z on smithi203 2024-04-25T17:26:19.283 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T17:26:19.283 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T17:26:19.283 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000268824 s, 1.9 MB/s 2024-04-25T17:26:19.284 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T17:26:19.342 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-04-25T17:26:19.397 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T17:26:19.398 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:19.398 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T17:26:19.398 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:19.398 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:19.398 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 17:26:19.160898362 +0000 2024-04-25T17:26:19.398 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 17:21:04.213728648 +0000 2024-04-25T17:26:19.398 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 17:21:04.213728648 +0000 2024-04-25T17:26:19.398 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-25 17:21:04.213728648 +0000 2024-04-25T17:26:19.398 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T17:26:19.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:19 smithi098 ceph-mon[29192]: Saving service mgr spec with placement smithi098=x;smithi173=y;smithi203=z;count:3 2024-04-25T17:26:19.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:19 smithi098 ceph-mon[29192]: Deploying daemon mgr.z on smithi203 2024-04-25T17:26:19.460 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T17:26:19.460 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T17:26:19.460 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000244808 s, 2.1 MB/s 2024-04-25T17:26:19.462 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T17:26:19.517 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-04-25T17:26:19.573 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T17:26:19.573 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:19.573 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-25T17:26:19.573 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:19.573 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:19.574 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 17:26:19.161898338 +0000 2024-04-25T17:26:19.574 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 17:21:04.560719578 +0000 2024-04-25T17:26:19.574 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 17:21:04.560719578 +0000 2024-04-25T17:26:19.574 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-25 17:21:04.560719578 +0000 2024-04-25T17:26:19.575 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T17:26:19.637 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T17:26:19.637 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T17:26:19.638 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000194596 s, 2.6 MB/s 2024-04-25T17:26:19.639 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T17:26:19.694 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-04-25T17:26:19.749 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T17:26:19.749 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T17:26:19.749 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T17:26:19.749 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T17:26:19.750 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T17:26:19.750 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 17:26:19.161898338 +0000 2024-04-25T17:26:19.750 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 17:21:04.893710875 +0000 2024-04-25T17:26:19.750 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 17:21:04.893710875 +0000 2024-04-25T17:26:19.750 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-25 17:21:04.893710875 +0000 2024-04-25T17:26:19.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T17:26:19.810 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:19 smithi203 systemd[1]: Started Ceph mgr.z for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:26:19.813 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T17:26:19.813 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T17:26:19.813 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000212595 s, 2.4 MB/s 2024-04-25T17:26:19.814 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T17:26:19.871 INFO:tasks.cephadm:Deploying osd.0 on smithi098 with /dev/vg_nvme/lv_4... 2024-04-25T17:26:19.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T17:26:20.097 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:19 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:19.809+0000 7f650abd6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T17:26:20.097 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:19 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:19.996+0000 7f650abd6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:26:20.097 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:20.096+0000 7f650abd6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T17:26:20.111 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:26:20.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:20 smithi098 ceph-mon[29192]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T17:26:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T17:26:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T17:26:20.208 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:20.405 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-mon[32834]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:20.405 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:20.405 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:20.405 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:20.405 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T17:26:20.405 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T17:26:20.405 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T17:26:20.405 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:20 smithi173 ceph-mon[32856]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T17:26:20.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:20.713 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:20.403+0000 7f650abd6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T17:26:20.713 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:20 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:20.487+0000 7f650abd6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T17:26:21.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:21 smithi203 ceph-mon[32834]: Deploying daemon mgr.y on smithi173 2024-04-25T17:26:21.360 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:21 smithi173 ceph-mon[32856]: Deploying daemon mgr.y on smithi173 2024-04-25T17:26:21.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:21 smithi098 ceph-mon[29192]: Deploying daemon mgr.y on smithi173 2024-04-25T17:26:21.988 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:21 smithi173 systemd[1]: Starting Ceph mgr.y for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:26:22.025 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T17:26:22.025 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T17:26:22.025 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-25T17:26:22.025 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-25T17:26:22.025 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144016 s, 728 MB/s 2024-04-25T17:26:22.025 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-25T17:26:22.085 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:21 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:21.737+0000 7f650abd6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T17:26:22.085 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:21 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:21.909+0000 7f650abd6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T17:26:22.086 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:22.084+0000 7f650abd6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T17:26:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:22 smithi173 ceph-mon[32856]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:22.273 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:21 smithi173 podman[34266]: 2024-04-25 17:26:21.891428812 +0000 UTC m=+0.018074198 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:26:22.273 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:21 smithi173 podman[34266]: 2024-04-25 17:26:21.99523643 +0000 UTC m=+0.121881793 container create 4799370d0d30dc8f63f2936ad7b9bc9df77c809a09b53b3e977576841f0b09a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-25T17:26:22.273 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:22 smithi173 podman[34266]: 2024-04-25 17:26:22.197697554 +0000 UTC m=+0.324342922 container init 4799370d0d30dc8f63f2936ad7b9bc9df77c809a09b53b3e977576841f0b09a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:26:22.273 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:22 smithi173 podman[34266]: 2024-04-25 17:26:22.200328937 +0000 UTC m=+0.326974299 container start 4799370d0d30dc8f63f2936ad7b9bc9df77c809a09b53b3e977576841f0b09a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:26:22.273 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:22 smithi173 ceph-mgr[34285]: -- 172.21.15.173:0/3825351888 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f9ba8065a0 con 0x55f9ba7db000 2024-04-25T17:26:22.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:22 smithi098 ceph-mon[29192]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:22.374 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:22 smithi203 ceph-mon[32834]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:22.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-04-25T17:26:22.649 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:22 smithi173 bash[34266]: 4799370d0d30dc8f63f2936ad7b9bc9df77c809a09b53b3e977576841f0b09a6 2024-04-25T17:26:22.650 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:22 smithi173 systemd[1]: Started Ceph mgr.y for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:26:22.650 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:22.356+0000 7f6b811b8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T17:26:22.650 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:22.544+0000 7f6b811b8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:26:22.650 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:22.649+0000 7f6b811b8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T17:26:22.680 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:22.373+0000 7f650abd6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T17:26:22.680 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:22.470+0000 7f650abd6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T17:26:22.680 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:22.679+0000 7f650abd6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T17:26:22.867 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:26:22.962 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:22.880+0000 7f650abd6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:26:23.273 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:22.974+0000 7f6b811b8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T17:26:23.273 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:23 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:23.067+0000 7f6b811b8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T17:26:23.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:23.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:23 smithi098 ceph-mon[29192]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:23.713 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:23 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:23.525+0000 7f650abd6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T17:26:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:23 smithi203 ceph-mon[32834]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:23.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:23.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:23.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:23 smithi173 ceph-mon[32856]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:24.262 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:23 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:23.963+0000 7f650abd6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T17:26:24.263 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:24.059+0000 7f650abd6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T17:26:24.263 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:24.261+0000 7f650abd6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T17:26:24.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.563+0000 7f12643af700 1 -- 172.21.15.98:0/3133202111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c108830 msgr2=0x7f125c10ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:24.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.563+0000 7f12643af700 1 --2- 172.21.15.98:0/3133202111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c108830 0x7f125c10ac20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f125400b0a0 tx=0x7f125400b3b0 comp rx=0 tx=0).stop 2024-04-25T17:26:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.563+0000 7f12643af700 1 -- 172.21.15.98:0/3133202111 shutdown_connections 2024-04-25T17:26:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.563+0000 7f12643af700 1 --2- 172.21.15.98:0/3133202111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c108830 0x7f125c10ac20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.563+0000 7f12643af700 1 --2- 172.21.15.98:0/3133202111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c1035d0 0x7f125c1059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.563+0000 7f12643af700 1 --2- 172.21.15.98:0/3133202111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f125c105f00 0x7f125c1082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.563+0000 7f12643af700 1 -- 172.21.15.98:0/3133202111 >> 172.21.15.98:0/3133202111 conn(0x7f125c0fcf40 msgr2=0x7f125c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:26:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.563+0000 7f12643af700 1 -- 172.21.15.98:0/3133202111 shutdown_connections 2024-04-25T17:26:24.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.563+0000 7f12643af700 1 -- 172.21.15.98:0/3133202111 wait complete. 2024-04-25T17:26:24.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.564+0000 7f12643af700 1 Processor -- start 2024-04-25T17:26:24.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.564+0000 7f12643af700 1 -- start start 2024-04-25T17:26:24.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.564+0000 7f12643af700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f125c1035d0 0x7f125c198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.564+0000 7f12643af700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c105f00 0x7f125c199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.564+0000 7f12643af700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c108830 0x7f125c1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.564+0000 7f12643af700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f125c10e530 con 0x7f125c105f00 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.564+0000 7f12643af700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f125c10e3b0 con 0x7f125c108830 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.564+0000 7f12643af700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f125c10e6b0 con 0x7f125c1035d0 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126214b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f125c1035d0 0x7f125c198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126194a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c105f00 0x7f125c199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126214b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f125c1035d0 0x7f125c198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:52312/0 (socket says 172.21.15.98:52312) 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126194a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c105f00 0x7f125c199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41120/0 (socket says 172.21.15.98:41120) 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126214b700 1 -- 172.21.15.98:0/1158504485 learned_addr learned my addr 172.21.15.98:0/1158504485 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126294c700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c108830 0x7f125c1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126214b700 1 -- 172.21.15.98:0/1158504485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c108830 msgr2=0x7f125c1a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126214b700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c108830 0x7f125c1a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:24.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126214b700 1 -- 172.21.15.98:0/1158504485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c105f00 msgr2=0x7f125c199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126214b700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c105f00 0x7f125c199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126214b700 1 -- 172.21.15.98:0/1158504485 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125c1a0870 con 0x7f125c1035d0 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126194a700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c105f00 0x7f125c199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f126214b700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f125c1035d0 0x7f125c198c20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f125800d950 tx=0x7f125800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f12537fe700 1 -- 172.21.15.98:0/1158504485 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258016070 con 0x7f125c1035d0 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f12643af700 1 -- 172.21.15.98:0/1158504485 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125c101420 con 0x7f125c1035d0 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f12537fe700 1 -- 172.21.15.98:0/1158504485 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f125800fd40 con 0x7f125c1035d0 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f12537fe700 1 -- 172.21.15.98:0/1158504485 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125800e3b0 con 0x7f125c1035d0 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.565+0000 7f12643af700 1 -- 172.21.15.98:0/1158504485 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125c1019b0 con 0x7f125c1035d0 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.566+0000 7f12537fe700 1 -- 172.21.15.98:0/1158504485 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1258019070 con 0x7f125c1035d0 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.566+0000 7f12643af700 1 -- 172.21.15.98:0/1158504485 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f125c069730 con 0x7f125c1035d0 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.567+0000 7f12537fe700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1248039890 0x7f124803bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.567+0000 7f12537fe700 1 -- 172.21.15.98:0/1158504485 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f125804c560 con 0x7f125c1035d0 2024-04-25T17:26:24.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.567+0000 7f126194a700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1248039890 0x7f124803bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:24.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.567+0000 7f126194a700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1248039890 0x7f124803bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f124c009710 tx=0x7f124c00f7d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:26:24.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.577+0000 7f12537fe700 1 -- 172.21.15.98:0/1158504485 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1258010750 con 0x7f125c1035d0 2024-04-25T17:26:24.642 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:24.302+0000 7f6b811b8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T17:26:24.642 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:24.470+0000 7f6b811b8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T17:26:24.642 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:24.640+0000 7f6b811b8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T17:26:24.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:24.797+0000 7f12643af700 1 -- 172.21.15.98:0/1158504485 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f125c062f40 con 0x7f1248039890 2024-04-25T17:26:24.904 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.904 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.904 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.904 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.904 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:24.904 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:24.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T17:26:24.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T17:26:24.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T17:26:24.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:24.905 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:24 smithi098 ceph-mon[29192]: Reconfiguring daemon mgr.x on smithi098 2024-04-25T17:26:24.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:24.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:24.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.923 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:24.924 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T17:26:24.924 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T17:26:24.924 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T17:26:24.924 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:24.924 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-mon[32856]: Reconfiguring daemon mgr.x on smithi098 2024-04-25T17:26:25.126 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:24.790+0000 7f650abd6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T17:26:25.126 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:24.883+0000 7f650abd6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T17:26:25.126 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:25.126 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:25.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:24 smithi203 ceph-mon[32834]: Reconfiguring daemon mgr.x on smithi098 2024-04-25T17:26:25.217 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:24.921+0000 7f6b811b8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T17:26:25.217 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:25.012+0000 7f6b811b8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T17:26:25.218 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:25.216+0000 7f6b811b8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T17:26:25.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:25.125+0000 7f650abd6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T17:26:25.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:25.258+0000 7f650abd6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T17:26:25.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:25.352+0000 7f650abd6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T17:26:25.522 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:25.405+0000 7f6b811b8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:26:25.784 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:25.466+0000 7f650abd6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T17:26:25.785 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:25.783+0000 7f650abd6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T17:26:26.060 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:26.060 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:26:26.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:26:26.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:26.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:26.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:26.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:26.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:26.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:26.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:26.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:26.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:26.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:26:26.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:26:26.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:26.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:26.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:26.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:26.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:26.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:26.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:26.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:25 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:26.272 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:26.006+0000 7f6b811b8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T17:26:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:26:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:26:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:26.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:26.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:26.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:25 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:26.713 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:26 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:26.387+0000 7f650abd6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T17:26:26.713 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:26:26 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:26:26.600+0000 7f650abd6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T17:26:26.717 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:26.420+0000 7f6b811b8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T17:26:26.717 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:26.516+0000 7f6b811b8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T17:26:26.717 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:26.715+0000 7f6b811b8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T17:26:26.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:26.807+0000 7f12537fe700 1 -- 172.21.15.98:0/1158504485 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f125800ad10 con 0x7f125c1035d0 2024-04-25T17:26:27.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:26 smithi173 ceph-mon[32856]: Standby manager daemon z started 2024-04-25T17:26:27.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:26 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T17:26:27.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:26 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:26:27.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:26 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T17:26:27.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:26 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:26:27.117 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:26 smithi098 ceph-mon[29192]: Standby manager daemon z started 2024-04-25T17:26:27.117 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:26 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T17:26:27.117 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:26 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:26:27.117 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:26 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T17:26:27.117 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:26 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:26:27.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:26 smithi203 ceph-mon[32834]: Standby manager daemon z started 2024-04-25T17:26:27.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:26 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T17:26:27.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:26 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:26:27.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:26 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T17:26:27.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:26 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.203:0/3912944827' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:26:27.522 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:27.240+0000 7f6b811b8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T17:26:27.523 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:27.331+0000 7f6b811b8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T17:26:27.914 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:27.574+0000 7f6b811b8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T17:26:27.914 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:27.705+0000 7f6b811b8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T17:26:27.915 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:27.798+0000 7f6b811b8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T17:26:27.915 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:27.913+0000 7f6b811b8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T17:26:27.915 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-mon[32856]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-25T17:26:27.915 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T17:26:27.915 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-mon[32856]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:27.915 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1986525667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91254e6f-c19b-4a74-8b41-718ef0f5eb4d"}]: dispatch 2024-04-25T17:26:27.915 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1986525667' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91254e6f-c19b-4a74-8b41-718ef0f5eb4d"}]': finished 2024-04-25T17:26:27.915 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-mon[32856]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T17:26:27.915 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:28.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:27 smithi098 ceph-mon[29192]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-25T17:26:28.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T17:26:28.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:27 smithi098 ceph-mon[29192]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:28.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:27 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1986525667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91254e6f-c19b-4a74-8b41-718ef0f5eb4d"}]: dispatch 2024-04-25T17:26:28.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:27 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1986525667' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91254e6f-c19b-4a74-8b41-718ef0f5eb4d"}]': finished 2024-04-25T17:26:28.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:27 smithi098 ceph-mon[29192]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T17:26:28.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:28.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:27 smithi203 ceph-mon[32834]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-25T17:26:28.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T17:26:28.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:27 smithi203 ceph-mon[32834]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:28.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:27 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1986525667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91254e6f-c19b-4a74-8b41-718ef0f5eb4d"}]: dispatch 2024-04-25T17:26:28.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:27 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1986525667' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91254e6f-c19b-4a74-8b41-718ef0f5eb4d"}]': finished 2024-04-25T17:26:28.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:27 smithi203 ceph-mon[32834]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T17:26:28.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:28.522 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:28 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:28.216+0000 7f6b811b8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T17:26:29.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:28 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3629794727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:26:29.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:28 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3629794727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:26:29.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:28 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3629794727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:26:29.272 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:28 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:28.806+0000 7f6b811b8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T17:26:29.273 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:26:29 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:26:29.017+0000 7f6b811b8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T17:26:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:29.820+0000 7f12537fe700 1 -- 172.21.15.98:0/1158504485 <== mon.1 v2:172.21.15.203:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f125800e550 con 0x7f125c1035d0 2024-04-25T17:26:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:29 smithi098 ceph-mon[29192]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:29 smithi098 ceph-mon[29192]: Standby manager daemon y started 2024-04-25T17:26:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:29 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T17:26:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:29 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:26:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:29 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T17:26:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:29 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:26:30.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:29 smithi203 ceph-mon[32834]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:30.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:29 smithi203 ceph-mon[32834]: Standby manager daemon y started 2024-04-25T17:26:30.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:29 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T17:26:30.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:29 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:26:30.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:29 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T17:26:30.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:29 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:26:30.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:29 smithi173 ceph-mon[32856]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:30.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:29 smithi173 ceph-mon[32856]: Standby manager daemon y started 2024-04-25T17:26:30.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:29 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T17:26:30.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:29 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:26:30.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:29 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T17:26:30.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:29 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.173:0/1614764907' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:26:31.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:30 smithi098 ceph-mon[29192]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T17:26:31.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:30 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T17:26:31.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:30 smithi203 ceph-mon[32834]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T17:26:31.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:30 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T17:26:31.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:30 smithi173 ceph-mon[32856]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T17:26:31.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:30 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T17:26:31.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:31 smithi098 ceph-mon[29192]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:32.179 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:31 smithi203 ceph-mon[32834]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:32.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:31 smithi173 ceph-mon[32856]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:34.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:33 smithi098 ceph-mon[29192]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:34.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:33 smithi203 ceph-mon[32834]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:34.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:33 smithi173 ceph-mon[32856]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:35.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:34 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T17:26:35.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:34 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:35.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:34 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T17:26:35.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:34 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:35.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:34 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T17:26:35.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:34 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:36.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:35 smithi098 ceph-mon[29192]: Deploying daemon osd.0 on smithi098 2024-04-25T17:26:36.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:35 smithi098 ceph-mon[29192]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:36.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:35 smithi203 ceph-mon[32834]: Deploying daemon osd.0 on smithi098 2024-04-25T17:26:36.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:35 smithi203 ceph-mon[32834]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:36.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:35 smithi173 ceph-mon[32856]: Deploying daemon osd.0 on smithi098 2024-04-25T17:26:36.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:35 smithi173 ceph-mon[32856]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:38.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:37 smithi098 ceph-mon[29192]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:38.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:37 smithi203 ceph-mon[32834]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:38.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:37 smithi173 ceph-mon[32856]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:40.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:39 smithi173 ceph-mon[32856]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:40.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:39 smithi098 ceph-mon[29192]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:39 smithi203 ceph-mon[32834]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:41.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:41 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:41.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:41 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:41.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:41 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:41.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:41 smithi203 ceph-mon[32834]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:41.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:41 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:41 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:41 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:41 smithi173 ceph-mon[32856]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:41 smithi098 ceph-mon[29192]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:43.836 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 0 on host 'smithi098' 2024-04-25T17:26:43.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.834+0000 7f12537fe700 1 -- 172.21.15.98:0/1158504485 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f125c062f40 con 0x7f1248039890 2024-04-25T17:26:43.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 -- 172.21.15.98:0/1158504485 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1248039890 msgr2=0x7f124803bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:43.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1248039890 0x7f124803bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f124c009710 tx=0x7f124c00f7d0 comp rx=0 tx=0).stop 2024-04-25T17:26:43.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 -- 172.21.15.98:0/1158504485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f125c1035d0 msgr2=0x7f125c198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:43.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f125c1035d0 0x7f125c198c20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f125800d950 tx=0x7f125800dc60 comp rx=0 tx=0).stop 2024-04-25T17:26:43.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 -- 172.21.15.98:0/1158504485 shutdown_connections 2024-04-25T17:26:43.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1248039890 0x7f124803bd50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:43.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f125c1035d0 0x7f125c198c20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:43.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f125c108830 0x7f125c1a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:43.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 --2- 172.21.15.98:0/1158504485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f125c105f00 0x7f125c199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:43.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 -- 172.21.15.98:0/1158504485 >> 172.21.15.98:0/1158504485 conn(0x7f125c0fcf40 msgr2=0x7f125c104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:26:43.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 -- 172.21.15.98:0/1158504485 shutdown_connections 2024-04-25T17:26:43.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:26:43.837+0000 7f12643af700 1 -- 172.21.15.98:0/1158504485 wait complete. 2024-04-25T17:26:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:43 smithi098 ceph-mon[29192]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:44.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:43 smithi203 ceph-mon[32834]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:44.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:44.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:44.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:44.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:43 smithi173 ceph-mon[32856]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:44.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:44.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:44.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:44.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:26:44.515 DEBUG:teuthology.orchestra.run.smithi098:osd.0> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T17:26:44.517 INFO:tasks.cephadm:Deploying osd.1 on smithi173 with /dev/vg_nvme/lv_4... 2024-04-25T17:26:44.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T17:26:44.762 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:26:45.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:44 smithi098 ceph-mon[29192]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T17:26:45.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:44 smithi203 ceph-mon[32834]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T17:26:45.239 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:44 smithi173 ceph-mon[32856]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T17:26:46.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:45 smithi098 ceph-mon[29192]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:46.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:45 smithi098 ceph-mon[29192]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T17:26:46.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:45 smithi098 ceph-mon[29192]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T17:26:46.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:45 smithi098 ceph-mon[29192]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:26:46.047 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:46.047 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:26:45 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0[44416]: 2024-04-25T17:26:45.951+0000 7f9874c00700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T17:26:46.047 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:26:45 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0[44416]: 2024-04-25T17:26:45.953+0000 7f98709f3700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:26:46.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:45 smithi173 ceph-mon[32856]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:46.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:45 smithi173 ceph-mon[32856]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T17:26:46.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:45 smithi173 ceph-mon[32856]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T17:26:46.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:45 smithi173 ceph-mon[32856]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:26:46.096 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:46.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:45 smithi203 ceph-mon[32834]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:46.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:45 smithi203 ceph-mon[32834]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T17:26:46.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:45 smithi203 ceph-mon[32834]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T17:26:46.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:45 smithi203 ceph-mon[32834]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:26:46.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:46.640 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T17:26:46.640 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T17:26:46.640 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-25T17:26:46.640 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-25T17:26:46.640 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123602 s, 848 MB/s 2024-04-25T17:26:46.640 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-25T17:26:47.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:46 smithi098 ceph-mon[29192]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:26:47.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:46 smithi098 ceph-mon[29192]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T17:26:47.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:46 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:47.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:46 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:47.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:46 smithi203 ceph-mon[32834]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:26:47.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:46 smithi203 ceph-mon[32834]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T17:26:47.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:46 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:47.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:46 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:47.217 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:46 smithi173 ceph-mon[32856]: from='osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:26:47.217 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:46 smithi173 ceph-mon[32856]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T17:26:47.217 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:46 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:47.217 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:46 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:47.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-04-25T17:26:47.482 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:26:48.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:26:48.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] boot 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:48.174 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:47 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:26:48.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:26:48.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:48.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:48.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] boot 2024-04-25T17:26:48.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T17:26:48.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:48.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:26:48.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:48.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:48.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:47 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: osd.0 [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] boot 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:26:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:48.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:26:48.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:47 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.928+0000 7f9e9c697700 1 -- 172.21.15.173:0/2985459358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e94103520 msgr2=0x7f9e94105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.928+0000 7f9e9c697700 1 --2- 172.21.15.173:0/2985459358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e94103520 0x7f9e94105910 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000b0a0 tx=0x7f9e9000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.928+0000 7f9e9c697700 1 -- 172.21.15.173:0/2985459358 shutdown_connections 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.928+0000 7f9e9c697700 1 --2- 172.21.15.173:0/2985459358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e94108780 0x7f9e9410abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.928+0000 7f9e9c697700 1 --2- 172.21.15.173:0/2985459358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e94103520 0x7f9e94105910 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.928+0000 7f9e9c697700 1 --2- 172.21.15.173:0/2985459358 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e94105e50 0x7f9e94108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.928+0000 7f9e9c697700 1 -- 172.21.15.173:0/2985459358 >> 172.21.15.173:0/2985459358 conn(0x7f9e940fcf30 msgr2=0x7f9e940ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.928+0000 7f9e9c697700 1 -- 172.21.15.173:0/2985459358 shutdown_connections 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.928+0000 7f9e9c697700 1 -- 172.21.15.173:0/2985459358 wait complete. 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9c697700 1 Processor -- start 2024-04-25T17:26:48.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9c697700 1 -- start start 2024-04-25T17:26:48.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9c697700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e94103520 0x7f9e94198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:48.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9c697700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e94105e50 0x7f9e94199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:48.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9c697700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e94108780 0x7f9e941a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:48.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9c697700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9410e510 con 0x7f9e94103520 2024-04-25T17:26:48.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9c697700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9410e390 con 0x7f9e94108780 2024-04-25T17:26:48.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9c697700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9410e690 con 0x7f9e94105e50 2024-04-25T17:26:48.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9ac34700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e94108780 0x7f9e941a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:48.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9ac34700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e94108780 0x7f9e941a00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:52000/0 (socket says 172.21.15.173:52000) 2024-04-25T17:26:48.933 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9ac34700 1 -- 172.21.15.173:0/3844228749 learned_addr learned my addr 172.21.15.173:0/3844228749 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e99c32700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e94105e50 0x7f9e94199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9a433700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e94103520 0x7f9e94198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9ac34700 1 -- 172.21.15.173:0/3844228749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e94105e50 msgr2=0x7f9e94199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9ac34700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e94105e50 0x7f9e94199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9ac34700 1 -- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e94103520 msgr2=0x7f9e94198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9ac34700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e94103520 0x7f9e94198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9ac34700 1 -- 172.21.15.173:0/3844228749 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e941a0780 con 0x7f9e94108780 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.929+0000 7f9e9a433700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e94103520 0x7f9e94198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.930+0000 7f9e9ac34700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e94108780 0x7f9e941a00e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00d740 tx=0x7f9e8c00da50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.930+0000 7f9e8b7fe700 1 -- 172.21.15.173:0/3844228749 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c016070 con 0x7f9e94108780 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.930+0000 7f9e8b7fe700 1 -- 172.21.15.173:0/3844228749 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e8c00f690 con 0x7f9e94108780 2024-04-25T17:26:48.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.930+0000 7f9e9c697700 1 -- 172.21.15.173:0/3844228749 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e941a0a70 con 0x7f9e94108780 2024-04-25T17:26:48.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.930+0000 7f9e8b7fe700 1 -- 172.21.15.173:0/3844228749 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c003ae0 con 0x7f9e94108780 2024-04-25T17:26:48.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.930+0000 7f9e9c697700 1 -- 172.21.15.173:0/3844228749 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e941a0f60 con 0x7f9e94108780 2024-04-25T17:26:48.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.931+0000 7f9e9c697700 1 -- 172.21.15.173:0/3844228749 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e94069730 con 0x7f9e94108780 2024-04-25T17:26:48.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.932+0000 7f9e8b7fe700 1 -- 172.21.15.173:0/3844228749 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f9e8c003c80 con 0x7f9e94108780 2024-04-25T17:26:48.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.932+0000 7f9e8b7fe700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9e800a3bb0 0x7f9e800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:26:48.936 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.932+0000 7f9e8b7fe700 1 -- 172.21.15.173:0/3844228749 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f9e8c0ce900 con 0x7f9e94108780 2024-04-25T17:26:48.936 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.933+0000 7f9e9a433700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9e800a3bb0 0x7f9e800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:26:48.936 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.933+0000 7f9e9a433700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9e800a3bb0 0x7f9e800a6070 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009e60 tx=0x7f9e90009cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:26:48.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:48.936+0000 7f9e8b7fe700 1 -- 172.21.15.173:0/3844228749 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e8c09cd50 con 0x7f9e94108780 2024-04-25T17:26:49.147 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:26:49.146+0000 7f9e9c697700 1 -- 172.21.15.173:0/3844228749 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9e94062e80 con 0x7f9e800a3bb0 2024-04-25T17:26:49.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:48 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi098 2024-04-25T17:26:49.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:48 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi098 to 17088M 2024-04-25T17:26:49.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:48 smithi173 ceph-mon[32856]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T17:26:49.324 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:48 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi098 2024-04-25T17:26:49.324 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:48 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi098 to 17088M 2024-04-25T17:26:49.324 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:48 smithi203 ceph-mon[32834]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T17:26:49.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:48 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi098 2024-04-25T17:26:49.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:48 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi098 to 17088M 2024-04-25T17:26:49.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:48 smithi098 ceph-mon[29192]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T17:26:50.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:49 smithi173 ceph-mon[32856]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:50.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:49 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:26:50.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:49 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:26:50.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:49 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:49 smithi098 ceph-mon[29192]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:26:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:26:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:49 smithi203 ceph-mon[32834]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:49 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:26:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:49 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:26:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:49 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:51.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:50 smithi173 ceph-mon[32856]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:51.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:50 smithi098 ceph-mon[29192]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:51.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:50 smithi203 ceph-mon[32834]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:26:52.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:52 smithi173 ceph-mon[32856]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:52.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:52 smithi173 ceph-mon[32856]: from='client.? 172.21.15.173:0/682548514' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1450194d-3e02-4d6a-ac19-364cb679d83b"}]: dispatch 2024-04-25T17:26:52.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:52 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1450194d-3e02-4d6a-ac19-364cb679d83b"}]: dispatch 2024-04-25T17:26:52.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:52 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1450194d-3e02-4d6a-ac19-364cb679d83b"}]': finished 2024-04-25T17:26:52.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:52 smithi173 ceph-mon[32856]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T17:26:52.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:52 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:26:52.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:52 smithi098 ceph-mon[29192]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:52.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:52 smithi098 ceph-mon[29192]: from='client.? 172.21.15.173:0/682548514' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1450194d-3e02-4d6a-ac19-364cb679d83b"}]: dispatch 2024-04-25T17:26:52.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:52 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1450194d-3e02-4d6a-ac19-364cb679d83b"}]: dispatch 2024-04-25T17:26:52.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:52 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1450194d-3e02-4d6a-ac19-364cb679d83b"}]': finished 2024-04-25T17:26:52.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:52 smithi098 ceph-mon[29192]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T17:26:52.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:52 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:26:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:52 smithi203 ceph-mon[32834]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:52 smithi203 ceph-mon[32834]: from='client.? 172.21.15.173:0/682548514' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1450194d-3e02-4d6a-ac19-364cb679d83b"}]: dispatch 2024-04-25T17:26:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:52 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1450194d-3e02-4d6a-ac19-364cb679d83b"}]: dispatch 2024-04-25T17:26:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:52 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1450194d-3e02-4d6a-ac19-364cb679d83b"}]': finished 2024-04-25T17:26:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:52 smithi203 ceph-mon[32834]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T17:26:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:52 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:26:53.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:53 smithi173 ceph-mon[32856]: from='client.? 172.21.15.173:0/3378619639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:26:53.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:53 smithi098 ceph-mon[29192]: from='client.? 172.21.15.173:0/3378619639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:26:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:53 smithi203 ceph-mon[32834]: from='client.? 172.21.15.173:0/3378619639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:26:54.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:54 smithi173 ceph-mon[32856]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:54 smithi098 ceph-mon[29192]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:54 smithi203 ceph-mon[32834]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:56.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:56 smithi173 ceph-mon[32856]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:56.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:56 smithi098 ceph-mon[29192]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:56 smithi203 ceph-mon[32834]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:58.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:58 smithi173 ceph-mon[32856]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:58 smithi098 ceph-mon[29192]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:58 smithi203 ceph-mon[32834]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:26:59.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T17:26:59.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:26:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:59.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T17:26:59.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:26:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:26:59.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T17:26:59.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:26:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:00.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:00 smithi173 ceph-mon[32856]: Deploying daemon osd.1 on smithi173 2024-04-25T17:27:00.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:00 smithi173 ceph-mon[32856]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:00 smithi098 ceph-mon[29192]: Deploying daemon osd.1 on smithi173 2024-04-25T17:27:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:00 smithi098 ceph-mon[29192]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:00 smithi203 ceph-mon[32834]: Deploying daemon osd.1 on smithi173 2024-04-25T17:27:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:00 smithi203 ceph-mon[32834]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:02.212 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:02 smithi173 ceph-mon[32856]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:02.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:02 smithi098 ceph-mon[29192]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:02 smithi203 ceph-mon[32834]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:04.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:04 smithi173 ceph-mon[32856]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:04 smithi098 ceph-mon[29192]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:04 smithi203 ceph-mon[32834]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:05.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:05 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:05.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:05 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:05.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:05 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:05.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:05 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:05.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:05 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:05.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:05 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:05.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:05 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:05.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:05 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:05.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:05 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:06 smithi098 ceph-mon[29192]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:06.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:06 smithi203 ceph-mon[32834]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:06 smithi173 ceph-mon[32856]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:07.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:07.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:07.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:07.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:07.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:07.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:07 smithi173 ceph-mon[32856]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:08.012 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.012+0000 7f9e8b7fe700 1 -- 172.21.15.173:0/3844228749 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9e94062e80 con 0x7f9e800a3bb0 2024-04-25T17:27:08.013 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 1 on host 'smithi173' 2024-04-25T17:27:08.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 -- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9e800a3bb0 msgr2=0x7f9e800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:08.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9e800a3bb0 0x7f9e800a6070 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009e60 tx=0x7f9e90009cb0 comp rx=0 tx=0).stop 2024-04-25T17:27:08.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 -- 172.21.15.173:0/3844228749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e94108780 msgr2=0x7f9e941a00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:08.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e94108780 0x7f9e941a00e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00d740 tx=0x7f9e8c00da50 comp rx=0 tx=0).stop 2024-04-25T17:27:08.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 -- 172.21.15.173:0/3844228749 shutdown_connections 2024-04-25T17:27:08.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9e800a3bb0 0x7f9e800a6070 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:08.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e94108780 0x7f9e941a00e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:08.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e94103520 0x7f9e94198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:08.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 --2- 172.21.15.173:0/3844228749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e94105e50 0x7f9e94199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:08.015 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 -- 172.21.15.173:0/3844228749 >> 172.21.15.173:0/3844228749 conn(0x7f9e940fcf30 msgr2=0x7f9e94103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:27:08.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 -- 172.21.15.173:0/3844228749 shutdown_connections 2024-04-25T17:27:08.016 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:27:08.014+0000 7f9e9c697700 1 -- 172.21.15.173:0/3844228749 wait complete. 2024-04-25T17:27:08.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:08.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:08.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:08.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:08.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:08.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:07 smithi098 ceph-mon[29192]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:08.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:08.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:08.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:08.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:08.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:08.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:07 smithi203 ceph-mon[32834]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:08.812 DEBUG:teuthology.orchestra.run.smithi173:osd.1> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T17:27:08.814 INFO:tasks.cephadm:Deploying osd.2 on smithi203 with /dev/vg_nvme/lv_4... 2024-04-25T17:27:08.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T17:27:09.060 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:27:09.182 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:09.182 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:09.182 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:09.182 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:08 smithi203 ceph-mon[32834]: from='osd.1 [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T17:27:09.182 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:08 smithi203 ceph-mon[32834]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T17:27:09.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:09.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:09.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:09.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:08 smithi173 ceph-mon[32856]: from='osd.1 [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T17:27:09.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:08 smithi173 ceph-mon[32856]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T17:27:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:08 smithi098 ceph-mon[29192]: from='osd.1 [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T17:27:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:08 smithi098 ceph-mon[29192]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T17:27:10.236 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:10 smithi173 ceph-mon[32856]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:10 smithi173 ceph-mon[32856]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T17:27:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:10 smithi173 ceph-mon[32856]: from='osd.1 [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:27:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:10 smithi173 ceph-mon[32856]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T17:27:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:10.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:10 smithi173 ceph-mon[32856]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:27:10.237 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:27:10 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1[37499]: 2024-04-25T17:27:10.012+0000 7f023826b700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T17:27:10.237 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:27:10 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1[37499]: 2024-04-25T17:27:10.017+0000 7f023405e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:27:10.302 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:10 smithi203 ceph-mon[32834]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:10.303 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:10 smithi203 ceph-mon[32834]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T17:27:10.303 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:10 smithi203 ceph-mon[32834]: from='osd.1 [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:27:10.303 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:10 smithi203 ceph-mon[32834]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T17:27:10.303 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:10.303 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:10 smithi203 ceph-mon[32834]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:27:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:10 smithi098 ceph-mon[29192]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:10 smithi098 ceph-mon[29192]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T17:27:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:10 smithi098 ceph-mon[29192]: from='osd.1 [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:27:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:10 smithi098 ceph-mon[29192]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T17:27:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:10 smithi098 ceph-mon[29192]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:27:10.944 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T17:27:10.944 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T17:27:10.944 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-25T17:27:10.944 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-25T17:27:10.944 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113572 s, 923 MB/s 2024-04-25T17:27:10.944 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-25T17:27:11.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:11 smithi203 ceph-mon[32834]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:27:11.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:11 smithi203 ceph-mon[32834]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T17:27:11.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:11.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:11.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:11 smithi173 ceph-mon[32856]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:27:11.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:11 smithi173 ceph-mon[32856]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T17:27:11.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:11.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:11 smithi098 ceph-mon[29192]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:27:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:11 smithi098 ceph-mon[29192]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T17:27:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:11.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:11.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_4 2024-04-25T17:27:11.784 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:27:12.120 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: osd.1 [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] boot 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:12.121 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: osd.1 [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] boot 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:12.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:27:12.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:27:12.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T17:27:12.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: osd.1 [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] boot 2024-04-25T17:27:12.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T17:27:12.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:27:12.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:27:12.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:12.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:12.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:12.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:13.138 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:13 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi173 2024-04-25T17:27:13.138 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:13 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-25T17:27:13.138 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:13 smithi203 ceph-mon[32834]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T17:27:13.352 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.346+0000 7f84a151c700 1 -- 172.21.15.203:0/3284260037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f849c103cb0 msgr2=0x7f849c104110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.346+0000 7f84a151c700 1 --2- 172.21.15.203:0/3284260037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f849c103cb0 0x7f849c104110 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8490009990 tx=0x7f8490009ca0 comp rx=0 tx=0).stop 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.346+0000 7f84a151c700 1 -- 172.21.15.203:0/3284260037 shutdown_connections 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.346+0000 7f84a151c700 1 --2- 172.21.15.203:0/3284260037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f849c104650 0x7f849c10af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.346+0000 7f84a151c700 1 --2- 172.21.15.203:0/3284260037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f849c102ab0 0x7f849c102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.346+0000 7f84a151c700 1 --2- 172.21.15.203:0/3284260037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f849c103cb0 0x7f849c104110 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.346+0000 7f84a151c700 1 -- 172.21.15.203:0/3284260037 >> 172.21.15.203:0/3284260037 conn(0x7f849c0fe050 msgr2=0x7f849c100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.346+0000 7f84a151c700 1 -- 172.21.15.203:0/3284260037 shutdown_connections 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.346+0000 7f84a151c700 1 -- 172.21.15.203:0/3284260037 wait complete. 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.347+0000 7f84a151c700 1 Processor -- start 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.347+0000 7f84a151c700 1 -- start start 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.347+0000 7f84a151c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f849c102ab0 0x7f849c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:27:13.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.347+0000 7f84a151c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f849c103cb0 0x7f849c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.347+0000 7f84a151c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f849c104650 0x7f849c1a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.347+0000 7f84a151c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10e520 con 0x7f849c104650 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.347+0000 7f84a151c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10e3a0 con 0x7f849c102ab0 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.347+0000 7f84a151c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10e6a0 con 0x7f849c103cb0 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.347+0000 7f849a7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f849c103cb0 0x7f849c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f849a7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f849c103cb0 0x7f849c199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:55840/0 (socket says 172.21.15.203:55840) 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f849a7fc700 1 -- 172.21.15.203:0/2449032213 learned_addr learned my addr 172.21.15.203:0/2449032213 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f849b7fe700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f849c104650 0x7f849c1a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f849affd700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f849c102ab0 0x7f849c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f849a7fc700 1 -- 172.21.15.203:0/2449032213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f849c102ab0 msgr2=0x7f849c198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f849a7fc700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f849c102ab0 0x7f849c198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f849a7fc700 1 -- 172.21.15.203:0/2449032213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f849c104650 msgr2=0x7f849c1a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:13.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f849a7fc700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f849c104650 0x7f849c1a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f849a7fc700 1 -- 172.21.15.203:0/2449032213 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849c1a0800 con 0x7f849c103cb0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f849a7fc700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f849c103cb0 0x7f849c199130 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f84900036f0 tx=0x7f84900047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f8483fff700 1 -- 172.21.15.203:0/2449032213 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8490027070 con 0x7f849c103cb0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f84a151c700 1 -- 172.21.15.203:0/2449032213 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f849c1a0a90 con 0x7f849c103cb0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f8483fff700 1 -- 172.21.15.203:0/2449032213 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8490004000 con 0x7f849c103cb0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f8483fff700 1 -- 172.21.15.203:0/2449032213 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8490022c60 con 0x7f849c103cb0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.348+0000 7f84a151c700 1 -- 172.21.15.203:0/2449032213 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f849c1a0f40 con 0x7f849c103cb0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.349+0000 7f84a151c700 1 -- 172.21.15.203:0/2449032213 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f849c069730 con 0x7f849c103cb0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.350+0000 7f8483fff700 1 -- 172.21.15.203:0/2449032213 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f849002a070 con 0x7f849c103cb0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.351+0000 7f8483fff700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f84840a3bb0 0x7f84840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.351+0000 7f8483fff700 1 -- 172.21.15.203:0/2449032213 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f84900cfb00 con 0x7f849c103cb0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.351+0000 7f849affd700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f84840a3bb0 0x7f84840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:27:13.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.351+0000 7f849affd700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f84840a3bb0 0x7f84840a6070 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f849c103b10 tx=0x7f848c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:27:13.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.356+0000 7f8483fff700 1 -- 172.21.15.203:0/2449032213 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f849005eff0 con 0x7f849c103cb0 2024-04-25T17:27:13.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:13 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi173 2024-04-25T17:27:13.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:13 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-25T17:27:13.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:13 smithi098 ceph-mon[29192]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T17:27:13.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:13 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi173 2024-04-25T17:27:13.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:13 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-25T17:27:13.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:13 smithi173 ceph-mon[32856]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T17:27:13.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:13.564+0000 7f84a151c700 1 -- 172.21.15.203:0/2449032213 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f849c062e80 con 0x7f84840a3bb0 2024-04-25T17:27:14.355 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:14 smithi203 ceph-mon[32834]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:14.355 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:14 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:27:14.355 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:14 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:27:14.355 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:14 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:14.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:14 smithi098 ceph-mon[29192]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:14.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:14 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:27:14.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:14 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:27:14.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:14 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:14.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:14 smithi173 ceph-mon[32856]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:14.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:14 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:27:14.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:14 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:27:14.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:14 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:15.093 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:15 smithi203 ceph-mon[32834]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:27:15.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:15 smithi098 ceph-mon[29192]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:27:15.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:15 smithi173 ceph-mon[32856]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:27:16.357 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:16 smithi098 ceph-mon[29192]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:16 smithi203 ceph-mon[32834]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:16.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:16 smithi173 ceph-mon[32856]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:17.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:17 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/2077653910' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf18400e-90d7-46f8-af4a-1abca0ed8066"}]: dispatch 2024-04-25T17:27:18.122 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:17 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf18400e-90d7-46f8-af4a-1abca0ed8066"}]: dispatch 2024-04-25T17:27:18.122 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:17 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf18400e-90d7-46f8-af4a-1abca0ed8066"}]': finished 2024-04-25T17:27:18.122 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:17 smithi098 ceph-mon[29192]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T17:27:18.122 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:18.122 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:17 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/3794035535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:27:18.123 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:17 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/2077653910' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf18400e-90d7-46f8-af4a-1abca0ed8066"}]: dispatch 2024-04-25T17:27:18.123 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:17 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf18400e-90d7-46f8-af4a-1abca0ed8066"}]: dispatch 2024-04-25T17:27:18.124 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:17 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf18400e-90d7-46f8-af4a-1abca0ed8066"}]': finished 2024-04-25T17:27:18.124 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:17 smithi173 ceph-mon[32856]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T17:27:18.124 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:18.124 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:17 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/3794035535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:27:18.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:17 smithi203 ceph-mon[32834]: from='client.? 172.21.15.203:0/2077653910' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf18400e-90d7-46f8-af4a-1abca0ed8066"}]: dispatch 2024-04-25T17:27:18.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:17 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf18400e-90d7-46f8-af4a-1abca0ed8066"}]: dispatch 2024-04-25T17:27:18.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:17 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf18400e-90d7-46f8-af4a-1abca0ed8066"}]': finished 2024-04-25T17:27:18.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:17 smithi203 ceph-mon[32834]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T17:27:18.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:18.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:17 smithi203 ceph-mon[32834]: from='client.? 172.21.15.203:0/3794035535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:27:18.365 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:18 smithi203 ceph-mon[32834]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:18 smithi098 ceph-mon[29192]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:18.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:18 smithi173 ceph-mon[32856]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:20.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:20 smithi203 ceph-mon[32834]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:20.325 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:20 smithi173 ceph-mon[32856]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:20.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:20 smithi098 ceph-mon[29192]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:22.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:22 smithi203 ceph-mon[32834]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:22.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:22 smithi098 ceph-mon[29192]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:22.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:22 smithi173 ceph-mon[32856]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:24.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:24 smithi203 ceph-mon[32834]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:24.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T17:27:24.127 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:24.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:24 smithi098 ceph-mon[29192]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:24.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T17:27:24.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:24 smithi173 ceph-mon[32856]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T17:27:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:25.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:25 smithi203 ceph-mon[32834]: Deploying daemon osd.2 on smithi203 2024-04-25T17:27:25.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:25 smithi098 ceph-mon[29192]: Deploying daemon osd.2 on smithi203 2024-04-25T17:27:25.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:25 smithi173 ceph-mon[32856]: Deploying daemon osd.2 on smithi203 2024-04-25T17:27:26.317 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:26 smithi203 ceph-mon[32834]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:26.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:26 smithi098 ceph-mon[29192]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:26.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:26 smithi173 ceph-mon[32856]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:28.363 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:28 smithi203 ceph-mon[32834]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:28.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:28 smithi098 ceph-mon[29192]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:28.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:28 smithi173 ceph-mon[32856]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:29.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:29.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:29.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:29.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:29 smithi098 ceph-mon[29192]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:29.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:29.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:29.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:29.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:29 smithi203 ceph-mon[32834]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:30.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:30.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:30.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:30.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:29 smithi173 ceph-mon[32856]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:32.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:31 smithi203 ceph-mon[32834]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:32.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:32.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:32.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:32.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:32.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:31 smithi173 ceph-mon[32856]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:32.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.382+0000 7f8483fff700 1 -- 172.21.15.203:0/2449032213 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f849c062e80 con 0x7f84840a3bb0 2024-04-25T17:27:32.383 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 2 on host 'smithi203' 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.384+0000 7f84a151c700 1 -- 172.21.15.203:0/2449032213 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f84840a3bb0 msgr2=0x7f84840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.384+0000 7f84a151c700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f84840a3bb0 0x7f84840a6070 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f849c103b10 tx=0x7f848c00a380 comp rx=0 tx=0).stop 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.384+0000 7f84a151c700 1 -- 172.21.15.203:0/2449032213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f849c103cb0 msgr2=0x7f849c199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.385+0000 7f84a151c700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f849c103cb0 0x7f849c199130 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f84900036f0 tx=0x7f84900047b0 comp rx=0 tx=0).stop 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.385+0000 7f84a151c700 1 -- 172.21.15.203:0/2449032213 shutdown_connections 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.385+0000 7f84a151c700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f84840a3bb0 0x7f84840a6070 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.385+0000 7f84a151c700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f849c104650 0x7f849c1a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.385+0000 7f84a151c700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f849c102ab0 0x7f849c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.385+0000 7f84a151c700 1 --2- 172.21.15.203:0/2449032213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f849c103cb0 0x7f849c199130 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.385+0000 7f84a151c700 1 -- 172.21.15.203:0/2449032213 >> 172.21.15.203:0/2449032213 conn(0x7f849c0fe050 msgr2=0x7f849c106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.385+0000 7f84a151c700 1 -- 172.21.15.203:0/2449032213 shutdown_connections 2024-04-25T17:27:32.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:27:32.385+0000 7f84a151c700 1 -- 172.21.15.203:0/2449032213 wait complete. 2024-04-25T17:27:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:31 smithi098 ceph-mon[29192]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:33.136 DEBUG:teuthology.orchestra.run.smithi203:osd.2> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T17:27:33.139 INFO:tasks.cephadm:Deploying osd.3 on smithi098 with /dev/vg_nvme/lv_3... 2024-04-25T17:27:33.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T17:27:33.385 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:27:33.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:33 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:33.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:33 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:33.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:33 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:33.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:33 smithi098 ceph-mon[29192]: from='osd.2 [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T17:27:33.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:33 smithi098 ceph-mon[29192]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T17:27:33.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:33 smithi098 ceph-mon[29192]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:33 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:33 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:33 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:33 smithi203 ceph-mon[32834]: from='osd.2 [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T17:27:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:33 smithi203 ceph-mon[32834]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T17:27:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:33 smithi203 ceph-mon[32834]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:33.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:33 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:33.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:33 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:33.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:33 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:33.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:33 smithi173 ceph-mon[32856]: from='osd.2 [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T17:27:33.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:33 smithi173 ceph-mon[32856]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T17:27:33.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:33 smithi173 ceph-mon[32856]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:34.601 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:34 smithi203 ceph-mon[32834]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T17:27:34.601 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:34 smithi203 ceph-mon[32834]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T17:27:34.601 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:34 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:34.601 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:34 smithi203 ceph-mon[32834]: from='osd.2 [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:27:34.601 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:34 smithi203 ceph-mon[32834]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:27:34.602 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:27:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2[37999]: 2024-04-25T17:27:34.377+0000 7fe7deb10700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T17:27:34.602 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:27:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2[37999]: 2024-04-25T17:27:34.381+0000 7fe7da903700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:27:34.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:34 smithi098 ceph-mon[29192]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T17:27:34.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:34 smithi098 ceph-mon[29192]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T17:27:34.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:34 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:34.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:34 smithi098 ceph-mon[29192]: from='osd.2 [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:27:34.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:34 smithi098 ceph-mon[29192]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:27:34.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:34 smithi173 ceph-mon[32856]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T17:27:34.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:34 smithi173 ceph-mon[32856]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T17:27:34.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:34 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:34.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:34 smithi173 ceph-mon[32856]: from='osd.2 [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:27:34.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:34 smithi173 ceph-mon[32856]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:27:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T17:27:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T17:27:35.367 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-25T17:27:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-25T17:27:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137423 s, 763 MB/s 2024-04-25T17:27:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-25T17:27:35.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:35 smithi098 ceph-mon[29192]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:27:35.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:35 smithi098 ceph-mon[29192]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T17:27:35.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:35 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:35.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:35 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:35.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:35 smithi098 ceph-mon[29192]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:35.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:35 smithi203 ceph-mon[32834]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:27:35.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:35 smithi203 ceph-mon[32834]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T17:27:35.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:35 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:35.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:35 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:35.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:35 smithi203 ceph-mon[32834]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:35.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:35 smithi173 ceph-mon[32856]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:27:35.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:35 smithi173 ceph-mon[32856]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T17:27:35.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:35 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:35.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:35 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:35.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:35 smithi173 ceph-mon[32856]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T17:27:36.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-04-25T17:27:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:27:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:27:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:27:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: osd.2 [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] boot 2024-04-25T17:27:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T17:27:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:36.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:27:36.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:36.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:36.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:36 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:27:36.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:27:36.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: osd.2 [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] boot 2024-04-25T17:27:36.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T17:27:36.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:36.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:27:36.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:36.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:36.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:36 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:27:36.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:27:36.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: osd.2 [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] boot 2024-04-25T17:27:36.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T17:27:36.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:27:36.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:27:36.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:36.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:36.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:27:36.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:36 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:37.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:37 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi203 2024-04-25T17:27:37.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:37 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-25T17:27:37.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:37 smithi098 ceph-mon[29192]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T17:27:37.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:37 smithi098 ceph-mon[29192]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:37.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T17:27:37.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:37 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi203 2024-04-25T17:27:37.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:37 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-25T17:27:37.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:37 smithi203 ceph-mon[32834]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T17:27:37.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:37 smithi203 ceph-mon[32834]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:37.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T17:27:37.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:37 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi203 2024-04-25T17:27:37.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:37 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-25T17:27:37.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:37 smithi173 ceph-mon[32856]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T17:27:37.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:37 smithi173 ceph-mon[32856]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:37.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T17:27:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.013+0000 7f0171f1a700 1 -- 172.21.15.98:0/1952694737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f016c102ba0 msgr2=0x7f016c103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.013+0000 7f0171f1a700 1 --2- 172.21.15.98:0/1952694737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f016c102ba0 0x7f016c103000 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f01600099c0 tx=0x7f0160009cd0 comp rx=0 tx=0).stop 2024-04-25T17:27:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.013+0000 7f0171f1a700 1 -- 172.21.15.98:0/1952694737 shutdown_connections 2024-04-25T17:27:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.013+0000 7f0171f1a700 1 --2- 172.21.15.98:0/1952694737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f016c103540 0x7f016c109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.013+0000 7f0171f1a700 1 --2- 172.21.15.98:0/1952694737 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f016c1019a0 0x7f016c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.013+0000 7f0171f1a700 1 --2- 172.21.15.98:0/1952694737 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f016c102ba0 0x7f016c103000 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:38.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.013+0000 7f0171f1a700 1 -- 172.21.15.98:0/1952694737 >> 172.21.15.98:0/1952694737 conn(0x7f016c0fcf40 msgr2=0x7f016c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.013+0000 7f0171f1a700 1 -- 172.21.15.98:0/1952694737 shutdown_connections 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.013+0000 7f0171f1a700 1 -- 172.21.15.98:0/1952694737 wait complete. 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f0171f1a700 1 Processor -- start 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f0171f1a700 1 -- start start 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f0171f1a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f016c1019a0 0x7f016c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f0171f1a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f016c102ba0 0x7f016c199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f0171f1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f016c103540 0x7f016c1a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f0171f1a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f016c10d410 con 0x7f016c103540 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f0171f1a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f016c10d290 con 0x7f016c102ba0 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f0171f1a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f016c10d590 con 0x7f016c1019a0 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f016bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f016c103540 0x7f016c1a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:27:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f016bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f016c103540 0x7f016c1a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52060/0 (socket says 172.21.15.98:52060) 2024-04-25T17:27:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.014+0000 7f016bfff700 1 -- 172.21.15.98:0/3409714679 learned_addr learned my addr 172.21.15.98:0/3409714679 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:27:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f016b7fe700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f016c1019a0 0x7f016c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:27:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f016affd700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f016c102ba0 0x7f016c199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:27:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f016bfff700 1 -- 172.21.15.98:0/3409714679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f016c1019a0 msgr2=0x7f016c198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f016bfff700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f016c1019a0 0x7f016c198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f016bfff700 1 -- 172.21.15.98:0/3409714679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f016c102ba0 msgr2=0x7f016c199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f016bfff700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f016c102ba0 0x7f016c199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f016bfff700 1 -- 172.21.15.98:0/3409714679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f016c1a0860 con 0x7f016c103540 2024-04-25T17:27:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f016bfff700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f016c103540 0x7f016c1a0190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0164007ee0 tx=0x7f016400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:27:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f0168ff9700 1 -- 172.21.15.98:0/3409714679 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016400ddd0 con 0x7f016c103540 2024-04-25T17:27:38.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f0168ff9700 1 -- 172.21.15.98:0/3409714679 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0164004020 con 0x7f016c103540 2024-04-25T17:27:38.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f0171f1a700 1 -- 172.21.15.98:0/3409714679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f016c1a0b50 con 0x7f016c103540 2024-04-25T17:27:38.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f0168ff9700 1 -- 172.21.15.98:0/3409714679 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0164015660 con 0x7f016c103540 2024-04-25T17:27:38.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.015+0000 7f0171f1a700 1 -- 172.21.15.98:0/3409714679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f016c1a10e0 con 0x7f016c103540 2024-04-25T17:27:38.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.016+0000 7f0171f1a700 1 -- 172.21.15.98:0/3409714679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f016c069730 con 0x7f016c103540 2024-04-25T17:27:38.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.017+0000 7f0168ff9700 1 -- 172.21.15.98:0/3409714679 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f01640044f0 con 0x7f016c103540 2024-04-25T17:27:38.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.018+0000 7f0168ff9700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01540a3c30 0x7f01540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:27:38.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.018+0000 7f016b7fe700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01540a3c30 0x7f01540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:27:38.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.018+0000 7f0168ff9700 1 -- 172.21.15.98:0/3409714679 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7f01640cf180 con 0x7f016c103540 2024-04-25T17:27:38.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.018+0000 7f016b7fe700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01540a3c30 0x7f01540a60f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f016c19a0f0 tx=0x7f015c00a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:27:38.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.022+0000 7f0168ff9700 1 -- 172.21.15.98:0/3409714679 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0164010070 con 0x7f016c103540 2024-04-25T17:27:38.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:38.227+0000 7f0171f1a700 1 -- 172.21.15.98:0/3409714679 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f016c0ff160 con 0x7f01540a3c30 2024-04-25T17:27:38.666 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T17:27:38.666 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 ceph-mon[32834]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T17:27:38.666 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T17:27:38.666 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:27:38.666 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:27:38.666 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:38.666 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41598]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T17:27:38.667 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41598]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:38.667 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41598]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:38.667 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41598]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:38.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T17:27:38.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 ceph-mon[29192]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T17:27:38.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T17:27:38.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:27:38.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:27:38.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:38.707 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49165]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T17:27:38.707 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49165]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:38.707 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49165]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:38.707 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49165]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:38.707 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49168]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T17:27:38.707 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49168]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:38.707 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49168]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:38.707 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49168]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:38.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T17:27:38.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 ceph-mon[32856]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T17:27:38.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T17:27:38.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:27:38.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:27:38.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:38.774 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41323]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T17:27:38.774 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41323]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:38.774 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41323]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:38.774 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41323]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:38.774 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41326]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T17:27:38.774 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41326]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:38.774 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41326]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:38.774 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41326]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:38.963 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41601]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T17:27:38.963 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41601]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:38.963 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41601]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:38.963 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41601]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:38.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41604]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T17:27:38.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41604]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:38.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41604]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:38.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41604]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:38.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41607]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T17:27:38.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41607]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:38.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41607]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:38.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:38 smithi203 sudo[41607]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:39.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49194]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T17:27:39.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49194]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:39.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49194]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:39.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49194]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:39.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49209]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T17:27:39.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49209]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:39.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49209]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:39.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:38 smithi098 sudo[49209]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:39.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41329]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T17:27:39.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41329]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:39.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41329]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:39.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41329]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:39.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41332]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T17:27:39.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41332]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T17:27:39.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41332]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T17:27:39.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:38 smithi173 sudo[41332]: pam_unix(sudo:session): session closed for user root 2024-04-25T17:27:39.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:27:39.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T17:27:39.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:27:39.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:27:39.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:27:39.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T17:27:39.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T17:27:39.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:27:39.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:27:39.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:27:39.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T17:27:39.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:39 smithi098 ceph-mon[29192]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:39.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:27:39.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T17:27:39.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T17:27:39.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:39.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:27:39.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T17:27:39.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T17:27:39.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:27:39.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:27:39.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:27:39.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T17:27:39.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:39 smithi203 ceph-mon[32834]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:39.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:39.713+0000 7f0168ff9700 1 -- 172.21.15.98:0/3409714679 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0164098f90 con 0x7f016c103540 2024-04-25T17:27:39.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:27:39.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T17:27:39.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T17:27:39.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:39.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:39.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:39.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T17:27:39.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:27:39.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:27:39.775 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T17:27:39.775 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:39 smithi173 ceph-mon[32856]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:40.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:40 smithi098 ceph-mon[29192]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T17:27:40.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:40 smithi098 ceph-mon[29192]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T17:27:40.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:40 smithi203 ceph-mon[32834]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T17:27:40.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:40 smithi203 ceph-mon[32834]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T17:27:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:40 smithi173 ceph-mon[32856]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T17:27:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:40 smithi173 ceph-mon[32856]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T17:27:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:41 smithi098 ceph-mon[29192]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:41 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/60710609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2b72e19-266c-42bf-9cce-045bba17c014"}]: dispatch 2024-04-25T17:27:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:41 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/60710609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2b72e19-266c-42bf-9cce-045bba17c014"}]': finished 2024-04-25T17:27:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:41 smithi098 ceph-mon[29192]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T17:27:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:27:41.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:41 smithi203 ceph-mon[32834]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:41.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:41 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/60710609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2b72e19-266c-42bf-9cce-045bba17c014"}]: dispatch 2024-04-25T17:27:41.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:41 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/60710609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2b72e19-266c-42bf-9cce-045bba17c014"}]': finished 2024-04-25T17:27:41.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:41 smithi203 ceph-mon[32834]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T17:27:41.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:41 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:27:41.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:41 smithi173 ceph-mon[32856]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:41.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:41 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/60710609' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2b72e19-266c-42bf-9cce-045bba17c014"}]: dispatch 2024-04-25T17:27:41.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:41 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/60710609' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2b72e19-266c-42bf-9cce-045bba17c014"}]': finished 2024-04-25T17:27:41.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:41 smithi173 ceph-mon[32856]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T17:27:41.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:41 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:27:42.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:42 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2827477087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:27:42.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:42 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2827477087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:27:42.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:42 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2827477087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:27:43.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:43 smithi098 ceph-mon[29192]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:43.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:43 smithi203 ceph-mon[32834]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:43.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:43 smithi173 ceph-mon[32856]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:46.239 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:45 smithi098 ceph-mon[29192]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:46.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:45 smithi173 ceph-mon[32856]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:46.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:45 smithi203 ceph-mon[32834]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:48.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:47 smithi098 ceph-mon[29192]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:48.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:47 smithi203 ceph-mon[32834]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:48.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:47 smithi173 ceph-mon[32856]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:49.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:48 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T17:27:49.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:48 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:49.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:48 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T17:27:49.281 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:48 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:49.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:48 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T17:27:49.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:48 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:27:50.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:49 smithi173 ceph-mon[32856]: Deploying daemon osd.3 on smithi098 2024-04-25T17:27:50.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:49 smithi173 ceph-mon[32856]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:50.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:49 smithi098 ceph-mon[29192]: Deploying daemon osd.3 on smithi098 2024-04-25T17:27:50.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:49 smithi098 ceph-mon[29192]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:50.460 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:49 smithi203 ceph-mon[32834]: Deploying daemon osd.3 on smithi098 2024-04-25T17:27:50.460 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:49 smithi203 ceph-mon[32834]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:52.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:51 smithi098 ceph-mon[29192]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:52.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:51 smithi173 ceph-mon[32856]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:51 smithi203 ceph-mon[32834]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:54.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:53 smithi173 ceph-mon[32856]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:53 smithi098 ceph-mon[29192]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:54.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:53 smithi203 ceph-mon[32834]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:55 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:55 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:55 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:55 smithi098 ceph-mon[29192]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:55 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:55 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:55 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:55 smithi203 ceph-mon[32834]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:55.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:55 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:55 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:55 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:27:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:55 smithi173 ceph-mon[32856]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:57.786 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 3 on host 'smithi098' 2024-04-25T17:27:57.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.785+0000 7f0168ff9700 1 -- 172.21.15.98:0/3409714679 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f016c0ff160 con 0x7f01540a3c30 2024-04-25T17:27:57.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 -- 172.21.15.98:0/3409714679 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01540a3c30 msgr2=0x7f01540a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:57.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01540a3c30 0x7f01540a60f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f016c19a0f0 tx=0x7f015c00a5e0 comp rx=0 tx=0).stop 2024-04-25T17:27:57.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 -- 172.21.15.98:0/3409714679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f016c103540 msgr2=0x7f016c1a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:27:57.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f016c103540 0x7f016c1a0190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0164007ee0 tx=0x7f016400d3b0 comp rx=0 tx=0).stop 2024-04-25T17:27:57.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 -- 172.21.15.98:0/3409714679 shutdown_connections 2024-04-25T17:27:57.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01540a3c30 0x7f01540a60f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:57.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f016c1019a0 0x7f016c198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:57.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f016c103540 0x7f016c1a0190 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:57.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 --2- 172.21.15.98:0/3409714679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f016c102ba0 0x7f016c199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:27:57.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 -- 172.21.15.98:0/3409714679 >> 172.21.15.98:0/3409714679 conn(0x7f016c0fcf40 msgr2=0x7f016c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:27:57.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 -- 172.21.15.98:0/3409714679 shutdown_connections 2024-04-25T17:27:57.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:27:57.787+0000 7f0171f1a700 1 -- 172.21.15.98:0/3409714679 wait complete. 2024-04-25T17:27:58.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:57 smithi173 ceph-mon[32856]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:58.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:57 smithi098 ceph-mon[29192]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:57 smithi203 ceph-mon[32834]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:27:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:27:58.511 DEBUG:teuthology.orchestra.run.smithi098:osd.3> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T17:27:58.514 INFO:tasks.cephadm:Deploying osd.4 on smithi173 with /dev/vg_nvme/lv_3... 2024-04-25T17:27:58.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T17:27:58.760 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:27:59.157 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:58 smithi173 ceph-mon[32856]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T17:27:59.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:58 smithi098 ceph-mon[29192]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T17:27:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:58 smithi203 ceph-mon[32834]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T17:28:00.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:59 smithi173 ceph-mon[32856]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:28:00.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:59 smithi173 ceph-mon[32856]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T17:28:00.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:59 smithi173 ceph-mon[32856]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T17:28:00.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:59 smithi173 ceph-mon[32856]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:28:00.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:27:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:00.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:59 smithi098 ceph-mon[29192]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:28:00.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:59 smithi098 ceph-mon[29192]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T17:28:00.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:59 smithi098 ceph-mon[29192]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T17:28:00.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:59 smithi098 ceph-mon[29192]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:28:00.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:27:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:00.246 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:27:59 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3[51380]: 2024-04-25T17:27:59.981+0000 7fe8b14f6700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T17:28:00.246 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:27:59 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3[51380]: 2024-04-25T17:27:59.985+0000 7fe8ad2e9700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:28:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:59 smithi203 ceph-mon[32834]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:28:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:59 smithi203 ceph-mon[32834]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T17:28:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:59 smithi203 ceph-mon[32834]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T17:28:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:59 smithi203 ceph-mon[32834]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:28:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:27:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:00.782 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T17:28:00.782 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T17:28:00.782 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-25T17:28:00.782 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-25T17:28:00.782 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114624 s, 915 MB/s 2024-04-25T17:28:00.782 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-25T17:28:01.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:00 smithi173 ceph-mon[32856]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:28:01.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:00 smithi173 ceph-mon[32856]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T17:28:01.092 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:00 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:01.093 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:00 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:01.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:00 smithi098 ceph-mon[29192]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:28:01.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:00 smithi098 ceph-mon[29192]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T17:28:01.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:01.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:01.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-04-25T17:28:01.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:00 smithi203 ceph-mon[32834]: from='osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:28:01.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:00 smithi203 ceph-mon[32834]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T17:28:01.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:01.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:01.602 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:28:02.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:28:02.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:28:02.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:28:02.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:02.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] boot 2024-04-25T17:28:02.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T17:28:02.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:02.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:28:02.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] boot 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:02.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:02.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:02.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:28:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:28:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-25T17:28:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: osd.3 [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] boot 2024-04-25T17:28:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T17:28:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:28:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:02.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:02.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:02.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:02.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:02.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:02.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:02.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:02.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:03.260 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:02 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi098 2024-04-25T17:28:03.260 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:02 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi098 to 8544M 2024-04-25T17:28:03.261 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:02 smithi173 ceph-mon[32856]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T17:28:03.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.261+0000 7fd78b343700 1 -- 172.21.15.173:0/3897719497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd784102fa0 msgr2=0x7fd7841033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:03.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.261+0000 7fd78b343700 1 --2- 172.21.15.173:0/3897719497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd784102fa0 0x7fd7841033c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd780009990 tx=0x7fd780009ca0 comp rx=0 tx=0).stop 2024-04-25T17:28:03.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 -- 172.21.15.173:0/3897719497 shutdown_connections 2024-04-25T17:28:03.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 --2- 172.21.15.173:0/3897719497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7841025c0 0x7fd784109e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:03.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 --2- 172.21.15.173:0/3897719497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd784102fa0 0x7fd7841033c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:03.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 --2- 172.21.15.173:0/3897719497 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd784101c00 0x7fd784102080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:03.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 -- 172.21.15.173:0/3897719497 >> 172.21.15.173:0/3897719497 conn(0x7fd7840fc6d0 msgr2=0x7fd7840feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:28:03.263 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 -- 172.21.15.173:0/3897719497 shutdown_connections 2024-04-25T17:28:03.264 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 -- 172.21.15.173:0/3897719497 wait complete. 2024-04-25T17:28:03.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 Processor -- start 2024-04-25T17:28:03.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 -- start start 2024-04-25T17:28:03.265 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784101c00 0x7fd784198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.262+0000 7fd78b343700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7841025c0 0x7fd7841990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd78b343700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd784102fa0 0x7fd7841a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd78b343700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd78410d410 con 0x7fd784102fa0 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd78b343700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd78410d290 con 0x7fd7841025c0 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd78b343700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd78410d590 con 0x7fd784101c00 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7888de700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7841025c0 0x7fd7841990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7888de700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7841025c0 0x7fd7841990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:52938/0 (socket says 172.21.15.173:52938) 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7888de700 1 -- 172.21.15.173:0/144548395 learned_addr learned my addr 172.21.15.173:0/144548395 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7890df700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784101c00 0x7fd784198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7898e0700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd784102fa0 0x7fd7841a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7888de700 1 -- 172.21.15.173:0/144548395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784101c00 msgr2=0x7fd784198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:03.266 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7888de700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784101c00 0x7fd784198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7888de700 1 -- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd784102fa0 msgr2=0x7fd7841a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7888de700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd784102fa0 0x7fd7841a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7888de700 1 -- 172.21.15.173:0/144548395 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7841a07d0 con 0x7fd7841025c0 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7890df700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784101c00 0x7fd784198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7898e0700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd784102fa0 0x7fd7841a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd7888de700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7841025c0 0x7fd7841990b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd77400d980 tx=0x7fd77400dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd77a7fc700 1 -- 172.21.15.173:0/144548395 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774016070 con 0x7fd7841025c0 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd77a7fc700 1 -- 172.21.15.173:0/144548395 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7740044d0 con 0x7fd7841025c0 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd78b343700 1 -- 172.21.15.173:0/144548395 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7841a0ac0 con 0x7fd7841025c0 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd77a7fc700 1 -- 172.21.15.173:0/144548395 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774004cf0 con 0x7fd7841025c0 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.263+0000 7fd78b343700 1 -- 172.21.15.173:0/144548395 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7841a1050 con 0x7fd7841025c0 2024-04-25T17:28:03.267 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.264+0000 7fd78b343700 1 -- 172.21.15.173:0/144548395 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd784069730 con 0x7fd7841025c0 2024-04-25T17:28:03.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.266+0000 7fd77a7fc700 1 -- 172.21.15.173:0/144548395 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd774004020 con 0x7fd7841025c0 2024-04-25T17:28:03.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.266+0000 7fd77a7fc700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7700a3cc0 0x7fd7700a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:03.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.266+0000 7fd77a7fc700 1 -- 172.21.15.173:0/144548395 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3698+0+0 (secure 0 0 0) 0x7fd7740cf5e0 con 0x7fd7841025c0 2024-04-25T17:28:03.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.266+0000 7fd7890df700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7700a3cc0 0x7fd7700a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:03.268 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.267+0000 7fd7890df700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7700a3cc0 0x7fd7700a6180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd780006010 tx=0x7fd780009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:28:03.271 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.270+0000 7fd77a7fc700 1 -- 172.21.15.173:0/144548395 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd77401f020 con 0x7fd7841025c0 2024-04-25T17:28:03.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:02 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi098 2024-04-25T17:28:03.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:02 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi098 to 8544M 2024-04-25T17:28:03.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:02 smithi098 ceph-mon[29192]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T17:28:03.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:02 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi098 2024-04-25T17:28:03.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:02 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi098 to 8544M 2024-04-25T17:28:03.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:02 smithi203 ceph-mon[32834]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T17:28:03.479 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:03.478+0000 7fd78b343700 1 -- 172.21.15.173:0/144548395 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd784101200 con 0x7fd7700a3cc0 2024-04-25T17:28:04.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:04 smithi173 ceph-mon[32856]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:04.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:04 smithi173 ceph-mon[32856]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T17:28:04.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:04 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:28:04.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:04 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:28:04.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:04 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:04 smithi098 ceph-mon[29192]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:04 smithi098 ceph-mon[29192]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T17:28:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:04 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:28:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:04 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:28:04.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:04 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:04 smithi203 ceph-mon[32834]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:04 smithi203 ceph-mon[32834]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T17:28:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:04 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:28:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:04 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:28:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:04 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:05.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:05 smithi173 ceph-mon[32856]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:28:05.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:05 smithi173 ceph-mon[32856]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T17:28:05.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:05 smithi098 ceph-mon[29192]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:28:05.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:05 smithi098 ceph-mon[29192]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T17:28:05.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:05 smithi203 ceph-mon[32834]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:28:05.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:05 smithi203 ceph-mon[32834]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T17:28:06.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:06 smithi173 ceph-mon[32856]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:06.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:06 smithi173 ceph-mon[32856]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T17:28:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:06 smithi098 ceph-mon[29192]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:06 smithi098 ceph-mon[29192]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T17:28:06.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:06 smithi203 ceph-mon[32834]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:06.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:06 smithi203 ceph-mon[32834]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T17:28:07.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:07 smithi173 ceph-mon[32856]: from='client.? 172.21.15.173:0/2348482509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2773c8f9-a178-40dc-9da0-51a8ac5d163b"}]: dispatch 2024-04-25T17:28:07.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:07 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2773c8f9-a178-40dc-9da0-51a8ac5d163b"}]: dispatch 2024-04-25T17:28:07.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:07 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2773c8f9-a178-40dc-9da0-51a8ac5d163b"}]': finished 2024-04-25T17:28:07.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:07 smithi173 ceph-mon[32856]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T17:28:07.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:07.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:07 smithi173 ceph-mon[32856]: from='client.? 172.21.15.173:0/1649069601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:28:07.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:07 smithi098 ceph-mon[29192]: from='client.? 172.21.15.173:0/2348482509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2773c8f9-a178-40dc-9da0-51a8ac5d163b"}]: dispatch 2024-04-25T17:28:07.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:07 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2773c8f9-a178-40dc-9da0-51a8ac5d163b"}]: dispatch 2024-04-25T17:28:07.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:07 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2773c8f9-a178-40dc-9da0-51a8ac5d163b"}]': finished 2024-04-25T17:28:07.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:07 smithi098 ceph-mon[29192]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T17:28:07.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:07.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:07 smithi098 ceph-mon[29192]: from='client.? 172.21.15.173:0/1649069601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:28:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:07 smithi203 ceph-mon[32834]: from='client.? 172.21.15.173:0/2348482509' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2773c8f9-a178-40dc-9da0-51a8ac5d163b"}]: dispatch 2024-04-25T17:28:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:07 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2773c8f9-a178-40dc-9da0-51a8ac5d163b"}]: dispatch 2024-04-25T17:28:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:07 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2773c8f9-a178-40dc-9da0-51a8ac5d163b"}]': finished 2024-04-25T17:28:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:07 smithi203 ceph-mon[32834]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T17:28:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:07 smithi203 ceph-mon[32834]: from='client.? 172.21.15.173:0/1649069601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:28:08.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:08 smithi173 ceph-mon[32856]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:08.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:08 smithi098 ceph-mon[29192]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:08 smithi203 ceph-mon[32834]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:10.233 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:10 smithi173 ceph-mon[32856]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T17:28:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:10 smithi098 ceph-mon[29192]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T17:28:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:10 smithi203 ceph-mon[32834]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T17:28:12.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:12 smithi173 ceph-mon[32856]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-25T17:28:12.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:12 smithi098 ceph-mon[29192]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-25T17:28:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:12 smithi203 ceph-mon[32834]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-25T17:28:14.357 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:14 smithi173 ceph-mon[32856]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T17:28:14.357 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:14 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T17:28:14.358 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:14 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:14.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:14 smithi098 ceph-mon[29192]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T17:28:14.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:14 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T17:28:14.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:14 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:14 smithi203 ceph-mon[32834]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T17:28:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:14 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T17:28:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:14 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:15.321 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:15 smithi173 ceph-mon[32856]: Deploying daemon osd.4 on smithi173 2024-04-25T17:28:15.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:15 smithi098 ceph-mon[29192]: Deploying daemon osd.4 on smithi173 2024-04-25T17:28:15.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:15 smithi203 ceph-mon[32834]: Deploying daemon osd.4 on smithi173 2024-04-25T17:28:16.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:16 smithi173 ceph-mon[32856]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T17:28:16.357 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:16 smithi098 ceph-mon[29192]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T17:28:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:16 smithi203 ceph-mon[32834]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T17:28:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:18 smithi098 ceph-mon[29192]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-25T17:28:18.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:18 smithi203 ceph-mon[32834]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-25T17:28:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:18 smithi173 ceph-mon[32856]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-04-25T17:28:20.147 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:20 smithi173 ceph-mon[32856]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T17:28:20.147 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:20.147 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:20.147 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:20.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:20 smithi098 ceph-mon[29192]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T17:28:20.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:20.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:20.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:20.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:20 smithi203 ceph-mon[32834]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T17:28:20.466 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:20.466 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:20.466 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:22.252 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:22 smithi173 ceph-mon[32856]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:22.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:22 smithi098 ceph-mon[29192]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:22.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:22 smithi203 ceph-mon[32834]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:23.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.343+0000 7fd77a7fc700 1 -- 172.21.15.173:0/144548395 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd784101200 con 0x7fd7700a3cc0 2024-04-25T17:28:23.345 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 4 on host 'smithi173' 2024-04-25T17:28:23.347 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 -- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7700a3cc0 msgr2=0x7fd7700a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7700a3cc0 0x7fd7700a6180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd780006010 tx=0x7fd780009510 comp rx=0 tx=0).stop 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 -- 172.21.15.173:0/144548395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7841025c0 msgr2=0x7fd7841990b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7841025c0 0x7fd7841990b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd77400d980 tx=0x7fd77400dd40 comp rx=0 tx=0).stop 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 -- 172.21.15.173:0/144548395 shutdown_connections 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7700a3cc0 0x7fd7700a6180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784101c00 0x7fd784198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd784102fa0 0x7fd7841a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 --2- 172.21.15.173:0/144548395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7841025c0 0x7fd7841990b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 -- 172.21.15.173:0/144548395 >> 172.21.15.173:0/144548395 conn(0x7fd7840fc6d0 msgr2=0x7fd784100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 -- 172.21.15.173:0/144548395 shutdown_connections 2024-04-25T17:28:23.348 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:28:23.346+0000 7fd78b343700 1 -- 172.21.15.173:0/144548395 wait complete. 2024-04-25T17:28:23.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:23.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:23.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:23 smithi173 ceph-mon[32856]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:23.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:23.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:23.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:23.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:23 smithi098 ceph-mon[29192]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:23.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:23 smithi203 ceph-mon[32834]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:23.977 DEBUG:teuthology.orchestra.run.smithi173:osd.4> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T17:28:23.980 INFO:tasks.cephadm:Deploying osd.5 on smithi203 with /dev/vg_nvme/lv_3... 2024-04-25T17:28:23.980 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T17:28:24.226 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:28:24.667 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:24 smithi203 ceph-mon[32834]: from='osd.4 [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T17:28:24.667 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:24 smithi203 ceph-mon[32834]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T17:28:24.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:24 smithi098 ceph-mon[29192]: from='osd.4 [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T17:28:24.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:24 smithi098 ceph-mon[29192]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T17:28:24.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:24 smithi173 ceph-mon[32856]: from='osd.4 [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T17:28:24.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:24 smithi173 ceph-mon[32856]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T17:28:25.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:25 smithi173 ceph-mon[32856]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T17:28:25.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:25 smithi173 ceph-mon[32856]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T17:28:25.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:25 smithi173 ceph-mon[32856]: from='osd.4 [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:28:25.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:25 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:25.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:25 smithi173 ceph-mon[32856]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:28:25.487 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:25 smithi173 ceph-mon[32856]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:25.668 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:25 smithi203 ceph-mon[32834]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T17:28:25.668 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:25 smithi203 ceph-mon[32834]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T17:28:25.668 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:25 smithi203 ceph-mon[32834]: from='osd.4 [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:28:25.668 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:25 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:25.668 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:25 smithi203 ceph-mon[32834]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:28:25.668 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:25 smithi203 ceph-mon[32834]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:25.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:25 smithi098 ceph-mon[29192]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T17:28:25.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:25 smithi098 ceph-mon[29192]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T17:28:25.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:25 smithi098 ceph-mon[29192]: from='osd.4 [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:28:25.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:25 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:25.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:25 smithi098 ceph-mon[29192]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:28:25.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:25 smithi098 ceph-mon[29192]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:26.233 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T17:28:26.233 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T17:28:26.233 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-25T17:28:26.233 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-25T17:28:26.233 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012745 s, 823 MB/s 2024-04-25T17:28:26.233 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-25T17:28:26.273 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:28:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4[44145]: 2024-04-25T17:28:25.874+0000 7fcfefd9e700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T17:28:26.273 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:28:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4[44145]: 2024-04-25T17:28:25.880+0000 7fcfe7b89700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:28:26.627 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:26 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:28:26.628 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:26 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:28:26.628 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:26 smithi203 ceph-mon[32834]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:28:26.628 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:26 smithi203 ceph-mon[32834]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T17:28:26.628 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:26 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:26.628 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:26 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:26.628 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:26 smithi203 ceph-mon[32834]: from='osd.4 ' entity='osd.4' 2024-04-25T17:28:26.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:26 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:28:26.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:26 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:28:26.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:26 smithi173 ceph-mon[32856]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:28:26.634 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:26 smithi173 ceph-mon[32856]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T17:28:26.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:26 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:26.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:26 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:26.635 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:26 smithi173 ceph-mon[32856]: from='osd.4 ' entity='osd.4' 2024-04-25T17:28:26.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:26 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:28:26.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:26 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:28:26.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:26 smithi098 ceph-mon[29192]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:28:26.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:26 smithi098 ceph-mon[29192]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T17:28:26.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:26 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:26.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:26 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:26.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:26 smithi098 ceph-mon[29192]: from='osd.4 ' entity='osd.4' 2024-04-25T17:28:26.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_3 2024-04-25T17:28:27.192 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:28:27.639 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:27.639 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: osd.4 [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] boot 2024-04-25T17:28:27.639 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T17:28:27.639 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi173 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:27.640 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:27.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: osd.4 [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] boot 2024-04-25T17:28:27.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T17:28:27.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:27.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:27.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi173 2024-04-25T17:28:27.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:27.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:27.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-25T17:28:27.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:27.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:27.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:27.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: osd.4 [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] boot 2024-04-25T17:28:27.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T17:28:27.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:28:27.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-04-25T17:28:27.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi173 2024-04-25T17:28:27.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:27.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:27.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-25T17:28:27.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:27.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:27.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:27.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:28.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.823+0000 7ff754cbe700 1 -- 172.21.15.203:0/145933063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7501019a0 msgr2=0x7ff750101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:28.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.823+0000 7ff754cbe700 1 --2- 172.21.15.203:0/145933063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7501019a0 0x7ff750101dc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff74000b0a0 tx=0x7ff74000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:28:28.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.823+0000 7ff754cbe700 1 -- 172.21.15.203:0/145933063 shutdown_connections 2024-04-25T17:28:28.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.823+0000 7ff754cbe700 1 --2- 172.21.15.203:0/145933063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7501019a0 0x7ff750101dc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:28.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.823+0000 7ff754cbe700 1 --2- 172.21.15.203:0/145933063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff750103540 0x7ff750109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:28.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.823+0000 7ff754cbe700 1 --2- 172.21.15.203:0/145933063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff750102ba0 0x7ff750103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:28.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.823+0000 7ff754cbe700 1 -- 172.21.15.203:0/145933063 >> 172.21.15.203:0/145933063 conn(0x7ff7500fcf40 msgr2=0x7ff7500ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:28:28.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.823+0000 7ff754cbe700 1 -- 172.21.15.203:0/145933063 shutdown_connections 2024-04-25T17:28:28.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.823+0000 7ff754cbe700 1 -- 172.21.15.203:0/145933063 wait complete. 2024-04-25T17:28:28.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff754cbe700 1 Processor -- start 2024-04-25T17:28:28.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff754cbe700 1 -- start start 2024-04-25T17:28:28.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff754cbe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7501019a0 0x7ff750198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:28.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff754cbe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff750102ba0 0x7ff750199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff754cbe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff750103540 0x7ff7501a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff754cbe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff75010e5b0 con 0x7ff7501019a0 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff754cbe700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff75010e430 con 0x7ff750103540 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff754cbe700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff75010e730 con 0x7ff750102ba0 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff74dd9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff750102ba0 0x7ff750199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff74dd9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff750102ba0 0x7ff750199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:41878/0 (socket says 172.21.15.203:41878) 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff74dd9b700 1 -- 172.21.15.203:0/1092099744 learned_addr learned my addr 172.21.15.203:0/1092099744 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff74ed9d700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff750103540 0x7ff7501a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.824+0000 7ff74e59c700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7501019a0 0x7ff750198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff74dd9b700 1 -- 172.21.15.203:0/1092099744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff750103540 msgr2=0x7ff7501a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff74dd9b700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff750103540 0x7ff7501a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff74dd9b700 1 -- 172.21.15.203:0/1092099744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7501019a0 msgr2=0x7ff750198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff74dd9b700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7501019a0 0x7ff750198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:28.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff74dd9b700 1 -- 172.21.15.203:0/1092099744 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7501a0950 con 0x7ff750102ba0 2024-04-25T17:28:28.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff74dd9b700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff750102ba0 0x7ff750199240 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff73800b530 tx=0x7ff73800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:28:28.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff7477fe700 1 -- 172.21.15.203:0/1092099744 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff738015020 con 0x7ff750102ba0 2024-04-25T17:28:28.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff7477fe700 1 -- 172.21.15.203:0/1092099744 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff738011ce0 con 0x7ff750102ba0 2024-04-25T17:28:28.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff7477fe700 1 -- 172.21.15.203:0/1092099744 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff738010dd0 con 0x7ff750102ba0 2024-04-25T17:28:28.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff754cbe700 1 -- 172.21.15.203:0/1092099744 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7501a0c40 con 0x7ff750102ba0 2024-04-25T17:28:28.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.825+0000 7ff754cbe700 1 -- 172.21.15.203:0/1092099744 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7501a1130 con 0x7ff750102ba0 2024-04-25T17:28:28.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.826+0000 7ff754cbe700 1 -- 172.21.15.203:0/1092099744 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff750069730 con 0x7ff750102ba0 2024-04-25T17:28:28.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.827+0000 7ff7477fe700 1 -- 172.21.15.203:0/1092099744 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff738018020 con 0x7ff750102ba0 2024-04-25T17:28:28.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.827+0000 7ff7477fe700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff73c0a3c40 0x7ff73c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:28.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.828+0000 7ff7477fe700 1 -- 172.21.15.203:0/1092099744 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7ff7380cea50 con 0x7ff750102ba0 2024-04-25T17:28:28.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.828+0000 7ff74e59c700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff73c0a3c40 0x7ff73c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:28.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.828+0000 7ff74e59c700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff73c0a3c40 0x7ff73c0a6100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff740009fd0 tx=0x7ff74000b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:28:28.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:28.832+0000 7ff7477fe700 1 -- 172.21.15.203:0/1092099744 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff73801f020 con 0x7ff750102ba0 2024-04-25T17:28:29.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:29.038+0000 7ff754cbe700 1 -- 172.21.15.203:0/1092099744 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff750062e80 con 0x7ff73c0a3c40 2024-04-25T17:28:29.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:29 smithi098 ceph-mon[29192]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T17:28:29.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:29 smithi098 ceph-mon[29192]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:29.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:29 smithi098 ceph-mon[29192]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:28:29.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:28:29.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:28:29.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:29.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:29 smithi203 ceph-mon[32834]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T17:28:29.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:29 smithi203 ceph-mon[32834]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:29.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:29 smithi203 ceph-mon[32834]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:28:29.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:28:29.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:28:29.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:29.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:29 smithi173 ceph-mon[32856]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T17:28:29.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:29 smithi173 ceph-mon[32856]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:29.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:29 smithi173 ceph-mon[32856]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:28:29.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:28:29.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:28:29.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:31 smithi173 ceph-mon[32856]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:31 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/46741496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21751edb-9f9a-4bce-a13a-2e728141a500"}]: dispatch 2024-04-25T17:28:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:31 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21751edb-9f9a-4bce-a13a-2e728141a500"}]: dispatch 2024-04-25T17:28:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:31 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21751edb-9f9a-4bce-a13a-2e728141a500"}]': finished 2024-04-25T17:28:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:31 smithi173 ceph-mon[32856]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T17:28:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:31 smithi098 ceph-mon[29192]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:31 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/46741496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21751edb-9f9a-4bce-a13a-2e728141a500"}]: dispatch 2024-04-25T17:28:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:31 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21751edb-9f9a-4bce-a13a-2e728141a500"}]: dispatch 2024-04-25T17:28:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:31 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21751edb-9f9a-4bce-a13a-2e728141a500"}]': finished 2024-04-25T17:28:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:31 smithi098 ceph-mon[29192]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T17:28:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:31 smithi203 ceph-mon[32834]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:31 smithi203 ceph-mon[32834]: from='client.? 172.21.15.203:0/46741496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21751edb-9f9a-4bce-a13a-2e728141a500"}]: dispatch 2024-04-25T17:28:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:31 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21751edb-9f9a-4bce-a13a-2e728141a500"}]: dispatch 2024-04-25T17:28:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:31 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21751edb-9f9a-4bce-a13a-2e728141a500"}]': finished 2024-04-25T17:28:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:31 smithi203 ceph-mon[32834]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T17:28:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:33.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:32 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/1117492551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:28:33.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:32 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/1117492551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:28:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:32 smithi203 ceph-mon[32834]: from='client.? 172.21.15.203:0/1117492551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:28:34.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:33 smithi173 ceph-mon[32856]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:34.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:33 smithi098 ceph-mon[29192]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:33 smithi203 ceph-mon[32834]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:36.078 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:35 smithi203 ceph-mon[32834]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:36.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:35 smithi173 ceph-mon[32856]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:36.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:35 smithi098 ceph-mon[29192]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:38.262 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:37 smithi203 ceph-mon[32834]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:38.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:37 smithi173 ceph-mon[32856]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:37 smithi098 ceph-mon[29192]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:40.110 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:39 smithi203 ceph-mon[32834]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:40.111 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T17:28:40.111 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:40.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:39 smithi173 ceph-mon[32856]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:40.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T17:28:40.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:40.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:39 smithi098 ceph-mon[29192]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:40.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T17:28:40.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:41.014 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:40 smithi203 ceph-mon[32834]: Deploying daemon osd.5 on smithi203 2024-04-25T17:28:41.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:40 smithi173 ceph-mon[32856]: Deploying daemon osd.5 on smithi203 2024-04-25T17:28:41.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:40 smithi098 ceph-mon[29192]: Deploying daemon osd.5 on smithi203 2024-04-25T17:28:42.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:41 smithi203 ceph-mon[32834]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:42.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:41 smithi173 ceph-mon[32856]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:41 smithi098 ceph-mon[29192]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:44.262 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:43 smithi203 ceph-mon[32834]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:44.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:43 smithi173 ceph-mon[32856]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:44.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:43 smithi098 ceph-mon[29192]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:46.204 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:46.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:46.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:46.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:45 smithi203 ceph-mon[32834]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:46.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:46.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:46.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:46.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:45 smithi098 ceph-mon[29192]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:45 smithi173 ceph-mon[32856]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:48.031 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:47 smithi203 ceph-mon[32834]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:48.103 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 5 on host 'smithi203' 2024-04-25T17:28:48.103 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.101+0000 7ff7477fe700 1 -- 172.21.15.203:0/1092099744 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff750062e80 con 0x7ff73c0a3c40 2024-04-25T17:28:48.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 -- 172.21.15.203:0/1092099744 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff73c0a3c40 msgr2=0x7ff73c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:48.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff73c0a3c40 0x7ff73c0a6100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff740009fd0 tx=0x7ff74000b7b0 comp rx=0 tx=0).stop 2024-04-25T17:28:48.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 -- 172.21.15.203:0/1092099744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff750102ba0 msgr2=0x7ff750199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:48.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff750102ba0 0x7ff750199240 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff73800b530 tx=0x7ff73800b840 comp rx=0 tx=0).stop 2024-04-25T17:28:48.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 -- 172.21.15.203:0/1092099744 shutdown_connections 2024-04-25T17:28:48.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff73c0a3c40 0x7ff73c0a6100 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:48.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff750103540 0x7ff7501a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:48.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7501019a0 0x7ff750198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:48.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 --2- 172.21.15.203:0/1092099744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff750102ba0 0x7ff750199240 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:48.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 -- 172.21.15.203:0/1092099744 >> 172.21.15.203:0/1092099744 conn(0x7ff7500fcf40 msgr2=0x7ff750106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:28:48.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 -- 172.21.15.203:0/1092099744 shutdown_connections 2024-04-25T17:28:48.106 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:28:48.104+0000 7ff754cbe700 1 -- 172.21.15.203:0/1092099744 wait complete. 2024-04-25T17:28:48.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:47 smithi173 ceph-mon[32856]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:48.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:47 smithi098 ceph-mon[29192]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:49.106 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T17:28:49.108 INFO:tasks.cephadm:Deploying osd.6 on smithi098 with /dev/vg_nvme/lv_2... 2024-04-25T17:28:49.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T17:28:49.351 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:28:49.376 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:49 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:49.376 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:49 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:49.376 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:49 smithi203 ceph-mon[32834]: from='osd.5 [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T17:28:49.376 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:49 smithi203 ceph-mon[32834]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T17:28:49.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:49.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:49.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:49 smithi098 ceph-mon[29192]: from='osd.5 [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T17:28:49.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:49 smithi098 ceph-mon[29192]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T17:28:49.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:49 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:49.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:49 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:49.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:49 smithi173 ceph-mon[32856]: from='osd.5 [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T17:28:49.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:49 smithi173 ceph-mon[32856]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T17:28:50.380 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:50 smithi203 ceph-mon[32834]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:50.380 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:50 smithi203 ceph-mon[32834]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T17:28:50.380 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:50 smithi203 ceph-mon[32834]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T17:28:50.380 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:50.380 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:50 smithi203 ceph-mon[32834]: from='osd.5 [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:28:50.381 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:50 smithi203 ceph-mon[32834]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:28:50.381 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:50.381 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:50.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:50 smithi173 ceph-mon[32856]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:50.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:50 smithi173 ceph-mon[32856]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T17:28:50.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:50 smithi173 ceph-mon[32856]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T17:28:50.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:50.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:50 smithi173 ceph-mon[32856]: from='osd.5 [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:28:50.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:50 smithi173 ceph-mon[32856]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:28:50.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:50.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:50 smithi098 ceph-mon[29192]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:50 smithi098 ceph-mon[29192]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T17:28:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:50 smithi098 ceph-mon[29192]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T17:28:50.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:50.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:50 smithi098 ceph-mon[29192]: from='osd.5 [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:28:50.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:50 smithi098 ceph-mon[29192]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:28:50.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:50.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:50.713 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:28:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:28:50.602+0000 7f5ac4961700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T17:28:50.713 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:28:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:28:50.609+0000 7f5abbf4b700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:28:51.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:51 smithi203 ceph-mon[32834]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:28:51.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:51 smithi203 ceph-mon[32834]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T17:28:51.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:51 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:51.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:51 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:51.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:51 smithi203 ceph-mon[32834]: from='osd.5 ' entity='osd.5' 2024-04-25T17:28:51.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:51 smithi173 ceph-mon[32856]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:28:51.439 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:51 smithi173 ceph-mon[32856]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T17:28:51.439 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:51 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:51.439 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:51 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:51.439 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:51 smithi173 ceph-mon[32856]: from='osd.5 ' entity='osd.5' 2024-04-25T17:28:51.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:51 smithi098 ceph-mon[29192]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:28:51.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:51 smithi098 ceph-mon[29192]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T17:28:51.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:51 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:51.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:51 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:51.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:51 smithi098 ceph-mon[29192]: from='osd.5 ' entity='osd.5' 2024-04-25T17:28:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T17:28:51.577 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T17:28:51.577 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-25T17:28:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-25T17:28:51.578 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124945 s, 839 MB/s 2024-04-25T17:28:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-25T17:28:52.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:52 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:28:52.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:52 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:28:52.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:52 smithi098 ceph-mon[29192]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:52.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:52 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:52.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:52 smithi098 ceph-mon[29192]: osd.5 [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] boot 2024-04-25T17:28:52.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:52 smithi098 ceph-mon[29192]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T17:28:52.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:52 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:52.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-04-25T17:28:52.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:52 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:28:52.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:52 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:28:52.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:52 smithi203 ceph-mon[32834]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:52.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:52 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:52.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:52 smithi203 ceph-mon[32834]: osd.5 [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] boot 2024-04-25T17:28:52.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:52 smithi203 ceph-mon[32834]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T17:28:52.401 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:52 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:52 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:28:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:52 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:28:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:52 smithi173 ceph-mon[32856]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T17:28:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:52 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:52 smithi173 ceph-mon[32856]: osd.5 [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] boot 2024-04-25T17:28:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:52 smithi173 ceph-mon[32856]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T17:28:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:52 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:28:52.550 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:28:53.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T17:28:53.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:53.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:53.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:53.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:53.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:53.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:53.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:53.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:53 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T17:28:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:53.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:53.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:53.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:53.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:53.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:53.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:53.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:53 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T17:28:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:28:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:53.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:53.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:53.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:28:53.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:53.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:28:53.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:53 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:28:54.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.212+0000 7f5e24eb1700 1 -- 172.21.15.98:0/386272945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e20101130 msgr2=0x7f5e20101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:54.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.213+0000 7f5e24eb1700 1 --2- 172.21.15.98:0/386272945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e20101130 0x7f5e20101550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5e100099c0 tx=0x7f5e10009cd0 comp rx=0 tx=0).stop 2024-04-25T17:28:54.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.213+0000 7f5e24eb1700 1 -- 172.21.15.98:0/386272945 shutdown_connections 2024-04-25T17:28:54.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.213+0000 7f5e24eb1700 1 --2- 172.21.15.98:0/386272945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e20102cd0 0x7f5e20109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:54.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.213+0000 7f5e24eb1700 1 --2- 172.21.15.98:0/386272945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e20101130 0x7f5e20101550 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.213+0000 7f5e24eb1700 1 --2- 172.21.15.98:0/386272945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e20102330 0x7f5e20102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.213+0000 7f5e24eb1700 1 -- 172.21.15.98:0/386272945 >> 172.21.15.98:0/386272945 conn(0x7f5e200fc6b0 msgr2=0x7f5e200feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:28:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.213+0000 7f5e24eb1700 1 -- 172.21.15.98:0/386272945 shutdown_connections 2024-04-25T17:28:54.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.213+0000 7f5e24eb1700 1 -- 172.21.15.98:0/386272945 wait complete. 2024-04-25T17:28:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e24eb1700 1 Processor -- start 2024-04-25T17:28:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e24eb1700 1 -- start start 2024-04-25T17:28:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e24eb1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e20101130 0x7f5e20198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e24eb1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e20102330 0x7f5e201991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e24eb1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e20102cd0 0x7f5e201a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e24eb1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2010c3c0 con 0x7f5e20102cd0 2024-04-25T17:28:54.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e24eb1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2010c240 con 0x7f5e20102330 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e24eb1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2010c540 con 0x7f5e20101130 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e20102cd0 0x7f5e201a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e20102cd0 0x7f5e201a01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47852/0 (socket says 172.21.15.98:47852) 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1ed9d700 1 -- 172.21.15.98:0/1182702372 learned_addr learned my addr 172.21.15.98:0/1182702372 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1dd9b700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e20102330 0x7f5e201991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1e59c700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e20101130 0x7f5e20198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1ed9d700 1 -- 172.21.15.98:0/1182702372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e20101130 msgr2=0x7f5e20198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1ed9d700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e20101130 0x7f5e20198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1ed9d700 1 -- 172.21.15.98:0/1182702372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e20102330 msgr2=0x7f5e201991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1ed9d700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e20102330 0x7f5e201991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:28:54.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1ed9d700 1 -- 172.21.15.98:0/1182702372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e201a08c0 con 0x7f5e20102cd0 2024-04-25T17:28:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1dd9b700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e20102330 0x7f5e201991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:28:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e1ed9d700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e20102cd0 0x7f5e201a01f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800b970 tx=0x7f5e1800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:28:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e0f7fe700 1 -- 172.21.15.98:0/1182702372 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e18019070 con 0x7f5e20102cd0 2024-04-25T17:28:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e24eb1700 1 -- 172.21.15.98:0/1182702372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e201a0bb0 con 0x7f5e20102cd0 2024-04-25T17:28:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e0f7fe700 1 -- 172.21.15.98:0/1182702372 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e18014440 con 0x7f5e20102cd0 2024-04-25T17:28:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.214+0000 7f5e0f7fe700 1 -- 172.21.15.98:0/1182702372 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1800ce60 con 0x7f5e20102cd0 2024-04-25T17:28:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.215+0000 7f5e24eb1700 1 -- 172.21.15.98:0/1182702372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e201a10a0 con 0x7f5e20102cd0 2024-04-25T17:28:54.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.215+0000 7f5e24eb1700 1 -- 172.21.15.98:0/1182702372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e20069730 con 0x7f5e20102cd0 2024-04-25T17:28:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.217+0000 7f5e0f7fe700 1 -- 172.21.15.98:0/1182702372 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5e1801c070 con 0x7f5e20102cd0 2024-04-25T17:28:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.218+0000 7f5e0f7fe700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5e080a3c40 0x7f5e080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:28:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.218+0000 7f5e1e59c700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5e080a3c40 0x7f5e080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:28:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.218+0000 7f5e0f7fe700 1 -- 172.21.15.98:0/1182702372 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f5e180cee50 con 0x7f5e20102cd0 2024-04-25T17:28:54.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.218+0000 7f5e1e59c700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5e080a3c40 0x7f5e080a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009990 tx=0x7f5e10005ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:28:54.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.222+0000 7f5e0f7fe700 1 -- 172.21.15.98:0/1182702372 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e1809c6f0 con 0x7f5e20102cd0 2024-04-25T17:28:54.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:28:54.426+0000 7f5e24eb1700 1 -- 172.21.15.98:0/1182702372 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5e20108f60 con 0x7f5e080a3c40 2024-04-25T17:28:54.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:54 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi203 2024-04-25T17:28:54.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:54 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-25T17:28:54.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:54 smithi098 ceph-mon[29192]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:28:54.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:54 smithi098 ceph-mon[29192]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T17:28:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:54 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi203 2024-04-25T17:28:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:54 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-25T17:28:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:54 smithi203 ceph-mon[32834]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:28:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:54 smithi203 ceph-mon[32834]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T17:28:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:54 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi203 2024-04-25T17:28:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:54 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-25T17:28:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:54 smithi173 ceph-mon[32856]: pgmap v142: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:28:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:54 smithi173 ceph-mon[32856]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T17:28:55.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:55 smithi098 ceph-mon[29192]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T17:28:55.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:55 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:28:55.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:55 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:28:55.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:55 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:55 smithi203 ceph-mon[32834]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T17:28:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:55 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:28:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:55 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:28:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:55 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:55 smithi173 ceph-mon[32856]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T17:28:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:55 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:28:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:55 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:28:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:55 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:28:56.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:56 smithi098 ceph-mon[29192]: from='client.14343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:28:56.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:56 smithi098 ceph-mon[29192]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:28:56.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:56 smithi098 ceph-mon[29192]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T17:28:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:56 smithi203 ceph-mon[32834]: from='client.14343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:28:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:56 smithi203 ceph-mon[32834]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:28:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:56 smithi203 ceph-mon[32834]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T17:28:56.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:56 smithi173 ceph-mon[32856]: from='client.14343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:28:56.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:56 smithi173 ceph-mon[32856]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:28:56.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:56 smithi173 ceph-mon[32856]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T17:28:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:58 smithi098 ceph-mon[29192]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:28:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:58 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1366986787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73964ed9-657d-4097-b943-83b96cc1d04e"}]: dispatch 2024-04-25T17:28:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:58 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1366986787' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73964ed9-657d-4097-b943-83b96cc1d04e"}]': finished 2024-04-25T17:28:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:58 smithi098 ceph-mon[29192]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T17:28:58.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:58 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:28:58.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:28:58 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2032069045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:28:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:58 smithi203 ceph-mon[32834]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:28:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:58 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1366986787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73964ed9-657d-4097-b943-83b96cc1d04e"}]: dispatch 2024-04-25T17:28:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:58 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1366986787' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73964ed9-657d-4097-b943-83b96cc1d04e"}]': finished 2024-04-25T17:28:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:58 smithi203 ceph-mon[32834]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T17:28:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:58 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:28:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:28:58 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2032069045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:28:58.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:58 smithi173 ceph-mon[32856]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:28:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:58 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1366986787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73964ed9-657d-4097-b943-83b96cc1d04e"}]: dispatch 2024-04-25T17:28:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:58 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1366986787' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73964ed9-657d-4097-b943-83b96cc1d04e"}]': finished 2024-04-25T17:28:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:58 smithi173 ceph-mon[32856]: osdmap e44: 7 total, 6 up, 7 in 2024-04-25T17:28:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:58 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:28:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:28:58 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2032069045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:29:00.454 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:00 smithi203 ceph-mon[32834]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-25T17:29:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:00 smithi098 ceph-mon[29192]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-25T17:29:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:00 smithi173 ceph-mon[32856]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-25T17:29:02.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:02 smithi098 ceph-mon[29192]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-25T17:29:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:02 smithi203 ceph-mon[32834]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-25T17:29:02.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:02 smithi173 ceph-mon[32856]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-25T17:29:04.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:04 smithi098 ceph-mon[29192]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T17:29:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:04 smithi203 ceph-mon[32834]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T17:29:04.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:04 smithi173 ceph-mon[32856]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T17:29:05.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:05 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T17:29:05.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:05 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:05.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:05 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T17:29:05.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:05 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:05.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:05 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T17:29:05.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:05 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:06 smithi098 ceph-mon[29192]: Deploying daemon osd.6 on smithi098 2024-04-25T17:29:06.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:06 smithi098 ceph-mon[29192]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-25T17:29:06.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:06 smithi203 ceph-mon[32834]: Deploying daemon osd.6 on smithi098 2024-04-25T17:29:06.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:06 smithi203 ceph-mon[32834]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-25T17:29:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:06 smithi173 ceph-mon[32856]: Deploying daemon osd.6 on smithi098 2024-04-25T17:29:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:06 smithi173 ceph-mon[32856]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-25T17:29:07.354 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:07 smithi098 ceph-mon[29192]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T17:29:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:07 smithi203 ceph-mon[32834]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T17:29:07.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:07 smithi173 ceph-mon[32856]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T17:29:10.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:09 smithi098 ceph-mon[29192]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:10.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:09 smithi173 ceph-mon[32856]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:09 smithi203 ceph-mon[32834]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:11.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:11.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:11.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:11 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:11.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:11 smithi098 ceph-mon[29192]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:11.965 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:11.965 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:11.966 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:11 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:11.966 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:11 smithi203 ceph-mon[32834]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:11 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:12.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:11 smithi173 ceph-mon[32856]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:14.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:13 smithi098 ceph-mon[29192]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:14.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:14.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:14.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:14.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:14.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:14.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:13 smithi173 ceph-mon[32856]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:14.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:14.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:14.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:14.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:14.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:13 smithi203 ceph-mon[32834]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:14.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.636+0000 7f5e0f7fe700 1 -- 172.21.15.98:0/1182702372 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5e20108f60 con 0x7f5e080a3c40 2024-04-25T17:29:14.638 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 6 on host 'smithi098' 2024-04-25T17:29:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 -- 172.21.15.98:0/1182702372 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5e080a3c40 msgr2=0x7f5e080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:29:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5e080a3c40 0x7f5e080a6100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009990 tx=0x7f5e10005ad0 comp rx=0 tx=0).stop 2024-04-25T17:29:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 -- 172.21.15.98:0/1182702372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e20102cd0 msgr2=0x7f5e201a01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:29:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e20102cd0 0x7f5e201a01f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800b970 tx=0x7f5e1800bd30 comp rx=0 tx=0).stop 2024-04-25T17:29:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 -- 172.21.15.98:0/1182702372 shutdown_connections 2024-04-25T17:29:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5e080a3c40 0x7f5e080a6100 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e20101130 0x7f5e20198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e20102cd0 0x7f5e201a01f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 --2- 172.21.15.98:0/1182702372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e20102330 0x7f5e201991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 -- 172.21.15.98:0/1182702372 >> 172.21.15.98:0/1182702372 conn(0x7f5e200fc6b0 msgr2=0x7f5e20107840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:29:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 -- 172.21.15.98:0/1182702372 shutdown_connections 2024-04-25T17:29:14.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:29:14.639+0000 7f5e24eb1700 1 -- 172.21.15.98:0/1182702372 wait complete. 2024-04-25T17:29:15.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:14 smithi173 ceph-mon[32856]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T17:29:15.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:14 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:15.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:14 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:15.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:14 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:15.368 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:14 smithi098 ceph-mon[29192]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T17:29:15.369 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:14 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:15.369 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:14 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:15.369 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:14 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:15.400 DEBUG:teuthology.orchestra.run.smithi098:osd.6> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T17:29:15.402 INFO:tasks.cephadm:Deploying osd.7 on smithi173 with /dev/vg_nvme/lv_2... 2024-04-25T17:29:15.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T17:29:15.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:14 smithi203 ceph-mon[32834]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T17:29:15.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:14 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:15.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:14 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:15.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:14 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:15.649 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:29:16.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:16 smithi173 ceph-mon[32856]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:16.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:16 smithi173 ceph-mon[32856]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T17:29:16.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:16 smithi173 ceph-mon[32856]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T17:29:16.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:16 smithi173 ceph-mon[32856]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:29:16.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:16 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:16.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:16 smithi098 ceph-mon[29192]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:16.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:16 smithi098 ceph-mon[29192]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T17:29:16.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:16 smithi098 ceph-mon[29192]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T17:29:16.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:16 smithi098 ceph-mon[29192]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:29:16.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:16 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:16.277 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:29:16 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:29:16.002+0000 7f2c6a779700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T17:29:16.277 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:29:16 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:29:16.014+0000 7f2c6656c700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:29:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:16 smithi203 ceph-mon[32834]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:16 smithi203 ceph-mon[32834]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T17:29:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:16 smithi203 ceph-mon[32834]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T17:29:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:16 smithi203 ceph-mon[32834]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:29:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:16 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:17.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:17 smithi173 ceph-mon[32856]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:29:17.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:17 smithi173 ceph-mon[32856]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T17:29:17.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:17.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:17.210 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:17 smithi098 ceph-mon[29192]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:29:17.210 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:17 smithi098 ceph-mon[29192]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T17:29:17.210 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:17.210 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:17.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:17 smithi203 ceph-mon[32834]: from='osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:29:17.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:17 smithi203 ceph-mon[32834]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T17:29:17.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:17.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:17.756 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T17:29:17.756 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T17:29:17.757 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-25T17:29:17.757 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-25T17:29:17.757 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122522 s, 856 MB/s 2024-04-25T17:29:17.757 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-25T17:29:18.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:18 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:29:18.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:18 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:29:18.139 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:18 smithi173 ceph-mon[32856]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:18.140 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:18.140 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:18 smithi173 ceph-mon[32856]: osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] boot 2024-04-25T17:29:18.140 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:18 smithi173 ceph-mon[32856]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T17:29:18.140 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:18 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:18.364 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:18 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:29:18.365 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:18 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:29:18.365 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:18 smithi203 ceph-mon[32834]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:18.365 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:18.365 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:18 smithi203 ceph-mon[32834]: osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] boot 2024-04-25T17:29:18.365 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:18 smithi203 ceph-mon[32834]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T17:29:18.365 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:18 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:18 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:29:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:18 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:29:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:18 smithi098 ceph-mon[29192]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-25T17:29:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:18 smithi098 ceph-mon[29192]: osd.6 [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] boot 2024-04-25T17:29:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:18 smithi098 ceph-mon[29192]: osdmap e47: 7 total, 7 up, 7 in 2024-04-25T17:29:18.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:18 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:29:18.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-04-25T17:29:18.750 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:29:19.040 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:19 smithi173 ceph-mon[32856]: osdmap e48: 7 total, 7 up, 7 in 2024-04-25T17:29:19.040 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:19 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.040 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:19 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.040 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:19 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:19.040 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:19 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:19.040 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:19 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:19.040 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:19 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.040 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:19 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:19.040 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:19 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:19.040 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:19 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:19 smithi098 ceph-mon[29192]: osdmap e48: 7 total, 7 up, 7 in 2024-04-25T17:29:19.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:19 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:19 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:19 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:19.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:19 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:19.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:19 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:19.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:19 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:19 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:19.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:19 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:19.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:19 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:19 smithi203 ceph-mon[32834]: osdmap e48: 7 total, 7 up, 7 in 2024-04-25T17:29:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:19 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:19 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:19 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:19 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:19 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:19 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:19 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:19 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:19 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:20.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:20 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi098 2024-04-25T17:29:20.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:20 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi098 to 5696M 2024-04-25T17:29:20.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:20 smithi203 ceph-mon[32834]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:20.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:20 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi098 2024-04-25T17:29:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:20 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi098 to 5696M 2024-04-25T17:29:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:20 smithi173 ceph-mon[32856]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:20.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:20 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi098 2024-04-25T17:29:20.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:20 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi098 to 5696M 2024-04-25T17:29:20.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:20 smithi098 ceph-mon[29192]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:20.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.531+0000 7f11ae91d700 1 -- 172.21.15.173:0/4127854939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11a8102ba0 msgr2=0x7f11a8103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:29:20.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.531+0000 7f11ae91d700 1 --2- 172.21.15.173:0/4127854939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11a8102ba0 0x7f11a8103000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1198009920 tx=0x7f1198009c30 comp rx=0 tx=0).stop 2024-04-25T17:29:20.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.531+0000 7f11ae91d700 1 -- 172.21.15.173:0/4127854939 shutdown_connections 2024-04-25T17:29:20.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.531+0000 7f11ae91d700 1 --2- 172.21.15.173:0/4127854939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a8103540 0x7f11a8109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:20.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.531+0000 7f11ae91d700 1 --2- 172.21.15.173:0/4127854939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a81019a0 0x7f11a8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:20.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.531+0000 7f11ae91d700 1 --2- 172.21.15.173:0/4127854939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11a8102ba0 0x7f11a8103000 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:20.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.531+0000 7f11ae91d700 1 -- 172.21.15.173:0/4127854939 >> 172.21.15.173:0/4127854939 conn(0x7f11a80fcf40 msgr2=0x7f11a80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:29:20.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.531+0000 7f11ae91d700 1 -- 172.21.15.173:0/4127854939 shutdown_connections 2024-04-25T17:29:20.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.531+0000 7f11ae91d700 1 -- 172.21.15.173:0/4127854939 wait complete. 2024-04-25T17:29:20.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.532+0000 7f11ae91d700 1 Processor -- start 2024-04-25T17:29:20.533 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.532+0000 7f11ae91d700 1 -- start start 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.532+0000 7f11ae91d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a81019a0 0x7f11a806b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.532+0000 7f11ae91d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8102ba0 0x7f11a819fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.532+0000 7f11ae91d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11a8103540 0x7f11a81a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.532+0000 7f11ae91d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11a810d410 con 0x7f11a8102ba0 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.532+0000 7f11ae91d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11a810d290 con 0x7f11a8103540 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.532+0000 7f11ae91d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f11a810d590 con 0x7f11a81019a0 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.532+0000 7f11aceba700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11a8103540 0x7f11a81a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11aceba700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11a8103540 0x7f11a81a2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:50320/0 (socket says 172.21.15.173:50320) 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a7fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a81019a0 0x7f11a806b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11aceba700 1 -- 172.21.15.173:0/4011965414 learned_addr learned my addr 172.21.15.173:0/4011965414 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a77fe700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8102ba0 0x7f11a819fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a77fe700 1 -- 172.21.15.173:0/4011965414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a81019a0 msgr2=0x7f11a806b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a77fe700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a81019a0 0x7f11a806b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:20.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a77fe700 1 -- 172.21.15.173:0/4011965414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11a8103540 msgr2=0x7f11a81a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:29:20.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a77fe700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11a8103540 0x7f11a81a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:20.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a77fe700 1 -- 172.21.15.173:0/4011965414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a81a2b60 con 0x7f11a8102ba0 2024-04-25T17:29:20.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11aceba700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11a8103540 0x7f11a81a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:29:20.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a77fe700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8102ba0 0x7f11a819fe30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1198000c00 tx=0x7f1198005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:29:20.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a57fa700 1 -- 172.21.15.173:0/4011965414 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1198027070 con 0x7f11a8102ba0 2024-04-25T17:29:20.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11ae91d700 1 -- 172.21.15.173:0/4011965414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a81a2df0 con 0x7f11a8102ba0 2024-04-25T17:29:20.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a57fa700 1 -- 172.21.15.173:0/4011965414 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1198005510 con 0x7f11a8102ba0 2024-04-25T17:29:20.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11a57fa700 1 -- 172.21.15.173:0/4011965414 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11980049e0 con 0x7f11a8102ba0 2024-04-25T17:29:20.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.533+0000 7f11ae91d700 1 -- 172.21.15.173:0/4011965414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a81a31f0 con 0x7f11a8102ba0 2024-04-25T17:29:20.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.534+0000 7f11ae91d700 1 -- 172.21.15.173:0/4011965414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a8069730 con 0x7f11a8102ba0 2024-04-25T17:29:20.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.536+0000 7f11a57fa700 1 -- 172.21.15.173:0/4011965414 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f119802a070 con 0x7f11a8102ba0 2024-04-25T17:29:20.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.536+0000 7f11a57fa700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11940a3c40 0x7f11940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:29:20.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.536+0000 7f11a57fa700 1 -- 172.21.15.173:0/4011965414 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f11980d0370 con 0x7f11a8102ba0 2024-04-25T17:29:20.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.536+0000 7f11a7fff700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11940a3c40 0x7f11940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:29:20.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.537+0000 7f11a7fff700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11940a3c40 0x7f11940a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f11a8102a00 tx=0x7f1190009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:29:20.541 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.540+0000 7f11a57fa700 1 -- 172.21.15.173:0/4011965414 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f119805eff0 con 0x7f11a8102ba0 2024-04-25T17:29:20.746 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:20.746+0000 7f11ae91d700 1 -- 172.21.15.173:0/4011965414 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f11a8062e80 con 0x7f11940a3c40 2024-04-25T17:29:21.356 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:21 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:29:21.357 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:21 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:29:21.357 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:21 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:21.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:21 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:29:21.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:21 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:29:21.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:21 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:21.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:21 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:29:21.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:21 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:29:21.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:21 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:22.394 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:22 smithi173 ceph-mon[32856]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:29:22.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:22 smithi173 ceph-mon[32856]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:22.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:22 smithi098 ceph-mon[29192]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:29:22.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:22 smithi098 ceph-mon[29192]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:22.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:22 smithi203 ceph-mon[32834]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:29:22.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:22 smithi203 ceph-mon[32834]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:24.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:24 smithi173 ceph-mon[32856]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:24.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:24 smithi173 ceph-mon[32856]: from='client.? 172.21.15.173:0/1173720923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e248182-1de5-462a-a6ab-ec33e5ef7e7c"}]: dispatch 2024-04-25T17:29:24.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:24 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e248182-1de5-462a-a6ab-ec33e5ef7e7c"}]: dispatch 2024-04-25T17:29:24.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:24 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e248182-1de5-462a-a6ab-ec33e5ef7e7c"}]': finished 2024-04-25T17:29:24.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:24 smithi173 ceph-mon[32856]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T17:29:24.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:24.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:24 smithi098 ceph-mon[29192]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:24.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:24 smithi098 ceph-mon[29192]: from='client.? 172.21.15.173:0/1173720923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e248182-1de5-462a-a6ab-ec33e5ef7e7c"}]: dispatch 2024-04-25T17:29:24.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:24 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e248182-1de5-462a-a6ab-ec33e5ef7e7c"}]: dispatch 2024-04-25T17:29:24.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:24 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e248182-1de5-462a-a6ab-ec33e5ef7e7c"}]': finished 2024-04-25T17:29:24.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:24 smithi098 ceph-mon[29192]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T17:29:24.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:24 smithi203 ceph-mon[32834]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:24 smithi203 ceph-mon[32834]: from='client.? 172.21.15.173:0/1173720923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e248182-1de5-462a-a6ab-ec33e5ef7e7c"}]: dispatch 2024-04-25T17:29:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:24 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e248182-1de5-462a-a6ab-ec33e5ef7e7c"}]: dispatch 2024-04-25T17:29:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:24 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e248182-1de5-462a-a6ab-ec33e5ef7e7c"}]': finished 2024-04-25T17:29:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:24 smithi203 ceph-mon[32834]: osdmap e49: 8 total, 7 up, 8 in 2024-04-25T17:29:24.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:25.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:25 smithi098 ceph-mon[29192]: from='client.? 172.21.15.173:0/823495278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:29:25.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:25 smithi203 ceph-mon[32834]: from='client.? 172.21.15.173:0/823495278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:29:25.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:25 smithi173 ceph-mon[32856]: from='client.? 172.21.15.173:0/823495278' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:29:26.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:26 smithi098 ceph-mon[29192]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:26.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:26 smithi203 ceph-mon[32834]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:26.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:26 smithi173 ceph-mon[32856]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:28.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:28 smithi173 ceph-mon[32856]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:28.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:28 smithi098 ceph-mon[29192]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:28.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:28 smithi203 ceph-mon[32834]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:30.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:30 smithi173 ceph-mon[32856]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:30.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:30 smithi098 ceph-mon[29192]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:30.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:30 smithi203 ceph-mon[32834]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:31.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T17:29:31.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:31.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T17:29:31.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:31.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T17:29:31.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:32 smithi173 ceph-mon[32856]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:32 smithi173 ceph-mon[32856]: Deploying daemon osd.7 on smithi173 2024-04-25T17:29:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:32 smithi098 ceph-mon[29192]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:32 smithi098 ceph-mon[29192]: Deploying daemon osd.7 on smithi173 2024-04-25T17:29:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:32 smithi203 ceph-mon[32834]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:32 smithi203 ceph-mon[32834]: Deploying daemon osd.7 on smithi173 2024-04-25T17:29:34.160 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:34 smithi173 ceph-mon[32856]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:34.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:34 smithi098 ceph-mon[29192]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:34.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:34 smithi203 ceph-mon[32834]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:36.331 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:36 smithi173 ceph-mon[32856]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:36.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:36 smithi098 ceph-mon[29192]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:36 smithi203 ceph-mon[32834]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:38.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:38.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:38.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:38.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:37 smithi173 ceph-mon[32856]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:38.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:38.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:38.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:38.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:37 smithi098 ceph-mon[29192]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:38.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:38.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:38.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:38.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:37 smithi203 ceph-mon[32834]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:40.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:39 smithi173 ceph-mon[32856]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:40.314 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.313+0000 7f11a57fa700 1 -- 172.21.15.173:0/4011965414 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f11a8062e80 con 0x7f11940a3c40 2024-04-25T17:29:40.315 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 7 on host 'smithi173' 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 -- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11940a3c40 msgr2=0x7f11940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11940a3c40 0x7f11940a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f11a8102a00 tx=0x7f1190009310 comp rx=0 tx=0).stop 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 -- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8102ba0 msgr2=0x7f11a819fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8102ba0 0x7f11a819fe30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1198000c00 tx=0x7f1198005270 comp rx=0 tx=0).stop 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 -- 172.21.15.173:0/4011965414 shutdown_connections 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f11940a3c40 0x7f11940a6100 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11a81019a0 0x7f11a806b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11a8103540 0x7f11a81a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 --2- 172.21.15.173:0/4011965414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11a8102ba0 0x7f11a819fe30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 -- 172.21.15.173:0/4011965414 >> 172.21.15.173:0/4011965414 conn(0x7f11a80fcf40 msgr2=0x7f11a8106060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:29:40.317 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.315+0000 7f11ae91d700 1 -- 172.21.15.173:0/4011965414 shutdown_connections 2024-04-25T17:29:40.318 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:29:40.316+0000 7f11ae91d700 1 -- 172.21.15.173:0/4011965414 wait complete. 2024-04-25T17:29:40.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:39 smithi098 ceph-mon[29192]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:39 smithi203 ceph-mon[32834]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:41.050 DEBUG:teuthology.orchestra.run.smithi173:osd.7> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T17:29:41.052 INFO:tasks.cephadm:Deploying osd.8 on smithi203 with /dev/vg_nvme/lv_2... 2024-04-25T17:29:41.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T17:29:41.302 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:29:41.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:41 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:41 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:41 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:41 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.425 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:41 smithi203 ceph-mon[32834]: from='osd.7 [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T17:29:41.425 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:41 smithi203 ceph-mon[32834]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T17:29:41.425 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:41 smithi203 ceph-mon[32834]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:41 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:41 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:41 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:41 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:41 smithi173 ceph-mon[32856]: from='osd.7 [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T17:29:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:41 smithi173 ceph-mon[32856]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T17:29:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:41 smithi173 ceph-mon[32856]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:41 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:41 smithi098 ceph-mon[29192]: from='osd.7 [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T17:29:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:41 smithi098 ceph-mon[29192]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T17:29:41.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:41 smithi098 ceph-mon[29192]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:42.757 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:42 smithi203 ceph-mon[32834]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T17:29:42.758 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:42 smithi203 ceph-mon[32834]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T17:29:42.758 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:42.758 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:42 smithi203 ceph-mon[32834]: from='osd.7 [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:29:42.758 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:42 smithi203 ceph-mon[32834]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:29:42.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:42 smithi098 ceph-mon[29192]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T17:29:42.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:42 smithi098 ceph-mon[29192]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T17:29:42.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:42.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:42 smithi098 ceph-mon[29192]: from='osd.7 [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:29:42.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:42 smithi098 ceph-mon[29192]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:29:43.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:42 smithi173 ceph-mon[32856]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T17:29:43.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:42 smithi173 ceph-mon[32856]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T17:29:43.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:43.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:42 smithi173 ceph-mon[32856]: from='osd.7 [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:29:43.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:42 smithi173 ceph-mon[32856]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:29:43.424 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:29:43 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:29:43.077+0000 7f7fe8a06700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T17:29:43.424 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:29:43 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:29:43.089+0000 7f7fe07f1700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:29:43.476 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T17:29:43.476 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T17:29:43.476 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-25T17:29:43.476 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-25T17:29:43.476 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130176 s, 806 MB/s 2024-04-25T17:29:43.477 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-25T17:29:43.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:43 smithi173 ceph-mon[32856]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:29:43.759 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:43 smithi173 ceph-mon[32856]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T17:29:43.759 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:43.759 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:43.759 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:43 smithi173 ceph-mon[32856]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:43.759 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:43 smithi173 ceph-mon[32856]: from='osd.7 ' entity='osd.7' 2024-04-25T17:29:43.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:43 smithi098 ceph-mon[29192]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:29:43.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:43 smithi098 ceph-mon[29192]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T17:29:43.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:43.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:43.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:43 smithi098 ceph-mon[29192]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:43.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:43 smithi098 ceph-mon[29192]: from='osd.7 ' entity='osd.7' 2024-04-25T17:29:43.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:43 smithi203 ceph-mon[32834]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:29:43.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:43 smithi203 ceph-mon[32834]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T17:29:43.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:43.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:43.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:43 smithi203 ceph-mon[32834]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-25T17:29:43.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:43 smithi203 ceph-mon[32834]: from='osd.7 ' entity='osd.7' 2024-04-25T17:29:44.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_2 2024-04-25T17:29:44.386 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:29:44.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:29:44.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:29:44.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:44.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: osd.7 [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] boot 2024-04-25T17:29:44.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T17:29:44.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:44.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:44.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:44.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:44.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:44.800 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:44.800 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:44.800 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:44.800 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:44.800 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:44.800 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:44.800 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:44.800 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:44.800 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:44.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:29:44.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:29:44.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: osd.7 [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] boot 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:44.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:45.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:29:45.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:29:45.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:45.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: osd.7 [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] boot 2024-04-25T17:29:45.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: osdmap e52: 8 total, 8 up, 8 in 2024-04-25T17:29:45.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:29:45.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:45.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:45.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:45.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:45.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:29:45.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:45.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:45.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:45.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:45.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:29:45.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:45.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:29:45.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:29:45.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:45 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi173 2024-04-25T17:29:45.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:45 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-25T17:29:45.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:45 smithi203 ceph-mon[32834]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T17:29:45.872 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:45 smithi203 ceph-mon[32834]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:45.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:45 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi173 2024-04-25T17:29:45.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:45 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-25T17:29:45.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:45 smithi098 ceph-mon[29192]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T17:29:45.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:45 smithi098 ceph-mon[29192]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:46.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:45 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi173 2024-04-25T17:29:46.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:45 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-25T17:29:46.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:45 smithi173 ceph-mon[32856]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T17:29:46.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:45 smithi173 ceph-mon[32856]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:46.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.091+0000 7f2f204e1700 1 -- 172.21.15.203:0/1261780940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f18101c70 msgr2=0x7f2f181020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.091+0000 7f2f204e1700 1 --2- 172.21.15.203:0/1261780940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f18101c70 0x7f2f181020f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009990 tx=0x7f2f08009ca0 comp rx=0 tx=0).stop 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.091+0000 7f2f204e1700 1 -- 172.21.15.203:0/1261780940 shutdown_connections 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.091+0000 7f2f204e1700 1 --2- 172.21.15.203:0/1261780940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18102630 0x7f2f18102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.091+0000 7f2f204e1700 1 --2- 172.21.15.203:0/1261780940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f18101310 0x7f2f18101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.091+0000 7f2f204e1700 1 --2- 172.21.15.203:0/1261780940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f18101c70 0x7f2f181020f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.091+0000 7f2f204e1700 1 -- 172.21.15.203:0/1261780940 >> 172.21.15.203:0/1261780940 conn(0x7f2f180fcf30 msgr2=0x7f2f180ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.091+0000 7f2f204e1700 1 -- 172.21.15.203:0/1261780940 shutdown_connections 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.091+0000 7f2f204e1700 1 -- 172.21.15.203:0/1261780940 wait complete. 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f204e1700 1 Processor -- start 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f204e1700 1 -- start start 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f204e1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f18101310 0x7f2f1819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:29:46.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f204e1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f18101c70 0x7f2f1819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f204e1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18102630 0x7f2f181a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f204e1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2f18110710 con 0x7f2f18102630 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f204e1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2f18110590 con 0x7f2f18101c70 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f204e1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2f18110890 con 0x7f2f18101310 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f1e27d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f18101310 0x7f2f1819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f1e27d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f18101310 0x7f2f1819ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:32952/0 (socket says 172.21.15.203:32952) 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f1e27d700 1 -- 172.21.15.203:0/374993513 learned_addr learned my addr 172.21.15.203:0/374993513 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.092+0000 7f2f1ea7e700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18102630 0x7f2f181a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f1e27d700 1 -- 172.21.15.203:0/374993513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f18101c70 msgr2=0x7f2f1819b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f1da7c700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f18101c70 0x7f2f1819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f1e27d700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f18101c70 0x7f2f1819b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f1e27d700 1 -- 172.21.15.203:0/374993513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18102630 msgr2=0x7f2f181a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f1e27d700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18102630 0x7f2f181a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:29:46.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f1e27d700 1 -- 172.21.15.203:0/374993513 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f181a2a40 con 0x7f2f18101310 2024-04-25T17:29:46.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f1e27d700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f18101310 0x7f2f1819ade0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f14007f40 tx=0x7f2f1400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:29:46.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f0f7fe700 1 -- 172.21.15.203:0/374993513 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f14004bb0 con 0x7f2f18101310 2024-04-25T17:29:46.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f204e1700 1 -- 172.21.15.203:0/374993513 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f181a2d30 con 0x7f2f18101310 2024-04-25T17:29:46.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f0f7fe700 1 -- 172.21.15.203:0/374993513 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f14004d50 con 0x7f2f18101310 2024-04-25T17:29:46.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f0f7fe700 1 -- 172.21.15.203:0/374993513 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f14005a00 con 0x7f2f18101310 2024-04-25T17:29:46.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.093+0000 7f2f204e1700 1 -- 172.21.15.203:0/374993513 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f181a31c0 con 0x7f2f18101310 2024-04-25T17:29:46.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.094+0000 7f2f204e1700 1 -- 172.21.15.203:0/374993513 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f18069730 con 0x7f2f18101310 2024-04-25T17:29:46.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.095+0000 7f2f0f7fe700 1 -- 172.21.15.203:0/374993513 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2f1401a020 con 0x7f2f18101310 2024-04-25T17:29:46.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.096+0000 7f2f0f7fe700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2f040a3c40 0x7f2f040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:29:46.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.096+0000 7f2f0f7fe700 1 -- 172.21.15.203:0/374993513 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f2f140cea90 con 0x7f2f18101310 2024-04-25T17:29:46.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.096+0000 7f2f1da7c700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2f040a3c40 0x7f2f040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:29:46.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.096+0000 7f2f1da7c700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2f040a3c40 0x7f2f040a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2f08006010 tx=0x7f2f08003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:29:46.101 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.100+0000 7f2f0f7fe700 1 -- 172.21.15.203:0/374993513 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f14010070 con 0x7f2f18101310 2024-04-25T17:29:46.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:29:46.308+0000 7f2f204e1700 1 -- 172.21.15.203:0/374993513 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2f18062e80 con 0x7f2f040a3c40 2024-04-25T17:29:46.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:46 smithi203 ceph-mon[32834]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T17:29:46.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:46 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:29:46.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:46 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:29:46.924 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:46 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:46.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:46 smithi098 ceph-mon[29192]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T17:29:46.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:46 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:29:46.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:46 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:29:46.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:46 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:47.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:46 smithi173 ceph-mon[32856]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T17:29:47.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:46 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:29:47.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:46 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:29:47.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:46 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:47.950 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:47 smithi203 ceph-mon[32834]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:29:47.950 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:47 smithi203 ceph-mon[32834]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T17:29:47.950 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:47 smithi203 ceph-mon[32834]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:47.950 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:47 smithi203 ceph-mon[32834]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T17:29:47.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:47 smithi098 ceph-mon[29192]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:29:47.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:47 smithi098 ceph-mon[29192]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T17:29:47.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:47 smithi098 ceph-mon[29192]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:47.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:47 smithi098 ceph-mon[29192]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T17:29:48.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:47 smithi173 ceph-mon[32856]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:29:48.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:47 smithi173 ceph-mon[32856]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T17:29:48.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:47 smithi173 ceph-mon[32856]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:48.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:47 smithi173 ceph-mon[32856]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T17:29:50.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:49 smithi173 ceph-mon[32856]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:50.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:49 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/631257776' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c486a634-e021-4c49-974f-56246017c27a"}]: dispatch 2024-04-25T17:29:50.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:49 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c486a634-e021-4c49-974f-56246017c27a"}]: dispatch 2024-04-25T17:29:50.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:49 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c486a634-e021-4c49-974f-56246017c27a"}]': finished 2024-04-25T17:29:50.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:49 smithi173 ceph-mon[32856]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T17:29:50.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:49 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:29:50.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:49 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/2996415334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:29:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:49 smithi098 ceph-mon[29192]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:49 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/631257776' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c486a634-e021-4c49-974f-56246017c27a"}]: dispatch 2024-04-25T17:29:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:49 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c486a634-e021-4c49-974f-56246017c27a"}]: dispatch 2024-04-25T17:29:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:49 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c486a634-e021-4c49-974f-56246017c27a"}]': finished 2024-04-25T17:29:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:49 smithi098 ceph-mon[29192]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T17:29:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:49 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:29:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:49 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/2996415334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:29:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:49 smithi203 ceph-mon[32834]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:49 smithi203 ceph-mon[32834]: from='client.? 172.21.15.203:0/631257776' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c486a634-e021-4c49-974f-56246017c27a"}]: dispatch 2024-04-25T17:29:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:49 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c486a634-e021-4c49-974f-56246017c27a"}]: dispatch 2024-04-25T17:29:50.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:49 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c486a634-e021-4c49-974f-56246017c27a"}]': finished 2024-04-25T17:29:50.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:49 smithi203 ceph-mon[32834]: osdmap e57: 9 total, 8 up, 9 in 2024-04-25T17:29:50.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:49 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:29:50.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:49 smithi203 ceph-mon[32834]: from='client.? 172.21.15.203:0/2996415334' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:29:52.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:51 smithi173 ceph-mon[32856]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:52.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:51 smithi098 ceph-mon[29192]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:51 smithi203 ceph-mon[32834]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:54.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:53 smithi173 ceph-mon[32856]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:54.301 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:53 smithi203 ceph-mon[32834]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:53 smithi098 ceph-mon[29192]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:56.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:55 smithi203 ceph-mon[32834]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:56.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:55 smithi173 ceph-mon[32856]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:56.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:55 smithi098 ceph-mon[29192]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:57.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T17:29:57.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:57.286 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T17:29:57.286 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T17:29:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:29:58.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:58 smithi173 ceph-mon[32856]: Deploying daemon osd.8 on smithi203 2024-04-25T17:29:58.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:29:58 smithi173 ceph-mon[32856]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:58.325 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:58 smithi203 ceph-mon[32834]: Deploying daemon osd.8 on smithi203 2024-04-25T17:29:58.325 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:29:58 smithi203 ceph-mon[32834]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:29:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:58 smithi098 ceph-mon[29192]: Deploying daemon osd.8 on smithi203 2024-04-25T17:29:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:29:58 smithi098 ceph-mon[29192]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:00.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:00 smithi203 ceph-mon[32834]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:00.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:00 smithi203 ceph-mon[32834]: overall HEALTH_OK 2024-04-25T17:30:00.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:00 smithi173 ceph-mon[32856]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:00.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:00 smithi173 ceph-mon[32856]: overall HEALTH_OK 2024-04-25T17:30:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:00 smithi098 ceph-mon[29192]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:00 smithi098 ceph-mon[29192]: overall HEALTH_OK 2024-04-25T17:30:02.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:02 smithi173 ceph-mon[32856]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:02.305 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:02 smithi203 ceph-mon[32834]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:02.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:02 smithi098 ceph-mon[29192]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:03.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:03.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:03.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:03.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:03 smithi098 ceph-mon[29192]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:03.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:03.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:03.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:03.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:03 smithi203 ceph-mon[32834]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:03 smithi173 ceph-mon[32856]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:06.176 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:05 smithi203 ceph-mon[32834]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:06.176 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:05 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:06.176 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:05 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:06.177 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:05 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:06.177 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:05 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:06.177 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:05 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:06.227 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 8 on host 'smithi203' 2024-04-25T17:30:06.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.226+0000 7f2f0f7fe700 1 -- 172.21.15.203:0/374993513 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2f18062e80 con 0x7f2f040a3c40 2024-04-25T17:30:06.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.228+0000 7f2f204e1700 1 -- 172.21.15.203:0/374993513 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2f040a3c40 msgr2=0x7f2f040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.228+0000 7f2f204e1700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2f040a3c40 0x7f2f040a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2f08006010 tx=0x7f2f08003860 comp rx=0 tx=0).stop 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.228+0000 7f2f204e1700 1 -- 172.21.15.203:0/374993513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f18101310 msgr2=0x7f2f1819ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.228+0000 7f2f204e1700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f18101310 0x7f2f1819ade0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2f14007f40 tx=0x7f2f1400d3b0 comp rx=0 tx=0).stop 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.228+0000 7f2f204e1700 1 -- 172.21.15.203:0/374993513 shutdown_connections 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.228+0000 7f2f204e1700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2f040a3c40 0x7f2f040a6100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.228+0000 7f2f204e1700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f18101310 0x7f2f1819ade0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.228+0000 7f2f204e1700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f18102630 0x7f2f181a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.228+0000 7f2f204e1700 1 --2- 172.21.15.203:0/374993513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f18101c70 0x7f2f1819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.228+0000 7f2f204e1700 1 -- 172.21.15.203:0/374993513 >> 172.21.15.203:0/374993513 conn(0x7f2f180fcf30 msgr2=0x7f2f180fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.229+0000 7f2f204e1700 1 -- 172.21.15.203:0/374993513 shutdown_connections 2024-04-25T17:30:06.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:30:06.229+0000 7f2f204e1700 1 -- 172.21.15.203:0/374993513 wait complete. 2024-04-25T17:30:06.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:05 smithi173 ceph-mon[32856]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:06.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:05 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:06.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:05 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:06.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:05 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:06.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:05 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:06.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:05 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:05 smithi098 ceph-mon[29192]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:05 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:05 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:05 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:05 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:06.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:05 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:06.897 DEBUG:teuthology.orchestra.run.smithi203:osd.8> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T17:30:06.899 INFO:tasks.cephadm:Deploying osd.9 on smithi098 with /dev/vg_nvme/lv_1... 2024-04-25T17:30:06.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T17:30:07.150 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:30:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:07 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:07 smithi203 ceph-mon[32834]: from='osd.8 [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T17:30:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:07 smithi203 ceph-mon[32834]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T17:30:07.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:07 smithi203 ceph-mon[32834]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:07.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:07.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:07.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:07 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:07.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:07 smithi173 ceph-mon[32856]: from='osd.8 [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T17:30:07.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:07 smithi173 ceph-mon[32856]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T17:30:07.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:07 smithi173 ceph-mon[32856]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:07.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:07.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:07.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:07 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:07.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:07 smithi098 ceph-mon[29192]: from='osd.8 [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T17:30:07.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:07 smithi098 ceph-mon[29192]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T17:30:07.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:07 smithi098 ceph-mon[29192]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:08.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:08 smithi203 ceph-mon[32834]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T17:30:08.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:08 smithi203 ceph-mon[32834]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T17:30:08.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:08.310 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:08 smithi203 ceph-mon[32834]: from='osd.8 [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:30:08.310 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:08 smithi203 ceph-mon[32834]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:30:08.310 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:30:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:30:08.235+0000 7f38c0efc700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T17:30:08.310 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:30:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:30:08.245+0000 7f38bccef700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:30:08.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:08 smithi098 ceph-mon[29192]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T17:30:08.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:08 smithi098 ceph-mon[29192]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T17:30:08.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:08.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:08 smithi098 ceph-mon[29192]: from='osd.8 [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:30:08.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:08 smithi098 ceph-mon[29192]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:30:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:08 smithi173 ceph-mon[32856]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T17:30:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:08 smithi173 ceph-mon[32856]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T17:30:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:08 smithi173 ceph-mon[32856]: from='osd.8 [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:30:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:08 smithi173 ceph-mon[32856]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:30:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T17:30:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T17:30:09.332 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-25T17:30:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-25T17:30:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0162875 s, 644 MB/s 2024-04-25T17:30:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-25T17:30:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:09 smithi203 ceph-mon[32834]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:30:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:09 smithi203 ceph-mon[32834]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T17:30:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:09 smithi203 ceph-mon[32834]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:09 smithi203 ceph-mon[32834]: osd.8 [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] boot 2024-04-25T17:30:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:09 smithi203 ceph-mon[32834]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T17:30:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:09.478 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:09 smithi098 ceph-mon[29192]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:30:09.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:09 smithi098 ceph-mon[29192]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T17:30:09.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:09.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:09.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:09 smithi098 ceph-mon[29192]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:09.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:09 smithi098 ceph-mon[29192]: osd.8 [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] boot 2024-04-25T17:30:09.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:09 smithi098 ceph-mon[29192]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T17:30:09.479 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:09 smithi173 ceph-mon[32856]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:30:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:09 smithi173 ceph-mon[32856]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T17:30:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:09.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:09.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:09 smithi173 ceph-mon[32856]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-25T17:30:09.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:09 smithi173 ceph-mon[32856]: osd.8 [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] boot 2024-04-25T17:30:09.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:09 smithi173 ceph-mon[32856]: osdmap e60: 9 total, 9 up, 9 in 2024-04-25T17:30:09.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:30:10.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-04-25T17:30:10.334 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:30:10.448 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi203 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:10 smithi098 ceph-mon[29192]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T17:30:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:30:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:30:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi203 2024-04-25T17:30:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-25T17:30:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:10 smithi173 ceph-mon[32856]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T17:30:10.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:30:10.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:30:10.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi203 2024-04-25T17:30:10.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:10.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:10.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:10.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-25T17:30:10.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:10.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:10.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:10.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:10 smithi203 ceph-mon[32834]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T17:30:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 -- 172.21.15.98:0/337539779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8103550 msgr2=0x7f63c8109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 --2- 172.21.15.98:0/337539779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8103550 0x7f63c8109e00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f63c000b0d0 tx=0x7f63c000b3e0 comp rx=0 tx=0).stop 2024-04-25T17:30:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 -- 172.21.15.98:0/337539779 shutdown_connections 2024-04-25T17:30:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 --2- 172.21.15.98:0/337539779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c81019b0 0x7f63c8101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 --2- 172.21.15.98:0/337539779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8103550 0x7f63c8109e00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 --2- 172.21.15.98:0/337539779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63c8102bb0 0x7f63c8103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 -- 172.21.15.98:0/337539779 >> 172.21.15.98:0/337539779 conn(0x7f63c80fcf70 msgr2=0x7f63c80ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:30:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 -- 172.21.15.98:0/337539779 shutdown_connections 2024-04-25T17:30:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 -- 172.21.15.98:0/337539779 wait complete. 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 Processor -- start 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.113+0000 7f63d0095700 1 -- start start 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63d0095700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c81019b0 0x7f63c806bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63d0095700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8102bb0 0x7f63c819fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63d0095700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63c8103550 0x7f63c81a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63d0095700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63c810d450 con 0x7f63c8102bb0 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63d0095700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63c810d2d0 con 0x7f63c8103550 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63d0095700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f63c810d5d0 con 0x7f63c81019b0 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63cd630700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8102bb0 0x7f63c819fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63cd630700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8102bb0 0x7f63c819fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45034/0 (socket says 172.21.15.98:45034) 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63cd630700 1 -- 172.21.15.98:0/1475125209 learned_addr learned my addr 172.21.15.98:0/1475125209 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63cde31700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c81019b0 0x7f63c806bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63ce632700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63c8103550 0x7f63c81a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63cd630700 1 -- 172.21.15.98:0/1475125209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c81019b0 msgr2=0x7f63c806bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63cd630700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c81019b0 0x7f63c806bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63cd630700 1 -- 172.21.15.98:0/1475125209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63c8103550 msgr2=0x7f63c81a2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63cd630700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63c8103550 0x7f63c81a2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63cd630700 1 -- 172.21.15.98:0/1475125209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c81a2a60 con 0x7f63c8102bb0 2024-04-25T17:30:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63cd630700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8102bb0 0x7f63c819fd30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f63b800d950 tx=0x7f63b800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:30:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63beffd700 1 -- 172.21.15.98:0/1475125209 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8016070 con 0x7f63c8102bb0 2024-04-25T17:30:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63beffd700 1 -- 172.21.15.98:0/1475125209 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b800fd40 con 0x7f63c8102bb0 2024-04-25T17:30:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63d0095700 1 -- 172.21.15.98:0/1475125209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c81a2d50 con 0x7f63c8102bb0 2024-04-25T17:30:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63beffd700 1 -- 172.21.15.98:0/1475125209 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b800e3b0 con 0x7f63c8102bb0 2024-04-25T17:30:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.114+0000 7f63d0095700 1 -- 172.21.15.98:0/1475125209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c81a3230 con 0x7f63c8102bb0 2024-04-25T17:30:12.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.115+0000 7f63d0095700 1 -- 172.21.15.98:0/1475125209 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c8069730 con 0x7f63c8102bb0 2024-04-25T17:30:12.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.117+0000 7f63beffd700 1 -- 172.21.15.98:0/1475125209 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f63b8019070 con 0x7f63c8102bb0 2024-04-25T17:30:12.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.117+0000 7f63beffd700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f63b40a3c40 0x7f63b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:30:12.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.117+0000 7f63cde31700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f63b40a3c40 0x7f63b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:30:12.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.117+0000 7f63beffd700 1 -- 172.21.15.98:0/1475125209 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f63b80cf4c0 con 0x7f63c8102bb0 2024-04-25T17:30:12.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.118+0000 7f63cde31700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f63b40a3c40 0x7f63b40a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f63c8102a10 tx=0x7f63c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:30:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.121+0000 7f63beffd700 1 -- 172.21.15.98:0/1475125209 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63b809c850 con 0x7f63c8102bb0 2024-04-25T17:30:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:11 smithi098 ceph-mon[29192]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T17:30:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:11 smithi098 ceph-mon[29192]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:12.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:11 smithi203 ceph-mon[32834]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T17:30:12.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:11 smithi203 ceph-mon[32834]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:12.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:11 smithi173 ceph-mon[32856]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T17:30:12.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:11 smithi173 ceph-mon[32856]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:12.330+0000 7f63d0095700 1 -- 172.21.15.98:0/1475125209 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f63c8062e80 con 0x7f63b40a3c40 2024-04-25T17:30:13.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:12 smithi098 ceph-mon[29192]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T17:30:13.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:30:13.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:30:13.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:13.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:12 smithi203 ceph-mon[32834]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T17:30:13.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:30:13.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:30:13.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:13.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:12 smithi173 ceph-mon[32856]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T17:30:13.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:30:13.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:30:13.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:14.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:13 smithi098 ceph-mon[29192]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:30:14.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:13 smithi098 ceph-mon[29192]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T17:30:14.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:13 smithi098 ceph-mon[29192]: pgmap v206: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:14.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:13 smithi203 ceph-mon[32834]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:30:14.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:13 smithi203 ceph-mon[32834]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T17:30:14.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:13 smithi203 ceph-mon[32834]: pgmap v206: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:14.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:13 smithi173 ceph-mon[32856]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:30:14.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:13 smithi173 ceph-mon[32856]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T17:30:14.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:13 smithi173 ceph-mon[32856]: pgmap v206: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:16.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:15 smithi173 ceph-mon[32856]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-25T17:30:16.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:15 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2833971901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "650eada2-90dc-441b-a6e3-05fde6577647"}]: dispatch 2024-04-25T17:30:16.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:15 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2833971901' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "650eada2-90dc-441b-a6e3-05fde6577647"}]': finished 2024-04-25T17:30:16.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:15 smithi173 ceph-mon[32856]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T17:30:16.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:15 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:16.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:15 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1565942526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:30:16.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:15 smithi098 ceph-mon[29192]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-25T17:30:16.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:15 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2833971901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "650eada2-90dc-441b-a6e3-05fde6577647"}]: dispatch 2024-04-25T17:30:16.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:15 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2833971901' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "650eada2-90dc-441b-a6e3-05fde6577647"}]': finished 2024-04-25T17:30:16.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:15 smithi098 ceph-mon[29192]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T17:30:16.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:15 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:16.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:15 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1565942526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:30:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:15 smithi203 ceph-mon[32834]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-25T17:30:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:15 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2833971901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "650eada2-90dc-441b-a6e3-05fde6577647"}]: dispatch 2024-04-25T17:30:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:15 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2833971901' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "650eada2-90dc-441b-a6e3-05fde6577647"}]': finished 2024-04-25T17:30:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:15 smithi203 ceph-mon[32834]: osdmap e65: 10 total, 9 up, 10 in 2024-04-25T17:30:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:15 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:15 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1565942526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:30:18.266 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:17 smithi203 ceph-mon[32834]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T17:30:18.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:17 smithi173 ceph-mon[32856]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T17:30:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:17 smithi098 ceph-mon[29192]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T17:30:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:19 smithi173 ceph-mon[32856]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-25T17:30:20.390 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:19 smithi098 ceph-mon[29192]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-25T17:30:20.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:19 smithi203 ceph-mon[32834]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-25T17:30:22.198 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:22 smithi098 ceph-mon[29192]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T17:30:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:22 smithi173 ceph-mon[32856]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T17:30:22.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:22 smithi203 ceph-mon[32834]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T17:30:23.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T17:30:23.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:23.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T17:30:23.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:23.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T17:30:23.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:24.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:24 smithi098 ceph-mon[29192]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T17:30:24.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:24 smithi098 ceph-mon[29192]: Deploying daemon osd.9 on smithi098 2024-04-25T17:30:24.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:24 smithi173 ceph-mon[32856]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T17:30:24.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:24 smithi173 ceph-mon[32856]: Deploying daemon osd.9 on smithi098 2024-04-25T17:30:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:24 smithi203 ceph-mon[32834]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T17:30:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:24 smithi203 ceph-mon[32834]: Deploying daemon osd.9 on smithi098 2024-04-25T17:30:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:26 smithi173 ceph-mon[32856]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:26.278 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:26 smithi098 ceph-mon[29192]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:26.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:26 smithi203 ceph-mon[32834]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:28.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:28 smithi098 ceph-mon[29192]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:28.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:28 smithi173 ceph-mon[32856]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:28.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:28 smithi203 ceph-mon[32834]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:29 smithi098 ceph-mon[29192]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:30.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:30.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:30.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:30.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:29 smithi203 ceph-mon[32834]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:30.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:30.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:30.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:30.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:29 smithi173 ceph-mon[32856]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:31 smithi173 ceph-mon[32856]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:32.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:32.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:31 smithi098 ceph-mon[29192]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:32.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:32.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:32.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:32.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:32.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:31 smithi203 ceph-mon[32834]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:32.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:32.894 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 9 on host 'smithi098' 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.889+0000 7f63beffd700 1 -- 172.21.15.98:0/1475125209 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f63c8062e80 con 0x7f63b40a3c40 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 -- 172.21.15.98:0/1475125209 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f63b40a3c40 msgr2=0x7f63b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f63b40a3c40 0x7f63b40a6100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f63c8102a10 tx=0x7f63c400b410 comp rx=0 tx=0).stop 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 -- 172.21.15.98:0/1475125209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8102bb0 msgr2=0x7f63c819fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8102bb0 0x7f63c819fd30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f63b800d950 tx=0x7f63b800dc60 comp rx=0 tx=0).stop 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 -- 172.21.15.98:0/1475125209 shutdown_connections 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f63b40a3c40 0x7f63b40a6100 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63c81019b0 0x7f63c806bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63c8103550 0x7f63c81a2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 --2- 172.21.15.98:0/1475125209 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63c8102bb0 0x7f63c819fd30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:32.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 -- 172.21.15.98:0/1475125209 >> 172.21.15.98:0/1475125209 conn(0x7f63c80fcf70 msgr2=0x7f63c8105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:30:32.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 -- 172.21.15.98:0/1475125209 shutdown_connections 2024-04-25T17:30:32.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:30:32.892+0000 7f63d0095700 1 -- 172.21.15.98:0/1475125209 wait complete. 2024-04-25T17:30:33.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:32 smithi098 ceph-mon[29192]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T17:30:33.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:32 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:33.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:32 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:33.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:32 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:33.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:32 smithi173 ceph-mon[32856]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T17:30:33.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:32 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:33.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:32 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:33.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:32 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:32 smithi203 ceph-mon[32834]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T17:30:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:32 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:32 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:32 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:33.713 DEBUG:teuthology.orchestra.run.smithi098:osd.9> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T17:30:33.716 INFO:tasks.cephadm:Deploying osd.10 on smithi173 with /dev/vg_nvme/lv_1... 2024-04-25T17:30:33.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T17:30:33.965 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:30:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:34 smithi098 ceph-mon[29192]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:34 smithi098 ceph-mon[29192]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T17:30:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:34 smithi098 ceph-mon[29192]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T17:30:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:34 smithi098 ceph-mon[29192]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:30:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:34 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:34.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:34 smithi173 ceph-mon[32856]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:34.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:34 smithi173 ceph-mon[32856]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T17:30:34.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:34 smithi173 ceph-mon[32856]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T17:30:34.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:34 smithi173 ceph-mon[32856]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:30:34.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:34 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:34 smithi203 ceph-mon[32834]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:34 smithi203 ceph-mon[32834]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T17:30:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:34 smithi203 ceph-mon[32834]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T17:30:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:34 smithi203 ceph-mon[32834]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T17:30:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:34 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:34.653 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:30:34 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:30:34.469+0000 7f3389e66700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T17:30:34.653 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:30:34 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:30:34.480+0000 7f3381c51700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:30:35.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:35 smithi098 ceph-mon[29192]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:30:35.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:35 smithi098 ceph-mon[29192]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T17:30:35.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:35 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:35.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:35 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:35.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:35 smithi098 ceph-mon[29192]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' 2024-04-25T17:30:35.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:35 smithi173 ceph-mon[32856]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:30:35.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:35 smithi173 ceph-mon[32856]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T17:30:35.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:35 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:35.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:35 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:35.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:35 smithi173 ceph-mon[32856]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' 2024-04-25T17:30:35.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:35 smithi203 ceph-mon[32834]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T17:30:35.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:35 smithi203 ceph-mon[32834]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T17:30:35.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:35 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:35.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:35 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:35.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:35 smithi203 ceph-mon[32834]: from='osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246]' entity='osd.9' 2024-04-25T17:30:36.076 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T17:30:36.076 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T17:30:36.076 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-25T17:30:36.076 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-25T17:30:36.077 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124256 s, 844 MB/s 2024-04-25T17:30:36.077 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-25T17:30:36.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:36 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:30:36.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:36 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:30:36.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:36 smithi173 ceph-mon[32856]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:36.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:36 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:36.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:36 smithi173 ceph-mon[32856]: osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] boot 2024-04-25T17:30:36.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:36 smithi173 ceph-mon[32856]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T17:30:36.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:36 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:36.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:36 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:30:36.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:36 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:30:36.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:36 smithi098 ceph-mon[29192]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:36.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:36 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:36.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:36 smithi098 ceph-mon[29192]: osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] boot 2024-04-25T17:30:36.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:36 smithi098 ceph-mon[29192]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T17:30:36.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:36 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:36 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:30:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:36 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:30:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:36 smithi203 ceph-mon[32834]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-25T17:30:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:36 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:36 smithi203 ceph-mon[32834]: osd.9 [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] boot 2024-04-25T17:30:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:36 smithi203 ceph-mon[32834]: osdmap e68: 10 total, 10 up, 10 in 2024-04-25T17:30:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:36 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:30:36.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-04-25T17:30:37.122 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:30:37.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi098 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi098 to 4272M 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:37 smithi173 ceph-mon[32856]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:37.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi098 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi098 to 4272M 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:37 smithi098 ceph-mon[29192]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:37.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T17:30:37.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi098 2024-04-25T17:30:37.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:30:37.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi098 to 4272M 2024-04-25T17:30:37.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:37.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:37.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:37.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:37 smithi203 ceph-mon[32834]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:38.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.807+0000 7face00e3700 1 -- 172.21.15.173:0/896845618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7facd8104640 msgr2=0x7facd810aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:38.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.807+0000 7face00e3700 1 --2- 172.21.15.173:0/896845618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7facd8104640 0x7facd810aef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7facd000b0d0 tx=0x7facd000b3e0 comp rx=0 tx=0).stop 2024-04-25T17:30:38.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 -- 172.21.15.173:0/896845618 shutdown_connections 2024-04-25T17:30:38.811 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 --2- 172.21.15.173:0/896845618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facd8102aa0 0x7facd8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 --2- 172.21.15.173:0/896845618 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7facd8104640 0x7facd810aef0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 --2- 172.21.15.173:0/896845618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd8103ca0 0x7facd8104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 -- 172.21.15.173:0/896845618 >> 172.21.15.173:0/896845618 conn(0x7facd80fe040 msgr2=0x7facd8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 -- 172.21.15.173:0/896845618 shutdown_connections 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 -- 172.21.15.173:0/896845618 wait complete. 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 Processor -- start 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 -- start start 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facd8102aa0 0x7facd80767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7facd8103ca0 0x7facd8074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd8104640 0x7facd8075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7facd810e540 con 0x7facd8104640 2024-04-25T17:30:38.812 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7facd810e3c0 con 0x7facd8103ca0 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7face00e3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7facd810e6c0 con 0x7facd8102aa0 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.808+0000 7facdd67e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7facd8103ca0 0x7facd8074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facdd67e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7facd8103ca0 0x7facd8074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:38796/0 (socket says 172.21.15.173:38796) 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facdd67e700 1 -- 172.21.15.173:0/3370731529 learned_addr learned my addr 172.21.15.173:0/3370731529 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facde680700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd8104640 0x7facd8075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facdde7f700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facd8102aa0 0x7facd80767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facdd67e700 1 -- 172.21.15.173:0/3370731529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facd8102aa0 msgr2=0x7facd80767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facdd67e700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facd8102aa0 0x7facd80767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facdd67e700 1 -- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd8104640 msgr2=0x7facd8075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facdd67e700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd8104640 0x7facd8075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facdd67e700 1 -- 172.21.15.173:0/3370731529 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd8075c20 con 0x7facd8103ca0 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facdde7f700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facd8102aa0 0x7facd80767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:30:38.813 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facde680700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd8104640 0x7facd8075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:30:38.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7facdd67e700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7facd8103ca0 0x7facd8074e30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7facd400c9b0 tx=0x7facd400ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:30:38.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7faccaffd700 1 -- 172.21.15.173:0/3370731529 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd4007b00 con 0x7facd8103ca0 2024-04-25T17:30:38.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7face00e3700 1 -- 172.21.15.173:0/3370731529 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd81a2c90 con 0x7facd8103ca0 2024-04-25T17:30:38.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7faccaffd700 1 -- 172.21.15.173:0/3370731529 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facd4007ca0 con 0x7facd8103ca0 2024-04-25T17:30:38.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7faccaffd700 1 -- 172.21.15.173:0/3370731529 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd4018660 con 0x7facd8103ca0 2024-04-25T17:30:38.814 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.809+0000 7face00e3700 1 -- 172.21.15.173:0/3370731529 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd81a3180 con 0x7facd8103ca0 2024-04-25T17:30:38.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.810+0000 7face00e3700 1 -- 172.21.15.173:0/3370731529 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facd8069730 con 0x7facd8103ca0 2024-04-25T17:30:38.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.811+0000 7faccaffd700 1 -- 172.21.15.173:0/3370731529 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7facd400f450 con 0x7facd8103ca0 2024-04-25T17:30:38.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.812+0000 7faccaffd700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7facc40a3c40 0x7facc40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:30:38.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.812+0000 7faccaffd700 1 -- 172.21.15.173:0/3370731529 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7facd40cfc10 con 0x7facd8103ca0 2024-04-25T17:30:38.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.812+0000 7facdde7f700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7facc40a3c40 0x7facc40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:30:38.815 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.813+0000 7facdde7f700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7facc40a3c40 0x7facc40a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faccc005950 tx=0x7faccc005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:30:38.818 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:38.817+0000 7faccaffd700 1 -- 172.21.15.173:0/3370731529 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7facd4014030 con 0x7facd8103ca0 2024-04-25T17:30:39.023 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:39.021+0000 7face00e3700 1 -- 172.21.15.173:0/3370731529 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7facd8062e80 con 0x7facc40a3c40 2024-04-25T17:30:40.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:39 smithi173 ceph-mon[32856]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:40.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:39 smithi173 ceph-mon[32856]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:30:40.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:30:40.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:30:40.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:40.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:39 smithi098 ceph-mon[29192]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:40.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:39 smithi098 ceph-mon[29192]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:30:40.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:30:40.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:30:40.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:39 smithi203 ceph-mon[32834]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:39 smithi203 ceph-mon[32834]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:30:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:30:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:30:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:42.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:42 smithi173 ceph-mon[32856]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:42.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:42 smithi173 ceph-mon[32856]: from='client.? 172.21.15.173:0/357873579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc1b690a-95fa-4a33-a0d4-185446d78a09"}]: dispatch 2024-04-25T17:30:42.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:42 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc1b690a-95fa-4a33-a0d4-185446d78a09"}]: dispatch 2024-04-25T17:30:42.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:42 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc1b690a-95fa-4a33-a0d4-185446d78a09"}]': finished 2024-04-25T17:30:42.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:42 smithi173 ceph-mon[32856]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T17:30:42.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:30:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:42 smithi098 ceph-mon[29192]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:42 smithi098 ceph-mon[29192]: from='client.? 172.21.15.173:0/357873579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc1b690a-95fa-4a33-a0d4-185446d78a09"}]: dispatch 2024-04-25T17:30:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:42 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc1b690a-95fa-4a33-a0d4-185446d78a09"}]: dispatch 2024-04-25T17:30:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:42 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc1b690a-95fa-4a33-a0d4-185446d78a09"}]': finished 2024-04-25T17:30:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:42 smithi098 ceph-mon[29192]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T17:30:42.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:30:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:42 smithi203 ceph-mon[32834]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:42 smithi203 ceph-mon[32834]: from='client.? 172.21.15.173:0/357873579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc1b690a-95fa-4a33-a0d4-185446d78a09"}]: dispatch 2024-04-25T17:30:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:42 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc1b690a-95fa-4a33-a0d4-185446d78a09"}]: dispatch 2024-04-25T17:30:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:42 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc1b690a-95fa-4a33-a0d4-185446d78a09"}]': finished 2024-04-25T17:30:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:42 smithi203 ceph-mon[32834]: osdmap e70: 11 total, 10 up, 11 in 2024-04-25T17:30:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:30:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:43 smithi173 ceph-mon[32856]: from='client.? 172.21.15.173:0/361762186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:30:43.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:43 smithi098 ceph-mon[29192]: from='client.? 172.21.15.173:0/361762186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:30:43.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:43 smithi203 ceph-mon[32834]: from='client.? 172.21.15.173:0/361762186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:30:44.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:44 smithi173 ceph-mon[32856]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:44.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:44 smithi098 ceph-mon[29192]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:44.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:44 smithi203 ceph-mon[32834]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:46.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:46 smithi173 ceph-mon[32856]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:46.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:46 smithi098 ceph-mon[29192]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:46.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:46 smithi203 ceph-mon[32834]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:48.038 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:48 smithi173 ceph-mon[32856]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:48.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:48 smithi203 ceph-mon[32834]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:48.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:48 smithi098 ceph-mon[29192]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:50.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:50 smithi173 ceph-mon[32856]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:50.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T17:30:50.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:50 smithi098 ceph-mon[29192]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T17:30:50.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:50 smithi203 ceph-mon[32834]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T17:30:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:51.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:51 smithi173 ceph-mon[32856]: Deploying daemon osd.10 on smithi173 2024-04-25T17:30:51.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:51 smithi098 ceph-mon[29192]: Deploying daemon osd.10 on smithi173 2024-04-25T17:30:51.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:51 smithi203 ceph-mon[32834]: Deploying daemon osd.10 on smithi173 2024-04-25T17:30:52.342 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:52 smithi173 ceph-mon[32856]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:52.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:52 smithi098 ceph-mon[29192]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:52 smithi203 ceph-mon[32834]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:54.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:54 smithi173 ceph-mon[32856]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:54 smithi098 ceph-mon[29192]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:54 smithi203 ceph-mon[32834]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:56.177 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:56 smithi173 ceph-mon[32856]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:56.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:56 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:56.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:56 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:56.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:56 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:56.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:56 smithi098 ceph-mon[29192]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:56.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:56 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:56.652 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:56 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:56.652 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:56 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:56.653 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:56 smithi203 ceph-mon[32834]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:56.653 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:56 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:56.653 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:56 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:56.653 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:56 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:58.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:58 smithi173 ceph-mon[32856]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:58 smithi098 ceph-mon[29192]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:58 smithi203 ceph-mon[32834]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:59.288 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.287+0000 7faccaffd700 1 -- 172.21.15.173:0/3370731529 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7facd8062e80 con 0x7facc40a3c40 2024-04-25T17:30:59.288 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 10 on host 'smithi173' 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 -- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7facc40a3c40 msgr2=0x7facc40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7facc40a3c40 0x7facc40a6100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faccc005950 tx=0x7faccc005f50 comp rx=0 tx=0).stop 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 -- 172.21.15.173:0/3370731529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7facd8103ca0 msgr2=0x7facd8074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7facd8103ca0 0x7facd8074e30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7facd400c9b0 tx=0x7facd400ccc0 comp rx=0 tx=0).stop 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 -- 172.21.15.173:0/3370731529 shutdown_connections 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7facc40a3c40 0x7facc40a6100 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7facd8102aa0 0x7facd80767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7facd8104640 0x7facd8075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 --2- 172.21.15.173:0/3370731529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7facd8103ca0 0x7facd8074e30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 -- 172.21.15.173:0/3370731529 >> 172.21.15.173:0/3370731529 conn(0x7facd80fe040 msgr2=0x7facd8106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 -- 172.21.15.173:0/3370731529 shutdown_connections 2024-04-25T17:30:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T17:30:59.289+0000 7face00e3700 1 -- 172.21.15.173:0/3370731529 wait complete. 2024-04-25T17:30:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: from='osd.10 [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T17:30:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T17:30:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:30:59 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:59.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:59.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:59.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:59.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: from='osd.10 [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T17:30:59.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T17:30:59.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:30:59 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:30:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:30:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:30:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:30:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: from='osd.10 [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T17:30:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T17:30:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:30:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:30:59 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:00.075 DEBUG:teuthology.orchestra.run.smithi173:osd.10> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T17:31:00.078 INFO:tasks.cephadm:Deploying osd.11 on smithi203 with /dev/vg_nvme/lv_1... 2024-04-25T17:31:00.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T17:31:00.342 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:31:00.611 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:00 smithi203 ceph-mon[32834]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T17:31:00.611 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:00 smithi203 ceph-mon[32834]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T17:31:00.611 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:00.611 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:00 smithi203 ceph-mon[32834]: from='osd.10 [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:31:00.611 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:00 smithi203 ceph-mon[32834]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:31:00.642 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:00 smithi173 ceph-mon[32856]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T17:31:00.643 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:00 smithi173 ceph-mon[32856]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T17:31:00.643 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:00 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:00.643 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:00 smithi173 ceph-mon[32856]: from='osd.10 [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:31:00.643 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:00 smithi173 ceph-mon[32856]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:31:00.643 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:31:00 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:31:00.360+0000 7f2afaff7700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T17:31:00.643 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:31:00 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:31:00.372+0000 7f2af6dea700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:31:00.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:00 smithi098 ceph-mon[29192]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T17:31:00.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:00 smithi098 ceph-mon[29192]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T17:31:00.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:00.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:00 smithi098 ceph-mon[29192]: from='osd.10 [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:31:00.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:00 smithi098 ceph-mon[29192]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T17:31:01.621 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:01 smithi173 ceph-mon[32856]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:31:01.621 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:01 smithi173 ceph-mon[32856]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T17:31:01.622 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:01.622 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:01 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:01.622 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:01 smithi173 ceph-mon[32856]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:31:01.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:01 smithi098 ceph-mon[29192]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:31:01.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:01 smithi098 ceph-mon[29192]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T17:31:01.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:01.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:01 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:01.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:01 smithi098 ceph-mon[29192]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:31:01.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:01 smithi203 ceph-mon[32834]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T17:31:01.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:01 smithi203 ceph-mon[32834]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T17:31:01.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:01.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:01 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:01.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:01 smithi203 ceph-mon[32834]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-25T17:31:02.484 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T17:31:02.484 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T17:31:02.484 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-25T17:31:02.484 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-25T17:31:02.484 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126729 s, 827 MB/s 2024-04-25T17:31:02.484 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-25T17:31:02.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:02 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:31:02.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:02 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:31:02.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:02 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:02.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:02 smithi098 ceph-mon[29192]: osd.10 [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] boot 2024-04-25T17:31:02.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:02 smithi098 ceph-mon[29192]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T17:31:02.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:02 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:02.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:02 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:31:02.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:02 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:31:02.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:02 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:02.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:02 smithi203 ceph-mon[32834]: osd.10 [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] boot 2024-04-25T17:31:02.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:02 smithi203 ceph-mon[32834]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T17:31:02.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:02 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:02.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:02 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:31:02.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:02 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:31:02.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:02.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:02 smithi173 ceph-mon[32856]: osd.10 [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] boot 2024-04-25T17:31:02.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:02 smithi173 ceph-mon[32856]: osdmap e73: 11 total, 11 up, 11 in 2024-04-25T17:31:02.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:31:03.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_1 2024-04-25T17:31:03.550 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:31:03.672 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T17:31:03.672 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:03.672 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.672 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.672 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.672 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.673 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.673 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.673 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.673 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:03.673 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:03.673 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.673 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:03 smithi203 ceph-mon[32834]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T17:31:03.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T17:31:03.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:03.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:03.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:03.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.707 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:03 smithi098 ceph-mon[29192]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:03.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:03.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:03.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:03 smithi173 ceph-mon[32856]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T17:31:04.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:04 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi173 2024-04-25T17:31:04.657 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:04 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-25T17:31:04.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:04 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi173 2024-04-25T17:31:04.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:04 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-25T17:31:04.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:04 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi173 2024-04-25T17:31:04.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:04 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-25T17:31:05.232 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.231+0000 7ff0be80f700 1 -- 172.21.15.203:0/782120694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0b8103ca0 msgr2=0x7ff0b8104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:05.232 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.231+0000 7ff0be80f700 1 --2- 172.21.15.203:0/782120694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0b8103ca0 0x7ff0b8104100 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac009990 tx=0x7ff0ac009ca0 comp rx=0 tx=0).stop 2024-04-25T17:31:05.232 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.231+0000 7ff0be80f700 1 -- 172.21.15.203:0/782120694 shutdown_connections 2024-04-25T17:31:05.232 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.231+0000 7ff0be80f700 1 --2- 172.21.15.203:0/782120694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0b8104640 0x7ff0b810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:05.232 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.231+0000 7ff0be80f700 1 --2- 172.21.15.203:0/782120694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0b8102aa0 0x7ff0b8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:05.232 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.231+0000 7ff0be80f700 1 --2- 172.21.15.203:0/782120694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0b8103ca0 0x7ff0b8104100 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:05.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.231+0000 7ff0be80f700 1 -- 172.21.15.203:0/782120694 >> 172.21.15.203:0/782120694 conn(0x7ff0b80fe040 msgr2=0x7ff0b8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:05.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.231+0000 7ff0be80f700 1 -- 172.21.15.203:0/782120694 shutdown_connections 2024-04-25T17:31:05.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.231+0000 7ff0be80f700 1 -- 172.21.15.203:0/782120694 wait complete. 2024-04-25T17:31:05.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0be80f700 1 Processor -- start 2024-04-25T17:31:05.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0be80f700 1 -- start start 2024-04-25T17:31:05.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0be80f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0b8102aa0 0x7ff0b8198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:05.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0be80f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0b8103ca0 0x7ff0b8199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:05.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0be80f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0b8104640 0x7ff0b81a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:05.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0be80f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b810e530 con 0x7ff0b8102aa0 2024-04-25T17:31:05.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0be80f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b810e3b0 con 0x7ff0b8103ca0 2024-04-25T17:31:05.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0be80f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b810e6b0 con 0x7ff0b8104640 2024-04-25T17:31:05.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0bcdac700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0b8104640 0x7ff0b81a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:05.235 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0bcdac700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0b8104640 0x7ff0b81a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48310/0 (socket says 172.21.15.203:48310) 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0bcdac700 1 -- 172.21.15.203:0/1446576019 learned_addr learned my addr 172.21.15.203:0/1446576019 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0b7fff700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0b8102aa0 0x7ff0b8198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0b77fe700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0b8103ca0 0x7ff0b8199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0bcdac700 1 -- 172.21.15.203:0/1446576019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0b8103ca0 msgr2=0x7ff0b8199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0bcdac700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0b8103ca0 0x7ff0b8199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0bcdac700 1 -- 172.21.15.203:0/1446576019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0b8102aa0 msgr2=0x7ff0b8198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0bcdac700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0b8102aa0 0x7ff0b8198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.232+0000 7ff0bcdac700 1 -- 172.21.15.203:0/1446576019 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b81a0890 con 0x7ff0b8104640 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.233+0000 7ff0bcdac700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0b8104640 0x7ff0b81a01c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0007ee0 tx=0x7ff0b000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.233+0000 7ff0b57fa700 1 -- 172.21.15.203:0/1446576019 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b000dbd0 con 0x7ff0b8104640 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.233+0000 7ff0b57fa700 1 -- 172.21.15.203:0/1446576019 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0b000dd70 con 0x7ff0b8104640 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.233+0000 7ff0be80f700 1 -- 172.21.15.203:0/1446576019 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b81a0b80 con 0x7ff0b8104640 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.233+0000 7ff0b57fa700 1 -- 172.21.15.203:0/1446576019 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0005990 con 0x7ff0b8104640 2024-04-25T17:31:05.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.233+0000 7ff0be80f700 1 -- 172.21.15.203:0/1446576019 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b81a10c0 con 0x7ff0b8104640 2024-04-25T17:31:05.237 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.234+0000 7ff0be80f700 1 -- 172.21.15.203:0/1446576019 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0b8069730 con 0x7ff0b8104640 2024-04-25T17:31:05.237 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.235+0000 7ff0b57fa700 1 -- 172.21.15.203:0/1446576019 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff0b001a020 con 0x7ff0b8104640 2024-04-25T17:31:05.237 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.236+0000 7ff0b57fa700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff0a00a3c40 0x7ff0a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:05.237 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.236+0000 7ff0b57fa700 1 -- 172.21.15.203:0/1446576019 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7ff0b00cf540 con 0x7ff0b8104640 2024-04-25T17:31:05.237 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.236+0000 7ff0b7fff700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff0a00a3c40 0x7ff0a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:05.238 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.237+0000 7ff0b7fff700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff0a00a3c40 0x7ff0a00a6100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff0b819a120 tx=0x7ff0a800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:05.242 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.241+0000 7ff0b57fa700 1 -- 172.21.15.203:0/1446576019 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0b0010070 con 0x7ff0b8104640 2024-04-25T17:31:05.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:05.448+0000 7ff0be80f700 1 -- 172.21.15.203:0/1446576019 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff0b8062e80 con 0x7ff0a00a3c40 2024-04-25T17:31:05.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:05 smithi098 ceph-mon[29192]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T17:31:05.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:05 smithi098 ceph-mon[29192]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T17:31:05.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:05 smithi203 ceph-mon[32834]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T17:31:05.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:05 smithi203 ceph-mon[32834]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T17:31:05.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:05 smithi173 ceph-mon[32856]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T17:31:05.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:05 smithi173 ceph-mon[32856]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T17:31:06.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:06 smithi098 ceph-mon[29192]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T17:31:06.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:06 smithi098 ceph-mon[29192]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:31:06.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:06 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:31:06.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:06 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:31:06.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:06 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:06.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:06 smithi203 ceph-mon[32834]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T17:31:06.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:06 smithi203 ceph-mon[32834]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:31:06.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:06 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:31:06.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:06 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:31:06.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:06 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:06.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:06 smithi173 ceph-mon[32856]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T17:31:06.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:06 smithi173 ceph-mon[32856]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:31:06.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:06 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T17:31:06.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:06 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T17:31:06.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:06 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:07.665 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:07 smithi203 ceph-mon[32834]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T17:31:07.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:07 smithi098 ceph-mon[29192]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T17:31:07.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:07 smithi173 ceph-mon[32856]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T17:31:08.663 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:08 smithi203 ceph-mon[32834]: from='client.? 172.21.15.203:0/3824592965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f36bfb-b366-4c58-a814-c4ddda0f32f5"}]: dispatch 2024-04-25T17:31:08.663 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:08 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f36bfb-b366-4c58-a814-c4ddda0f32f5"}]: dispatch 2024-04-25T17:31:08.663 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:08 smithi203 ceph-mon[32834]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44f36bfb-b366-4c58-a814-c4ddda0f32f5"}]': finished 2024-04-25T17:31:08.663 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:08 smithi203 ceph-mon[32834]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T17:31:08.663 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:08.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:08 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/3824592965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f36bfb-b366-4c58-a814-c4ddda0f32f5"}]: dispatch 2024-04-25T17:31:08.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:08 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f36bfb-b366-4c58-a814-c4ddda0f32f5"}]: dispatch 2024-04-25T17:31:08.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:08 smithi098 ceph-mon[29192]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44f36bfb-b366-4c58-a814-c4ddda0f32f5"}]': finished 2024-04-25T17:31:08.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:08 smithi098 ceph-mon[29192]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T17:31:08.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:08.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:08 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/3824592965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f36bfb-b366-4c58-a814-c4ddda0f32f5"}]: dispatch 2024-04-25T17:31:08.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:08 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44f36bfb-b366-4c58-a814-c4ddda0f32f5"}]: dispatch 2024-04-25T17:31:08.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:08 smithi173 ceph-mon[32856]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44f36bfb-b366-4c58-a814-c4ddda0f32f5"}]': finished 2024-04-25T17:31:08.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:08 smithi173 ceph-mon[32856]: osdmap e78: 12 total, 11 up, 12 in 2024-04-25T17:31:08.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:09.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:09 smithi098 ceph-mon[29192]: from='client.? 172.21.15.203:0/1288639023' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:31:09.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:09 smithi098 ceph-mon[29192]: pgmap v248: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-25T17:31:09.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:09 smithi203 ceph-mon[32834]: from='client.? 172.21.15.203:0/1288639023' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:31:09.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:09 smithi203 ceph-mon[32834]: pgmap v248: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-25T17:31:09.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:09 smithi173 ceph-mon[32856]: from='client.? 172.21.15.203:0/1288639023' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T17:31:09.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:09 smithi173 ceph-mon[32856]: pgmap v248: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-25T17:31:12.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:12 smithi173 ceph-mon[32856]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-25T17:31:12.374 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:12 smithi203 ceph-mon[32834]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-25T17:31:12.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:12 smithi098 ceph-mon[29192]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-25T17:31:14.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:14 smithi173 ceph-mon[32856]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:14.422 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:14 smithi203 ceph-mon[32834]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:14.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:14 smithi098 ceph-mon[29192]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:16.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:16 smithi173 ceph-mon[32856]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:16.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:16 smithi098 ceph-mon[29192]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:16 smithi203 ceph-mon[32834]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:17.048 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T17:31:17.048 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:17.048 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:17 smithi203 ceph-mon[32834]: Deploying daemon osd.11 on smithi203 2024-04-25T17:31:17.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T17:31:17.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:17.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:17 smithi173 ceph-mon[32856]: Deploying daemon osd.11 on smithi203 2024-04-25T17:31:17.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T17:31:17.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:17.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:17 smithi098 ceph-mon[29192]: Deploying daemon osd.11 on smithi203 2024-04-25T17:31:18.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:18 smithi203 ceph-mon[32834]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:18.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:18 smithi173 ceph-mon[32856]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:18 smithi098 ceph-mon[29192]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:20.047 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:20 smithi203 ceph-mon[32834]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:20.347 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:20 smithi173 ceph-mon[32856]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:20.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:20 smithi098 ceph-mon[29192]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:22.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:22 smithi098 ceph-mon[29192]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:22.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:22 smithi203 ceph-mon[32834]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:22.525 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:22 smithi173 ceph-mon[32856]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:23.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:23.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:23.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:23 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:23.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:23 smithi098 ceph-mon[29192]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:23 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:23.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:23 smithi203 ceph-mon[32834]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:23.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:23.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:23.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:23 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:23.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:23 smithi173 ceph-mon[32856]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:26.131 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:26 smithi203 ceph-mon[32834]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:26.132 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:26 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:26.132 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:26 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:26.132 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:26 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:26.132 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:26 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:26.132 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:26 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:26.218 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.216+0000 7ff0b57fa700 1 -- 172.21.15.203:0/1446576019 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff0b8062e80 con 0x7ff0a00a3c40 2024-04-25T17:31:26.218 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 11 on host 'smithi203' 2024-04-25T17:31:26.220 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 -- 172.21.15.203:0/1446576019 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff0a00a3c40 msgr2=0x7ff0a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:26.220 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff0a00a3c40 0x7ff0a00a6100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff0b819a120 tx=0x7ff0a800b410 comp rx=0 tx=0).stop 2024-04-25T17:31:26.220 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 -- 172.21.15.203:0/1446576019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0b8104640 msgr2=0x7ff0b81a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:26.220 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0b8104640 0x7ff0b81a01c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0007ee0 tx=0x7ff0b000d3b0 comp rx=0 tx=0).stop 2024-04-25T17:31:26.220 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 -- 172.21.15.203:0/1446576019 shutdown_connections 2024-04-25T17:31:26.220 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff0a00a3c40 0x7ff0a00a6100 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:26.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0b8104640 0x7ff0b81a01c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:26.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0b8102aa0 0x7ff0b8198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:26.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 --2- 172.21.15.203:0/1446576019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0b8103ca0 0x7ff0b8199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:26.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 -- 172.21.15.203:0/1446576019 >> 172.21.15.203:0/1446576019 conn(0x7ff0b80fe040 msgr2=0x7ff0b8107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:26.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 -- 172.21.15.203:0/1446576019 shutdown_connections 2024-04-25T17:31:26.221 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:26.219+0000 7ff0be80f700 1 -- 172.21.15.203:0/1446576019 wait complete. 2024-04-25T17:31:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:26 smithi173 ceph-mon[32856]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:26 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:26 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:26 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:26 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:26.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:26 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:26.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:26 smithi098 ceph-mon[29192]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:26.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:26 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:26.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:26 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:26.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:26 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:26.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:26 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:26.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:26 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:26.997 DEBUG:teuthology.orchestra.run.smithi203:osd.11> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T17:31:26.999 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T17:31:26.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T17:31:27.243 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:31:27.275 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:27.275 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:27.275 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:27.276 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:27 smithi203 ceph-mon[32834]: from='osd.11 [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T17:31:27.276 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:27 smithi203 ceph-mon[32834]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T17:31:27.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:27.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:27.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:27.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:27 smithi098 ceph-mon[29192]: from='osd.11 [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T17:31:27.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:27 smithi098 ceph-mon[29192]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T17:31:27.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:27.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:27.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:27.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:27 smithi173 ceph-mon[32856]: from='osd.11 [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T17:31:27.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:27 smithi173 ceph-mon[32856]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T17:31:28.396 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:28 smithi203 ceph-mon[32834]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:28.397 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:28 smithi203 ceph-mon[32834]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T17:31:28.397 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:28 smithi203 ceph-mon[32834]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T17:31:28.397 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:28 smithi203 ceph-mon[32834]: from='osd.11 [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:31:28.397 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:28 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:28.397 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:28 smithi203 ceph-mon[32834]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:31:28.397 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:31:28 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:31:28.179+0000 7fee05a7d700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T17:31:28.397 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:31:28 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:31:28.195+0000 7fee01870700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T17:31:28.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:28 smithi098 ceph-mon[29192]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:28.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:28 smithi098 ceph-mon[29192]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T17:31:28.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:28 smithi098 ceph-mon[29192]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T17:31:28.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:28 smithi098 ceph-mon[29192]: from='osd.11 [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:31:28.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:28 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:28.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:28 smithi098 ceph-mon[29192]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:31:28.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:28 smithi173 ceph-mon[32856]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:28.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:28 smithi173 ceph-mon[32856]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T17:31:28.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:28 smithi173 ceph-mon[32856]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T17:31:28.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:28 smithi173 ceph-mon[32856]: from='osd.11 [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:31:28.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:28 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:28.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:28 smithi173 ceph-mon[32856]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T17:31:29.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.011+0000 7f72db07a700 1 -- 172.21.15.98:0/3790429614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d4101990 msgr2=0x7f72d4101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:29.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 --2- 172.21.15.98:0/3790429614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d4101990 0x7f72d4101db0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f72c80099c0 tx=0x7f72c8009cd0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 -- 172.21.15.98:0/3790429614 shutdown_connections 2024-04-25T17:31:29.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 --2- 172.21.15.98:0/3790429614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d4103530 0x7f72d4109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 --2- 172.21.15.98:0/3790429614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d4101990 0x7f72d4101db0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 --2- 172.21.15.98:0/3790429614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d4102b90 0x7f72d4102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 -- 172.21.15.98:0/3790429614 >> 172.21.15.98:0/3790429614 conn(0x7f72d40fcf30 msgr2=0x7f72d40ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 -- 172.21.15.98:0/3790429614 shutdown_connections 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 -- 172.21.15.98:0/3790429614 wait complete. 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 Processor -- start 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 -- start start 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d4101990 0x7f72d4197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d4102b90 0x7f72d4197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d4103530 0x7f72d419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f72d410d420 con 0x7f72d4103530 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f72d410d2a0 con 0x7f72d4102b90 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72db07a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72d410d5a0 con 0x7f72d4101990 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72d9617700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d4103530 0x7f72d419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72d8e16700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d4101990 0x7f72d4197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:29.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72d9617700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d4103530 0x7f72d419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53262/0 (socket says 172.21.15.98:53262) 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.012+0000 7f72d9617700 1 -- 172.21.15.98:0/1963052156 learned_addr learned my addr 172.21.15.98:0/1963052156 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d3fff700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d4102b90 0x7f72d4197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d9617700 1 -- 172.21.15.98:0/1963052156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d4101990 msgr2=0x7f72d4197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d9617700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d4101990 0x7f72d4197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d9617700 1 -- 172.21.15.98:0/1963052156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d4102b90 msgr2=0x7f72d4197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d9617700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d4102b90 0x7f72d4197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d9617700 1 -- 172.21.15.98:0/1963052156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d419f6a0 con 0x7f72d4103530 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d8e16700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d4101990 0x7f72d4197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d9617700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d4103530 0x7f72d419f000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f72cc007f10 tx=0x7f72cc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d1ffb700 1 -- 172.21.15.98:0/1963052156 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc016030 con 0x7f72d4103530 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d1ffb700 1 -- 172.21.15.98:0/1963052156 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72cc004bb0 con 0x7f72d4103530 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72d1ffb700 1 -- 172.21.15.98:0/1963052156 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc012c90 con 0x7f72d4103530 2024-04-25T17:31:29.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72db07a700 1 -- 172.21.15.98:0/1963052156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d406b230 con 0x7f72d4103530 2024-04-25T17:31:29.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.013+0000 7f72db07a700 1 -- 172.21.15.98:0/1963052156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d406b770 con 0x7f72d4103530 2024-04-25T17:31:29.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.014+0000 7f72db07a700 1 -- 172.21.15.98:0/1963052156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d4069730 con 0x7f72d4103530 2024-04-25T17:31:29.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.015+0000 7f72d1ffb700 1 -- 172.21.15.98:0/1963052156 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f72cc012460 con 0x7f72d4103530 2024-04-25T17:31:29.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.016+0000 7f72d1ffb700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f72c40a3cc0 0x7f72c40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:29.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.016+0000 7f72d8e16700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f72c40a3cc0 0x7f72c40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:29.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.016+0000 7f72d1ffb700 1 -- 172.21.15.98:0/1963052156 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7f72cc0d1e00 con 0x7f72d4103530 2024-04-25T17:31:29.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.017+0000 7f72d8e16700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f72c40a3cc0 0x7f72c40a6180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f72c80038c0 tx=0x7f72c800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:29.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.020+0000 7f72d1ffb700 1 -- 172.21.15.98:0/1963052156 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72cc09ed00 con 0x7f72d4103530 2024-04-25T17:31:29.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.225+0000 7f72db07a700 1 -- 172.21.15.98:0/1963052156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f72d41087b0 con 0x7f72d4103530 2024-04-25T17:31:29.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.225+0000 7f72d1ffb700 1 -- 172.21.15.98:0/1963052156 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f72d41087b0 con 0x7f72d4103530 2024-04-25T17:31:29.227 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:31:29.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 -- 172.21.15.98:0/1963052156 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f72c40a3cc0 msgr2=0x7f72c40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f72c40a3cc0 0x7f72c40a6180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f72c80038c0 tx=0x7f72c800b540 comp rx=0 tx=0).stop 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 -- 172.21.15.98:0/1963052156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d4103530 msgr2=0x7f72d419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d4103530 0x7f72d419f000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f72cc007f10 tx=0x7f72cc00d3b0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 -- 172.21.15.98:0/1963052156 shutdown_connections 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f72c40a3cc0 0x7f72c40a6180 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d4101990 0x7f72d4197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d4103530 0x7f72d419f000 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 --2- 172.21.15.98:0/1963052156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d4102b90 0x7f72d4197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 -- 172.21.15.98:0/1963052156 >> 172.21.15.98:0/1963052156 conn(0x7f72d40fcf30 msgr2=0x7f72d4105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 -- 172.21.15.98:0/1963052156 shutdown_connections 2024-04-25T17:31:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:29.227+0000 7f72db07a700 1 -- 172.21.15.98:0/1963052156 wait complete. 2024-04-25T17:31:29.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:29 smithi098 ceph-mon[29192]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:31:29.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:29 smithi098 ceph-mon[29192]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T17:31:29.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:29.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:29 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:29.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:29 smithi098 ceph-mon[29192]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:29.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:29 smithi203 ceph-mon[32834]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:31:29.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:29 smithi203 ceph-mon[32834]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T17:31:29.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:29.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:29 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:29.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:29 smithi203 ceph-mon[32834]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:29.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:29 smithi173 ceph-mon[32856]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T17:31:29.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:29 smithi173 ceph-mon[32856]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T17:31:29.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:29.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:29 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:29.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:29 smithi173 ceph-mon[32856]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-25T17:31:29.886 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1714066289,"num_in_osds":12,"osd_in_since":1714066268,"num_remapped_pgs":0} 2024-04-25T17:31:29.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T17:31:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:31:30.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:30 smithi098 ceph-mon[29192]: purged_snaps scrub starts 2024-04-25T17:31:30.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:30 smithi098 ceph-mon[29192]: purged_snaps scrub ok 2024-04-25T17:31:30.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:30 smithi098 ceph-mon[29192]: osd.11 [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] boot 2024-04-25T17:31:30.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:30 smithi098 ceph-mon[29192]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T17:31:30.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:30 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:30.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:30 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1963052156' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T17:31:30.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:30 smithi203 ceph-mon[32834]: purged_snaps scrub starts 2024-04-25T17:31:30.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:30 smithi203 ceph-mon[32834]: purged_snaps scrub ok 2024-04-25T17:31:30.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:30 smithi203 ceph-mon[32834]: osd.11 [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] boot 2024-04-25T17:31:30.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:30 smithi203 ceph-mon[32834]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T17:31:30.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:30 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:30.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:30 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1963052156' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T17:31:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:30 smithi173 ceph-mon[32856]: purged_snaps scrub starts 2024-04-25T17:31:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:30 smithi173 ceph-mon[32856]: purged_snaps scrub ok 2024-04-25T17:31:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:30 smithi173 ceph-mon[32856]: osd.11 [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] boot 2024-04-25T17:31:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:30 smithi173 ceph-mon[32856]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T17:31:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:30 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:31:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:30 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1963052156' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T17:31:31.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T17:31:31.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: Detected new or changed devices on smithi203 2024-04-25T17:31:31.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-25T17:31:31.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:31.412 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:31.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:31 smithi098 ceph-mon[29192]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:31.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T17:31:31.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: Detected new or changed devices on smithi203 2024-04-25T17:31:31.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-25T17:31:31.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:31.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:31.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:31 smithi203 ceph-mon[32834]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:31.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T17:31:31.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: Detected new or changed devices on smithi203 2024-04-25T17:31:31.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:31:31.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-25T17:31:31.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:31.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:31.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:31.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:31 smithi173 ceph-mon[32856]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.947+0000 7fd862808700 1 -- 172.21.15.98:0/3978102749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c108a30 msgr2=0x7fd85c108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.947+0000 7fd862808700 1 --2- 172.21.15.98:0/3978102749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c108a30 0x7fd85c108e10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009990 tx=0x7fd84c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.948+0000 7fd862808700 1 -- 172.21.15.98:0/3978102749 shutdown_connections 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.948+0000 7fd862808700 1 --2- 172.21.15.98:0/3978102749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd85c06bb30 0x7fd85c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.948+0000 7fd862808700 1 --2- 172.21.15.98:0/3978102749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c108a30 0x7fd85c108e10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.948+0000 7fd862808700 1 --2- 172.21.15.98:0/3978102749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd85c06b190 0x7fd85c06b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.948+0000 7fd862808700 1 -- 172.21.15.98:0/3978102749 >> 172.21.15.98:0/3978102749 conn(0x7fd85c078910 msgr2=0x7fd85c078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.948+0000 7fd862808700 1 -- 172.21.15.98:0/3978102749 shutdown_connections 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.948+0000 7fd862808700 1 -- 172.21.15.98:0/3978102749 wait complete. 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.948+0000 7fd862808700 1 Processor -- start 2024-04-25T17:31:31.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.948+0000 7fd862808700 1 -- start start 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd862808700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c06b190 0x7fd85c101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd862808700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd85c06bb30 0x7fd85c101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd862808700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd85c108a30 0x7fd85c1024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd862808700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd85c112bd0 con 0x7fd85c06b190 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd862808700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd85c112a50 con 0x7fd85c06bb30 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd862808700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd85c112d50 con 0x7fd85c108a30 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c06b190 0x7fd85c101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c06b190 0x7fd85c101a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53278/0 (socket says 172.21.15.98:53278) 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85bfff700 1 -- 172.21.15.98:0/2252737327 learned_addr learned my addr 172.21.15.98:0/2252737327 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85b7fe700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd85c06bb30 0x7fd85c101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85bfff700 1 -- 172.21.15.98:0/2252737327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd85c108a30 msgr2=0x7fd85c1024c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85bfff700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd85c108a30 0x7fd85c1024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85bfff700 1 -- 172.21.15.98:0/2252737327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd85c06bb30 msgr2=0x7fd85c101f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:31.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85bfff700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd85c06bb30 0x7fd85c101f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:31.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85bfff700 1 -- 172.21.15.98:0/2252737327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85c1a7650 con 0x7fd85c06b190 2024-04-25T17:31:31.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85b7fe700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd85c06bb30 0x7fd85c101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:31:31.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd85bfff700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c06b190 0x7fd85c101a40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd84c000c00 tx=0x7fd84c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:31.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd8597fa700 1 -- 172.21.15.98:0/2252737327 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c027070 con 0x7fd85c06b190 2024-04-25T17:31:31.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd8597fa700 1 -- 172.21.15.98:0/2252737327 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd84c003fa0 con 0x7fd85c06b190 2024-04-25T17:31:31.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd862808700 1 -- 172.21.15.98:0/2252737327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd85c1a78e0 con 0x7fd85c06b190 2024-04-25T17:31:31.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd8597fa700 1 -- 172.21.15.98:0/2252737327 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c022db0 con 0x7fd85c06b190 2024-04-25T17:31:31.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.949+0000 7fd862808700 1 -- 172.21.15.98:0/2252737327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd85c1a7ce0 con 0x7fd85c06b190 2024-04-25T17:31:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.950+0000 7fd862808700 1 -- 172.21.15.98:0/2252737327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd85c069730 con 0x7fd85c06b190 2024-04-25T17:31:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.951+0000 7fd8597fa700 1 -- 172.21.15.98:0/2252737327 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd84c02a070 con 0x7fd85c06b190 2024-04-25T17:31:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.952+0000 7fd8597fa700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd8440a3c40 0x7fd8440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.952+0000 7fd85b7fe700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd8440a3c40 0x7fd8440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.952+0000 7fd8597fa700 1 -- 172.21.15.98:0/2252737327 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd84c0d1110 con 0x7fd85c06b190 2024-04-25T17:31:31.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.953+0000 7fd85b7fe700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd8440a3c40 0x7fd8440a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd850005fd0 tx=0x7fd850005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:31.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:31.957+0000 7fd8597fa700 1 -- 172.21.15.98:0/2252737327 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd84c013070 con 0x7fd85c06b190 2024-04-25T17:31:32.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.154+0000 7fd862808700 1 -- 172.21.15.98:0/2252737327 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd85c02e1e0 con 0x7fd85c06b190 2024-04-25T17:31:32.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.155+0000 7fd8597fa700 1 -- 172.21.15.98:0/2252737327 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18492 (secure 0 0 0) 0x7fd84c09df80 con 0x7fd85c06b190 2024-04-25T17:31:32.157 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:31:32.157 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":82,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:31:30.178812+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:31:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.157+0000 7fd862808700 1 -- 172.21.15.98:0/2252737327 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd8440a3c40 msgr2=0x7fd8440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.157+0000 7fd862808700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd8440a3c40 0x7fd8440a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd850005fd0 tx=0x7fd850005dc0 comp rx=0 tx=0).stop 2024-04-25T17:31:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.157+0000 7fd862808700 1 -- 172.21.15.98:0/2252737327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c06b190 msgr2=0x7fd85c101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.157+0000 7fd862808700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c06b190 0x7fd85c101a40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd84c000c00 tx=0x7fd84c0047b0 comp rx=0 tx=0).stop 2024-04-25T17:31:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.158+0000 7fd862808700 1 -- 172.21.15.98:0/2252737327 shutdown_connections 2024-04-25T17:31:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.158+0000 7fd862808700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd8440a3c40 0x7fd8440a6100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.158+0000 7fd862808700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd85c108a30 0x7fd85c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.158+0000 7fd862808700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd85c06b190 0x7fd85c101a40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.158+0000 7fd862808700 1 --2- 172.21.15.98:0/2252737327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd85c06bb30 0x7fd85c101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:32.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.158+0000 7fd862808700 1 -- 172.21.15.98:0/2252737327 >> 172.21.15.98:0/2252737327 conn(0x7fd85c078910 msgr2=0x7fd85c0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:32.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.158+0000 7fd862808700 1 -- 172.21.15.98:0/2252737327 shutdown_connections 2024-04-25T17:31:32.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:32.158+0000 7fd862808700 1 -- 172.21.15.98:0/2252737327 wait complete. 2024-04-25T17:31:32.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:32 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2252737327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:31:32.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:32 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2252737327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:31:32.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:32 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2252737327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:31:32.924 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T17:27:36.969831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T17:31:32.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T17:31:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:31:33.630 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:33 smithi098 ceph-mon[29192]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:33.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:33 smithi203 ceph-mon[32834]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:33.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:33 smithi173 ceph-mon[32856]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.901+0000 7f79fbe31700 1 -- 172.21.15.98:0/2728704681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4103450 msgr2=0x7f79f410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.901+0000 7f79fbe31700 1 --2- 172.21.15.98:0/2728704681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4103450 0x7f79f410f8b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00b0a0 tx=0x7f79ec00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:31:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.901+0000 7f79fbe31700 1 -- 172.21.15.98:0/2728704681 shutdown_connections 2024-04-25T17:31:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.901+0000 7f79fbe31700 1 --2- 172.21.15.98:0/2728704681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4103450 0x7f79f410f8b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.901+0000 7f79fbe31700 1 --2- 172.21.15.98:0/2728704681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79f4108a50 0x7f79f4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:34.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.901+0000 7f79fbe31700 1 --2- 172.21.15.98:0/2728704681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79f4102a90 0x7f79f4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.901+0000 7f79fbe31700 1 -- 172.21.15.98:0/2728704681 >> 172.21.15.98:0/2728704681 conn(0x7f79f4079ba0 msgr2=0x7f79f4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.901+0000 7f79fbe31700 1 -- 172.21.15.98:0/2728704681 shutdown_connections 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.901+0000 7f79fbe31700 1 -- 172.21.15.98:0/2728704681 wait complete. 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79fbe31700 1 Processor -- start 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79fbe31700 1 -- start start 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79fbe31700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79f4102a90 0x7f79f4199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79fbe31700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4103450 0x7f79f41996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79fbe31700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79f4108a50 0x7f79f419d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79fbe31700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79f4112c80 con 0x7f79f4103450 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79fbe31700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f79f4112b00 con 0x7f79f4102a90 2024-04-25T17:31:34.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79fbe31700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f79f4112e00 con 0x7f79f4108a50 2024-04-25T17:31:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79f93cc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4103450 0x7f79f41996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79f93cc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4103450 0x7f79f41996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48894/0 (socket says 172.21.15.98:48894) 2024-04-25T17:31:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79f93cc700 1 -- 172.21.15.98:0/2423367512 learned_addr learned my addr 172.21.15.98:0/2423367512 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:31:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79f9bcd700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79f4102a90 0x7f79f4199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79fa3ce700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79f4108a50 0x7f79f419d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79f93cc700 1 -- 172.21.15.98:0/2423367512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79f4108a50 msgr2=0x7f79f419d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79f93cc700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79f4108a50 0x7f79f419d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79f93cc700 1 -- 172.21.15.98:0/2423367512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79f4102a90 msgr2=0x7f79f4199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79f93cc700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79f4102a90 0x7f79f4199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:34.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79f93cc700 1 -- 172.21.15.98:0/2423367512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f419e130 con 0x7f79f4103450 2024-04-25T17:31:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79f93cc700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4103450 0x7f79f41996b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f79f000d980 tx=0x7f79f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.902+0000 7f79e6ffd700 1 -- 172.21.15.98:0/2423367512 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f0017070 con 0x7f79f4103450 2024-04-25T17:31:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.903+0000 7f79fbe31700 1 -- 172.21.15.98:0/2423367512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f419e420 con 0x7f79f4103450 2024-04-25T17:31:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.903+0000 7f79e6ffd700 1 -- 172.21.15.98:0/2423367512 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f0005020 con 0x7f79f4103450 2024-04-25T17:31:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.903+0000 7f79e6ffd700 1 -- 172.21.15.98:0/2423367512 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f000fa10 con 0x7f79f4103450 2024-04-25T17:31:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.903+0000 7f79fbe31700 1 -- 172.21.15.98:0/2423367512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f41a5ca0 con 0x7f79f4103450 2024-04-25T17:31:34.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.903+0000 7f79fbe31700 1 -- 172.21.15.98:0/2423367512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f4069730 con 0x7f79f4103450 2024-04-25T17:31:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.905+0000 7f79e6ffd700 1 -- 172.21.15.98:0/2423367512 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f79f0004a30 con 0x7f79f4103450 2024-04-25T17:31:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.905+0000 7f79e6ffd700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f79e00a3c40 0x7f79e00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.905+0000 7f79f9bcd700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f79e00a3c40 0x7f79e00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:34.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.905+0000 7f79e6ffd700 1 -- 172.21.15.98:0/2423367512 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f79f00d0230 con 0x7f79f4103450 2024-04-25T17:31:34.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.906+0000 7f79f9bcd700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f79e00a3c40 0x7f79e00a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009a10 tx=0x7f79e8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:34.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:34.909+0000 7f79e6ffd700 1 -- 172.21.15.98:0/2423367512 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79f0020020 con 0x7f79f4103450 2024-04-25T17:31:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.109+0000 7f79fbe31700 1 -- 172.21.15.98:0/2423367512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79f419a480 con 0x7f79f4103450 2024-04-25T17:31:35.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.110+0000 7f79e6ffd700 1 -- 172.21.15.98:0/2423367512 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f79f009d0b0 con 0x7f79f4103450 2024-04-25T17:31:35.111 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T17:31:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 -- 172.21.15.98:0/2423367512 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f79e00a3c40 msgr2=0x7f79e00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f79e00a3c40 0x7f79e00a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009a10 tx=0x7f79e8009190 comp rx=0 tx=0).stop 2024-04-25T17:31:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 -- 172.21.15.98:0/2423367512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4103450 msgr2=0x7f79f41996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4103450 0x7f79f41996b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f79f000d980 tx=0x7f79f000dc90 comp rx=0 tx=0).stop 2024-04-25T17:31:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 -- 172.21.15.98:0/2423367512 shutdown_connections 2024-04-25T17:31:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f79e00a3c40 0x7f79e00a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79f4108a50 0x7f79f419d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79f4102a90 0x7f79f4199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 --2- 172.21.15.98:0/2423367512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4103450 0x7f79f41996b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:35.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 -- 172.21.15.98:0/2423367512 >> 172.21.15.98:0/2423367512 conn(0x7f79f4079ba0 msgr2=0x7f79f41017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:35.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 -- 172.21.15.98:0/2423367512 shutdown_connections 2024-04-25T17:31:35.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:35.112+0000 7f79fbe31700 1 -- 172.21.15.98:0/2423367512 wait complete. 2024-04-25T17:31:35.940 INFO:tasks.cephadm:Adding mds.a on smithi098 2024-04-25T17:31:35.940 INFO:tasks.cephadm:Adding mds.d on smithi098 2024-04-25T17:31:35.940 INFO:tasks.cephadm:Adding mds.g on smithi098 2024-04-25T17:31:35.941 INFO:tasks.cephadm:Adding mds.j on smithi098 2024-04-25T17:31:35.941 INFO:tasks.cephadm:Adding mds.b on smithi173 2024-04-25T17:31:35.941 INFO:tasks.cephadm:Adding mds.e on smithi173 2024-04-25T17:31:35.941 INFO:tasks.cephadm:Adding mds.h on smithi173 2024-04-25T17:31:35.941 INFO:tasks.cephadm:Adding mds.k on smithi173 2024-04-25T17:31:35.941 INFO:tasks.cephadm:Adding mds.c on smithi203 2024-04-25T17:31:35.941 INFO:tasks.cephadm:Adding mds.f on smithi203 2024-04-25T17:31:35.941 INFO:tasks.cephadm:Adding mds.i on smithi203 2024-04-25T17:31:35.941 INFO:tasks.cephadm:Adding mds.l on smithi203 2024-04-25T17:31:35.941 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l' 2024-04-25T17:31:36.192 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:31:36.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:36 smithi098 ceph-mon[29192]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:36.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:36 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2423367512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T17:31:36.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:36 smithi173 ceph-mon[32856]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:36.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:36 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2423367512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T17:31:36.314 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:36 smithi203 ceph-mon[32834]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:36.314 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:36 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2423367512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T17:31:37.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.913+0000 7f70e7926700 1 -- 172.21.15.203:0/1553467114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70e0103cc0 msgr2=0x7f70e0104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.913+0000 7f70e7926700 1 --2- 172.21.15.203:0/1553467114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70e0103cc0 0x7f70e0104120 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009990 tx=0x7f70dc009ca0 comp rx=0 tx=0).stop 2024-04-25T17:31:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.913+0000 7f70e7926700 1 -- 172.21.15.203:0/1553467114 shutdown_connections 2024-04-25T17:31:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.913+0000 7f70e7926700 1 --2- 172.21.15.203:0/1553467114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70e0104660 0x7f70e010af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.913+0000 7f70e7926700 1 --2- 172.21.15.203:0/1553467114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e0102ac0 0x7f70e0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.913+0000 7f70e7926700 1 --2- 172.21.15.203:0/1553467114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70e0103cc0 0x7f70e0104120 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.913+0000 7f70e7926700 1 -- 172.21.15.203:0/1553467114 >> 172.21.15.203:0/1553467114 conn(0x7f70e00fe060 msgr2=0x7f70e01004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.913+0000 7f70e7926700 1 -- 172.21.15.203:0/1553467114 shutdown_connections 2024-04-25T17:31:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.913+0000 7f70e7926700 1 -- 172.21.15.203:0/1553467114 wait complete. 2024-04-25T17:31:37.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e7926700 1 Processor -- start 2024-04-25T17:31:37.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e7926700 1 -- start start 2024-04-25T17:31:37.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e7926700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70e0102ac0 0x7f70e0198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:37.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e7926700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70e0103cc0 0x7f70e01990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e7926700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e0104660 0x7f70e01a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e7926700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f70e010e560 con 0x7f70e0104660 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e7926700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f70e010e3e0 con 0x7f70e0102ac0 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e7926700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f70e010e6e0 con 0x7f70e0103cc0 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e4ec1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70e0103cc0 0x7f70e01990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e4ec1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70e0103cc0 0x7f70e01990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:53770/0 (socket says 172.21.15.203:53770) 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e4ec1700 1 -- 172.21.15.203:0/1320453126 learned_addr learned my addr 172.21.15.203:0/1320453126 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e56c2700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70e0102ac0 0x7f70e0198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e5ec3700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e0104660 0x7f70e01a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e4ec1700 1 -- 172.21.15.203:0/1320453126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70e0102ac0 msgr2=0x7f70e0198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e4ec1700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70e0102ac0 0x7f70e0198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e4ec1700 1 -- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e0104660 msgr2=0x7f70e01a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e4ec1700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e0104660 0x7f70e01a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e4ec1700 1 -- 172.21.15.203:0/1320453126 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e01a0800 con 0x7f70e0103cc0 2024-04-25T17:31:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.914+0000 7f70e5ec3700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e0104660 0x7f70e01a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:31:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.915+0000 7f70e4ec1700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70e0103cc0 0x7f70e01990e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0036f0 tx=0x7f70dc0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.915+0000 7f70d27fc700 1 -- 172.21.15.203:0/1320453126 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc027070 con 0x7f70e0103cc0 2024-04-25T17:31:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.915+0000 7f70e7926700 1 -- 172.21.15.203:0/1320453126 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e01a0a90 con 0x7f70e0103cc0 2024-04-25T17:31:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.915+0000 7f70d27fc700 1 -- 172.21.15.203:0/1320453126 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70dc004100 con 0x7f70e0103cc0 2024-04-25T17:31:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.915+0000 7f70d27fc700 1 -- 172.21.15.203:0/1320453126 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc022d10 con 0x7f70e0103cc0 2024-04-25T17:31:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.915+0000 7f70e7926700 1 -- 172.21.15.203:0/1320453126 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e01a0f40 con 0x7f70e0103cc0 2024-04-25T17:31:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.916+0000 7f70e7926700 1 -- 172.21.15.203:0/1320453126 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e0069730 con 0x7f70e0103cc0 2024-04-25T17:31:37.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.917+0000 7f70d27fc700 1 -- 172.21.15.203:0/1320453126 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f70dc02a070 con 0x7f70e0103cc0 2024-04-25T17:31:37.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.918+0000 7f70d27fc700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f70cc0a3cc0 0x7f70cc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:37.922 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.918+0000 7f70d27fc700 1 -- 172.21.15.203:0/1320453126 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f70dc0d0e80 con 0x7f70e0103cc0 2024-04-25T17:31:37.922 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.918+0000 7f70e56c2700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f70cc0a3cc0 0x7f70cc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:37.922 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.919+0000 7f70e56c2700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f70cc0a3cc0 0x7f70cc0a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f70e0103b20 tx=0x7f70d400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:37.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:37.922+0000 7f70d27fc700 1 -- 172.21.15.203:0/1320453126 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f70dc013070 con 0x7f70e0103cc0 2024-04-25T17:31:38.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.130+0000 7f70e7926700 1 -- 172.21.15.203:0/1320453126 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}) -- 0x7f70e0063140 con 0x7f70cc0a3cc0 2024-04-25T17:31:38.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.138+0000 7f70d27fc700 1 -- 172.21.15.203:0/1320453126 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f70e0063140 con 0x7f70cc0a3cc0 2024-04-25T17:31:38.140 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mds.all update... 2024-04-25T17:31:38.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 -- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f70cc0a3cc0 msgr2=0x7f70cc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:38.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f70cc0a3cc0 0x7f70cc0a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f70e0103b20 tx=0x7f70d400a380 comp rx=0 tx=0).stop 2024-04-25T17:31:38.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 -- 172.21.15.203:0/1320453126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70e0103cc0 msgr2=0x7f70e01990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:38.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70e0103cc0 0x7f70e01990e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0036f0 tx=0x7f70dc0047b0 comp rx=0 tx=0).stop 2024-04-25T17:31:38.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 -- 172.21.15.203:0/1320453126 shutdown_connections 2024-04-25T17:31:38.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f70cc0a3cc0 0x7f70cc0a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:38.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e0104660 0x7f70e01a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:38.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70e0102ac0 0x7f70e0198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:38.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 --2- 172.21.15.203:0/1320453126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70e0103cc0 0x7f70e01990e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:38.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 -- 172.21.15.203:0/1320453126 >> 172.21.15.203:0/1320453126 conn(0x7f70e00fe060 msgr2=0x7f70e0106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:38.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 -- 172.21.15.203:0/1320453126 shutdown_connections 2024-04-25T17:31:38.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T17:31:38.140+0000 7f70e7926700 1 -- 172.21.15.203:0/1320453126 wait complete. 2024-04-25T17:31:38.175 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:38 smithi203 ceph-mon[32834]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:38.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:38 smithi173 ceph-mon[32856]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:38 smithi098 ceph-mon[29192]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:39.002 DEBUG:teuthology.orchestra.run.smithi098:mds.a> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T17:31:39.005 DEBUG:teuthology.orchestra.run.smithi098:mds.d> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T17:31:39.048 DEBUG:teuthology.orchestra.run.smithi098:mds.g> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T17:31:39.091 DEBUG:teuthology.orchestra.run.smithi098:mds.j> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T17:31:39.134 DEBUG:teuthology.orchestra.run.smithi173:mds.b> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T17:31:39.136 DEBUG:teuthology.orchestra.run.smithi173:mds.e> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T17:31:39.180 DEBUG:teuthology.orchestra.run.smithi173:mds.h> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T17:31:39.223 DEBUG:teuthology.orchestra.run.smithi173:mds.k> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T17:31:39.267 DEBUG:teuthology.orchestra.run.smithi203:mds.c> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T17:31:39.269 DEBUG:teuthology.orchestra.run.smithi203:mds.f> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T17:31:39.288 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:39 smithi203 ceph-mon[32834]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:31:39.288 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:39 smithi203 ceph-mon[32834]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-25T17:31:39.288 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:39.288 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:39 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:39.289 DEBUG:teuthology.orchestra.run.smithi203:mds.i> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T17:31:39.332 DEBUG:teuthology.orchestra.run.smithi203:mds.l> sudo journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T17:31:39.375 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T17:31:39.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:31:39.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:39 smithi098 ceph-mon[29192]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:31:39.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:39 smithi098 ceph-mon[29192]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-25T17:31:39.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:39.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:39 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:39.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:39 smithi173 ceph-mon[32856]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:31:39.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:39 smithi173 ceph-mon[32856]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-25T17:31:39.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:39.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:39 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:31:39.608 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:31:40.427 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:40 smithi098 ceph-mon[29192]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:40.427 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:40 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:40.427 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:40 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:40.427 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:40 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:40.427 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:40 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:40.427 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:40 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:40.427 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:40 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:40 smithi203 ceph-mon[32834]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:40 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:40 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:40 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:40.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:40 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:40.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:40 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:40.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:40 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:40 smithi173 ceph-mon[32856]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:40 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:40 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:31:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:40 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:40 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:40 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:40 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.381+0000 7ff33cd77700 1 -- 172.21.15.98:0/3075307062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff338103450 msgr2=0x7ff33810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.381+0000 7ff33cd77700 1 --2- 172.21.15.98:0/3075307062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff338103450 0x7ff33810f8b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff33000b0a0 tx=0x7ff33000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:31:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.381+0000 7ff33cd77700 1 -- 172.21.15.98:0/3075307062 shutdown_connections 2024-04-25T17:31:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.381+0000 7ff33cd77700 1 --2- 172.21.15.98:0/3075307062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff338108a50 0x7ff338108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.381+0000 7ff33cd77700 1 --2- 172.21.15.98:0/3075307062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff338103450 0x7ff33810f8b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.381+0000 7ff33cd77700 1 --2- 172.21.15.98:0/3075307062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff338102a90 0x7ff338102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.381+0000 7ff33cd77700 1 -- 172.21.15.98:0/3075307062 >> 172.21.15.98:0/3075307062 conn(0x7ff338079ba0 msgr2=0x7ff338079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.381+0000 7ff33cd77700 1 -- 172.21.15.98:0/3075307062 shutdown_connections 2024-04-25T17:31:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.381+0000 7ff33cd77700 1 -- 172.21.15.98:0/3075307062 wait complete. 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33cd77700 1 Processor -- start 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33cd77700 1 -- start start 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33cd77700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff338102a90 0x7ff338199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33cd77700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff338103450 0x7ff3381995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33cd77700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff338108a50 0x7ff33819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33cd77700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff338112c60 con 0x7ff338102a90 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33cd77700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff338112ae0 con 0x7ff338103450 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33cd77700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff338112de0 con 0x7ff338108a50 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff338102a90 0x7ff338199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff338102a90 0x7ff338199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48912/0 (socket says 172.21.15.98:48912) 2024-04-25T17:31:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33659c700 1 -- 172.21.15.98:0/3025267036 learned_addr learned my addr 172.21.15.98:0/3025267036 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff336d9d700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff338108a50 0x7ff33819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff335d9b700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff338103450 0x7ff3381995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33659c700 1 -- 172.21.15.98:0/3025267036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff338108a50 msgr2=0x7ff33819d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33659c700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff338108a50 0x7ff33819d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33659c700 1 -- 172.21.15.98:0/3025267036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff338103450 msgr2=0x7ff3381995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33659c700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff338103450 0x7ff3381995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.382+0000 7ff33659c700 1 -- 172.21.15.98:0/3025267036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33819e030 con 0x7ff338102a90 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.383+0000 7ff33659c700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff338102a90 0x7ff338199070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff32800b530 tx=0x7ff32800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.383+0000 7ff3277fe700 1 -- 172.21.15.98:0/3025267036 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328015020 con 0x7ff338102a90 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.383+0000 7ff33cd77700 1 -- 172.21.15.98:0/3025267036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33819e320 con 0x7ff338102a90 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.383+0000 7ff3277fe700 1 -- 172.21.15.98:0/3025267036 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff328011ce0 con 0x7ff338102a90 2024-04-25T17:31:41.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.383+0000 7ff3277fe700 1 -- 172.21.15.98:0/3025267036 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328010dd0 con 0x7ff338102a90 2024-04-25T17:31:41.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.383+0000 7ff33cd77700 1 -- 172.21.15.98:0/3025267036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3381a5ba0 con 0x7ff338102a90 2024-04-25T17:31:41.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.384+0000 7ff33cd77700 1 -- 172.21.15.98:0/3025267036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff338069730 con 0x7ff338102a90 2024-04-25T17:31:41.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.385+0000 7ff3277fe700 1 -- 172.21.15.98:0/3025267036 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff328018020 con 0x7ff338102a90 2024-04-25T17:31:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.385+0000 7ff3277fe700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff3200a3c40 0x7ff3200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.386+0000 7ff335d9b700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff3200a3c40 0x7ff3200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.386+0000 7ff3277fe700 1 -- 172.21.15.98:0/3025267036 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff3280cfbe0 con 0x7ff338102a90 2024-04-25T17:31:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.386+0000 7ff335d9b700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff3200a3c40 0x7ff3200a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff33807a1a0 tx=0x7ff32c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:41.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.389+0000 7ff3277fe700 1 -- 172.21.15.98:0/3025267036 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff32801f020 con 0x7ff338102a90 2024-04-25T17:31:41.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:41 smithi098 ceph-mon[29192]: Deploying daemon mds.l on smithi203 2024-04-25T17:31:41.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:41 smithi203 ceph-mon[32834]: Deploying daemon mds.l on smithi203 2024-04-25T17:31:41.463 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:31:41 smithi203 systemd[1]: Starting Ceph mds.l for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:31:41.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:41 smithi173 ceph-mon[32856]: Deploying daemon mds.l on smithi203 2024-04-25T17:31:41.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.632+0000 7ff33cd77700 1 -- 172.21.15.98:0/3025267036 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff33819a380 con 0x7ff338102a90 2024-04-25T17:31:41.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.632+0000 7ff3277fe700 1 -- 172.21.15.98:0/3025267036 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7ff32809ca60 con 0x7ff338102a90 2024-04-25T17:31:41.634 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:31:41.634 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T17:31:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 -- 172.21.15.98:0/3025267036 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff3200a3c40 msgr2=0x7ff3200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff3200a3c40 0x7ff3200a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff33807a1a0 tx=0x7ff32c00a380 comp rx=0 tx=0).stop 2024-04-25T17:31:41.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 -- 172.21.15.98:0/3025267036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff338102a90 msgr2=0x7ff338199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff338102a90 0x7ff338199070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff32800b530 tx=0x7ff32800b840 comp rx=0 tx=0).stop 2024-04-25T17:31:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 -- 172.21.15.98:0/3025267036 shutdown_connections 2024-04-25T17:31:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff3200a3c40 0x7ff3200a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff338108a50 0x7ff33819d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff338102a90 0x7ff338199070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 --2- 172.21.15.98:0/3025267036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff338103450 0x7ff3381995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 -- 172.21.15.98:0/3025267036 >> 172.21.15.98:0/3025267036 conn(0x7ff338079ba0 msgr2=0x7ff3381016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 -- 172.21.15.98:0/3025267036 shutdown_connections 2024-04-25T17:31:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:41.634+0000 7ff33cd77700 1 -- 172.21.15.98:0/3025267036 wait complete. 2024-04-25T17:31:41.636 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 1 2024-04-25T17:31:41.806 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:31:41 smithi203 podman[62925]: 2024-04-25 17:31:41.42875049 +0000 UTC m=+0.018601179 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:31:41.806 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:31:41 smithi203 podman[62925]: 2024-04-25 17:31:41.552307609 +0000 UTC m=+0.142158267 container create e1e5757a8f297031bbadb2f1a58944e0d73b9f7cf0ec1a17c26b9d27bb27e1a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:31:41.806 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:31:41 smithi203 podman[62925]: 2024-04-25 17:31:41.805227613 +0000 UTC m=+0.395078279 container init e1e5757a8f297031bbadb2f1a58944e0d73b9f7cf0ec1a17c26b9d27bb27e1a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416) 2024-04-25T17:31:42.151 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:31:41 smithi203 podman[62925]: 2024-04-25 17:31:41.808333331 +0000 UTC m=+0.398183987 container start e1e5757a8f297031bbadb2f1a58944e0d73b9f7cf0ec1a17c26b9d27bb27e1a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:31:42.151 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:31:41 smithi203 ceph-mds[62944]: -- 172.21.15.203:0/2665286337 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562de2ad9a40 con 0x562de2aad000 2024-04-25T17:31:42.151 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:31:41 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-l[62940]: starting mds.l at 2024-04-25T17:31:42.152 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:31:41 smithi203 bash[62925]: e1e5757a8f297031bbadb2f1a58944e0d73b9f7cf0ec1a17c26b9d27bb27e1a6 2024-04-25T17:31:42.152 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:31:41 smithi203 systemd[1]: Started Ceph mds.l for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:31:42.422 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:42.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3025267036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:42.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.203:6832/2408762956,v1:172.21.15.203:6833/2408762956] up:boot 2024-04-25T17:31:42.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: fsmap 1 up:standby 2024-04-25T17:31:42.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T17:31:42.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:42.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:42.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:42.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:42.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:42.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3025267036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.203:6832/2408762956,v1:172.21.15.203:6833/2408762956] up:boot 2024-04-25T17:31:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: fsmap 1 up:standby 2024-04-25T17:31:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T17:31:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:42.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:42.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:42.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:42.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:42.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3025267036' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.203:6832/2408762956,v1:172.21.15.203:6833/2408762956] up:boot 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: fsmap 1 up:standby 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:43.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:43 smithi098 ceph-mon[29192]: Deploying daemon mds.k on smithi173 2024-04-25T17:31:43.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:43 smithi203 ceph-mon[32834]: Deploying daemon mds.k on smithi173 2024-04-25T17:31:43.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:43 smithi173 ceph-mon[32856]: Deploying daemon mds.k on smithi173 2024-04-25T17:31:44.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:31:44.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:44 smithi098 ceph-mon[29192]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:44.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:44 smithi203 ceph-mon[32834]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:44.468 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:44 smithi173 ceph-mon[32856]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:44.468 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:31:44 smithi173 systemd[1]: Starting Ceph mds.k for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:31:44.699 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:31:44.743 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:31:44 smithi173 podman[62181]: 2024-04-25 17:31:44.370276474 +0000 UTC m=+0.017690070 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:31:44.744 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:31:44 smithi173 podman[62181]: 2024-04-25 17:31:44.482399776 +0000 UTC m=+0.129813407 container create 48a48d1de11cafabf0c31d7f6e576ea5f3f83e1c42b5bc0f47646489d97abf9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T17:31:44.744 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:31:44 smithi173 podman[62181]: 2024-04-25 17:31:44.743445102 +0000 UTC m=+0.390858691 container init 48a48d1de11cafabf0c31d7f6e576ea5f3f83e1c42b5bc0f47646489d97abf9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:31:45.023 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:31:44 smithi173 podman[62181]: 2024-04-25 17:31:44.74630923 +0000 UTC m=+0.393722828 container start 48a48d1de11cafabf0c31d7f6e576ea5f3f83e1c42b5bc0f47646489d97abf9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T17:31:45.023 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:31:44 smithi173 ceph-mds[62200]: -- 172.21.15.173:0/3225386954 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b0348d9a40 con 0x55b0348acc00 2024-04-25T17:31:45.023 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:31:44 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-k[62196]: starting mds.k at 2024-04-25T17:31:45.023 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:31:44 smithi173 bash[62181]: 48a48d1de11cafabf0c31d7f6e576ea5f3f83e1c42b5bc0f47646489d97abf9e 2024-04-25T17:31:45.023 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:31:44 smithi173 systemd[1]: Started Ceph mds.k for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:31:46.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:46.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:46.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:46.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:46.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:46.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:46.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: Deploying daemon mds.j on smithi098 2024-04-25T17:31:46.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:46.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.173:6832/2248833890,v1:172.21.15.173:6833/2248833890] up:boot 2024-04-25T17:31:46.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: fsmap 2 up:standby 2024-04-25T17:31:46.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:45 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T17:31:46.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:46.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:46.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:46.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:46.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:46.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:46.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: Deploying daemon mds.j on smithi098 2024-04-25T17:31:46.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:46.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.173:6832/2248833890,v1:172.21.15.173:6833/2248833890] up:boot 2024-04-25T17:31:46.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: fsmap 2 up:standby 2024-04-25T17:31:46.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:45 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T17:31:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:46.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: Deploying daemon mds.j on smithi098 2024-04-25T17:31:46.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:46.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.173:6832/2248833890,v1:172.21.15.173:6833/2248833890] up:boot 2024-04-25T17:31:46.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: fsmap 2 up:standby 2024-04-25T17:31:46.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:45 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T17:31:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.704+0000 7faad9d83700 1 -- 172.21.15.98:0/1417244845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faad4102330 msgr2=0x7faad410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.704+0000 7faad9d83700 1 --2- 172.21.15.98:0/1417244845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faad4102330 0x7faad410e7f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faacc00b0a0 tx=0x7faacc00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.704+0000 7faad9d83700 1 -- 172.21.15.98:0/1417244845 shutdown_connections 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.704+0000 7faad9d83700 1 --2- 172.21.15.98:0/1417244845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faad4102330 0x7faad410e7f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.704+0000 7faad9d83700 1 --2- 172.21.15.98:0/1417244845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faad4107920 0x7faad4107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.704+0000 7faad9d83700 1 --2- 172.21.15.98:0/1417244845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad4101970 0x7faad4101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.704+0000 7faad9d83700 1 -- 172.21.15.98:0/1417244845 >> 172.21.15.98:0/1417244845 conn(0x7faad40788f0 msgr2=0x7faad4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.704+0000 7faad9d83700 1 -- 172.21.15.98:0/1417244845 shutdown_connections 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.704+0000 7faad9d83700 1 -- 172.21.15.98:0/1417244845 wait complete. 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad9d83700 1 Processor -- start 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad9d83700 1 -- start start 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad9d83700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faad4101970 0x7faad4199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad9d83700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad4102330 0x7faad4199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad9d83700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faad4107920 0x7faad419d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad9d83700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faad4111b60 con 0x7faad4101970 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad9d83700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faad41119e0 con 0x7faad4107920 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad9d83700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faad4111ce0 con 0x7faad4102330 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faad4101970 0x7faad4199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faad4101970 0x7faad4199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53434/0 (socket says 172.21.15.98:53434) 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad37fe700 1 -- 172.21.15.98:0/3458918733 learned_addr learned my addr 172.21.15.98:0/3458918733 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.705+0000 7faad2ffd700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad4102330 0x7faad4199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad3fff700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faad4107920 0x7faad419d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad37fe700 1 -- 172.21.15.98:0/3458918733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad4102330 msgr2=0x7faad4199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad37fe700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad4102330 0x7faad4199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad37fe700 1 -- 172.21.15.98:0/3458918733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faad4107920 msgr2=0x7faad419d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad37fe700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faad4107920 0x7faad419d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad37fe700 1 -- 172.21.15.98:0/3458918733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad419e120 con 0x7faad4101970 2024-04-25T17:31:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad3fff700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faad4107920 0x7faad419d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:31:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad37fe700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faad4101970 0x7faad4199110 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faac400b530 tx=0x7faac400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad0ff9700 1 -- 172.21.15.98:0/3458918733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4015020 con 0x7faad4101970 2024-04-25T17:31:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad9d83700 1 -- 172.21.15.98:0/3458918733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad419e410 con 0x7faad4101970 2024-04-25T17:31:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad0ff9700 1 -- 172.21.15.98:0/3458918733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faac4011ce0 con 0x7faad4101970 2024-04-25T17:31:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad9d83700 1 -- 172.21.15.98:0/3458918733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad41a5c90 con 0x7faad4101970 2024-04-25T17:31:46.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.706+0000 7faad0ff9700 1 -- 172.21.15.98:0/3458918733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4010d20 con 0x7faad4101970 2024-04-25T17:31:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.707+0000 7faad9d83700 1 -- 172.21.15.98:0/3458918733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac0005330 con 0x7faad4101970 2024-04-25T17:31:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.708+0000 7faad0ff9700 1 -- 172.21.15.98:0/3458918733 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faac4018020 con 0x7faad4101970 2024-04-25T17:31:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.709+0000 7faad0ff9700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faabc0a3c40 0x7faabc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.709+0000 7faad2ffd700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faabc0a3c40 0x7faabc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.709+0000 7faad0ff9700 1 -- 172.21.15.98:0/3458918733 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7faac40cfc60 con 0x7faad4101970 2024-04-25T17:31:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.709+0000 7faad2ffd700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faabc0a3c40 0x7faabc0a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faac8005950 tx=0x7faac80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:46.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.713+0000 7faad0ff9700 1 -- 172.21.15.98:0/3458918733 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faac401f020 con 0x7faad4101970 2024-04-25T17:31:46.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.960+0000 7faad9d83700 1 -- 172.21.15.98:0/3458918733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faac0005620 con 0x7faad4101970 2024-04-25T17:31:46.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.960+0000 7faad0ff9700 1 -- 172.21.15.98:0/3458918733 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7faac409cae0 con 0x7faad4101970 2024-04-25T17:31:46.962 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:31:46.962 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T17:31:46.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.962+0000 7faad9d83700 1 -- 172.21.15.98:0/3458918733 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faabc0a3c40 msgr2=0x7faabc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.962+0000 7faad9d83700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faabc0a3c40 0x7faabc0a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faac8005950 tx=0x7faac80058e0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.962+0000 7faad9d83700 1 -- 172.21.15.98:0/3458918733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faad4101970 msgr2=0x7faad4199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.962+0000 7faad9d83700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faad4101970 0x7faad4199110 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faac400b530 tx=0x7faac400b8f0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.962+0000 7faad9d83700 1 -- 172.21.15.98:0/3458918733 shutdown_connections 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.962+0000 7faad9d83700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faabc0a3c40 0x7faabc0a6100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.962+0000 7faad9d83700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faad4107920 0x7faad419d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.962+0000 7faad9d83700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faad4101970 0x7faad4199110 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.962+0000 7faad9d83700 1 --2- 172.21.15.98:0/3458918733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad4102330 0x7faad4199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.962+0000 7faad9d83700 1 -- 172.21.15.98:0/3458918733 >> 172.21.15.98:0/3458918733 conn(0x7faad40788f0 msgr2=0x7faad406b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.963+0000 7faad9d83700 1 -- 172.21.15.98:0/3458918733 shutdown_connections 2024-04-25T17:31:46.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:46.963+0000 7faad9d83700 1 -- 172.21.15.98:0/3458918733 wait complete. 2024-04-25T17:31:46.965 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2024-04-25T17:31:47.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:47 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3458918733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:47 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3458918733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:47.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:47 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3458918733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:48.201 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:48 smithi098 ceph-mon[29192]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:48.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:48 smithi173 ceph-mon[32856]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:48.365 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:48 smithi203 ceph-mon[32834]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:50.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:50 smithi098 ceph-mon[29192]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:50.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:50.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:50.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:50.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:50.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:50.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:50 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:50.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:31:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:50 smithi203 ceph-mon[32834]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:50 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:50.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:50 smithi173 ceph-mon[32856]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:50.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:50.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:50.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:50.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:50.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:50.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:50 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:51.000 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:31:51.161 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:51 smithi098 ceph-mon[29192]: Deploying daemon mds.d on smithi098 2024-04-25T17:31:51.161 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:51 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.98:6834/682650272,v1:172.21.15.98:6835/682650272] up:boot 2024-04-25T17:31:51.161 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:51 smithi098 ceph-mon[29192]: fsmap 3 up:standby 2024-04-25T17:31:51.161 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:51 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T17:31:51.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:51 smithi203 ceph-mon[32834]: Deploying daemon mds.d on smithi098 2024-04-25T17:31:51.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:51 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.98:6834/682650272,v1:172.21.15.98:6835/682650272] up:boot 2024-04-25T17:31:51.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:51 smithi203 ceph-mon[32834]: fsmap 3 up:standby 2024-04-25T17:31:51.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:51 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T17:31:51.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:51 smithi173 ceph-mon[32856]: Deploying daemon mds.d on smithi098 2024-04-25T17:31:51.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:51 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.98:6834/682650272,v1:172.21.15.98:6835/682650272] up:boot 2024-04-25T17:31:51.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:51 smithi173 ceph-mon[32856]: fsmap 3 up:standby 2024-04-25T17:31:51.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:51 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T17:31:52.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:52 smithi098 ceph-mon[29192]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:52 smithi203 ceph-mon[32834]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:52 smithi173 ceph-mon[32856]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:53.019 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:31:52 smithi098 podman[70552]: 2024-04-25 17:31:52.995483714 +0000 UTC m=+0.625623220 container start beecde30ebb8f2ad4cc9c5bcd638ed1111268cc747de198de52664ba884a1eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-25T17:31:53.336 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:31:53 smithi098 ceph-mds[70593]: -- 172.21.15.98:0/798707249 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559ce183da40 con 0x559ce1810800 2024-04-25T17:31:53.337 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:31:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-d[70584]: starting mds.d at 2024-04-25T17:31:53.337 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:31:53 smithi098 bash[70552]: beecde30ebb8f2ad4cc9c5bcd638ed1111268cc747de198de52664ba884a1eec 2024-04-25T17:31:53.337 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:31:53 smithi098 systemd[1]: Started Ceph mds.d for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:31:53.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.334+0000 7f7414049700 1 -- 172.21.15.98:0/17511995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f740c101970 msgr2=0x7f740c101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.334+0000 7f7414049700 1 --2- 172.21.15.98:0/17511995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f740c101970 0x7f740c101df0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f740000b0a0 tx=0x7f740000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.334+0000 7f7414049700 1 -- 172.21.15.98:0/17511995 shutdown_connections 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.334+0000 7f7414049700 1 --2- 172.21.15.98:0/17511995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f740c107920 0x7f740c107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.334+0000 7f7414049700 1 --2- 172.21.15.98:0/17511995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f740c102330 0x7f740c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.334+0000 7f7414049700 1 --2- 172.21.15.98:0/17511995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f740c101970 0x7f740c101df0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.334+0000 7f7414049700 1 -- 172.21.15.98:0/17511995 >> 172.21.15.98:0/17511995 conn(0x7f740c0779b0 msgr2=0x7f740c077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.335+0000 7f7414049700 1 -- 172.21.15.98:0/17511995 shutdown_connections 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.335+0000 7f7414049700 1 -- 172.21.15.98:0/17511995 wait complete. 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.335+0000 7f7414049700 1 Processor -- start 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.335+0000 7f7414049700 1 -- start start 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.335+0000 7f7414049700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f740c101970 0x7f740c199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.335+0000 7f7414049700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f740c102330 0x7f740c19a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.335+0000 7f7414049700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f740c107920 0x7f740c1a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:53.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.335+0000 7f7414049700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f740c111b60 con 0x7f740c101970 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.335+0000 7f7414049700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f740c1119e0 con 0x7f740c107920 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.335+0000 7f7414049700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f740c111ce0 con 0x7f740c102330 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7411de5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f740c101970 0x7f740c199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7411de5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f740c101970 0x7f740c199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53494/0 (socket says 172.21.15.98:53494) 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7411de5700 1 -- 172.21.15.98:0/2245791813 learned_addr learned my addr 172.21.15.98:0/2245791813 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f74125e6700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f740c107920 0x7f740c1a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f74115e4700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f740c102330 0x7f740c19a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7411de5700 1 -- 172.21.15.98:0/2245791813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f740c102330 msgr2=0x7f740c19a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7411de5700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f740c102330 0x7f740c19a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7411de5700 1 -- 172.21.15.98:0/2245791813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f740c107920 msgr2=0x7f740c1a5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7411de5700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f740c107920 0x7f740c1a5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7411de5700 1 -- 172.21.15.98:0/2245791813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740c1a5540 con 0x7f740c101970 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7411de5700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f740c101970 0x7f740c199f80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f740400c970 tx=0x7f740400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f740affd700 1 -- 172.21.15.98:0/2245791813 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74040078a0 con 0x7f740c101970 2024-04-25T17:31:53.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7414049700 1 -- 172.21.15.98:0/2245791813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f740c1a5770 con 0x7f740c101970 2024-04-25T17:31:53.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f740affd700 1 -- 172.21.15.98:0/2245791813 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7404004830 con 0x7f740c101970 2024-04-25T17:31:53.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f740affd700 1 -- 172.21.15.98:0/2245791813 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74040054c0 con 0x7f740c101970 2024-04-25T17:31:53.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.336+0000 7f7414049700 1 -- 172.21.15.98:0/2245791813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f740c1a5c80 con 0x7f740c101970 2024-04-25T17:31:53.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.337+0000 7f7414049700 1 -- 172.21.15.98:0/2245791813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f740c109eb0 con 0x7f740c101970 2024-04-25T17:31:53.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.338+0000 7f740affd700 1 -- 172.21.15.98:0/2245791813 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7404007a40 con 0x7f740c101970 2024-04-25T17:31:53.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.339+0000 7f740affd700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73f40a3c40 0x7f73f40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:53.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.339+0000 7f74115e4700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73f40a3c40 0x7f73f40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:53.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.339+0000 7f740affd700 1 -- 172.21.15.98:0/2245791813 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f74040cfc40 con 0x7f740c101970 2024-04-25T17:31:53.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.339+0000 7f74115e4700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73f40a3c40 0x7f73f40a6100 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f740c0780f0 tx=0x7f740000b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:53.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.344+0000 7f740affd700 1 -- 172.21.15.98:0/2245791813 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f740409cb50 con 0x7f740c101970 2024-04-25T17:31:53.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.583+0000 7f7414049700 1 -- 172.21.15.98:0/2245791813 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f740c109890 con 0x7f740c101970 2024-04-25T17:31:53.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.584+0000 7f740affd700 1 -- 172.21.15.98:0/2245791813 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f740409c450 con 0x7f740c101970 2024-04-25T17:31:53.588 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:31:53.588 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T17:31:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.586+0000 7f7414049700 1 -- 172.21.15.98:0/2245791813 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73f40a3c40 msgr2=0x7f73f40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.586+0000 7f7414049700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73f40a3c40 0x7f73f40a6100 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f740c0780f0 tx=0x7f740000b7b0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.586+0000 7f7414049700 1 -- 172.21.15.98:0/2245791813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f740c101970 msgr2=0x7f740c199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.586+0000 7f7414049700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f740c101970 0x7f740c199f80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f740400c970 tx=0x7f740400cc80 comp rx=0 tx=0).stop 2024-04-25T17:31:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.586+0000 7f7414049700 1 -- 172.21.15.98:0/2245791813 shutdown_connections 2024-04-25T17:31:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.586+0000 7f7414049700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73f40a3c40 0x7f73f40a6100 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.586+0000 7f7414049700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f740c107920 0x7f740c1a5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.586+0000 7f7414049700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f740c101970 0x7f740c199f80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.586+0000 7f7414049700 1 --2- 172.21.15.98:0/2245791813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f740c102330 0x7f740c19a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.586+0000 7f7414049700 1 -- 172.21.15.98:0/2245791813 >> 172.21.15.98:0/2245791813 conn(0x7f740c0779b0 msgr2=0x7f740c0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.587+0000 7f7414049700 1 -- 172.21.15.98:0/2245791813 shutdown_connections 2024-04-25T17:31:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:53.587+0000 7f7414049700 1 -- 172.21.15.98:0/2245791813 wait complete. 2024-04-25T17:31:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 5 2024-04-25T17:31:54.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:54.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.98:6836/3048794856,v1:172.21.15.98:6837/3048794856] up:boot 2024-04-25T17:31:54.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: fsmap 4 up:standby 2024-04-25T17:31:54.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T17:31:54.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:54.103 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:54.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:54.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:54.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:54.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:54.104 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:54 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2245791813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:54.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:54.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.98:6836/3048794856,v1:172.21.15.98:6837/3048794856] up:boot 2024-04-25T17:31:54.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: fsmap 4 up:standby 2024-04-25T17:31:54.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T17:31:54.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:54.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:54.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:54.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:54.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:54.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:54.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:54 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2245791813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.98:6836/3048794856,v1:172.21.15.98:6837/3048794856] up:boot 2024-04-25T17:31:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: fsmap 4 up:standby 2024-04-25T17:31:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T17:31:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:54 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2245791813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:55.199 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:55 smithi173 ceph-mon[32856]: Deploying daemon mds.e on smithi173 2024-04-25T17:31:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:55 smithi098 ceph-mon[29192]: Deploying daemon mds.e on smithi173 2024-04-25T17:31:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:55 smithi203 ceph-mon[32834]: Deploying daemon mds.e on smithi173 2024-04-25T17:31:56.363 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:31:56.389 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:56 smithi098 ceph-mon[29192]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:56 smithi203 ceph-mon[32834]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:56.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:56 smithi173 ceph-mon[32856]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:56.523 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:31:56 smithi173 systemd[1]: Started Ceph mds.e for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:31:56.611 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:31:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:57.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:57.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: Deploying daemon mds.f on smithi203 2024-04-25T17:31:57.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:57.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.173:6834/2662617129,v1:172.21.15.173:6835/2662617129] up:boot 2024-04-25T17:31:57.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: fsmap 5 up:standby 2024-04-25T17:31:57.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:57 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T17:31:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: Deploying daemon mds.f on smithi203 2024-04-25T17:31:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:57.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.173:6834/2662617129,v1:172.21.15.173:6835/2662617129] up:boot 2024-04-25T17:31:57.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: fsmap 5 up:standby 2024-04-25T17:31:57.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:57 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T17:31:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:31:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:31:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:31:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:31:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: Deploying daemon mds.f on smithi203 2024-04-25T17:31:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:31:57.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.173:6834/2662617129,v1:172.21.15.173:6835/2662617129] up:boot 2024-04-25T17:31:57.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: fsmap 5 up:standby 2024-04-25T17:31:57.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:57 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T17:31:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.466+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1130993049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f84107b20 msgr2=0x7f4f84107f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.466+0000 7f4f88b1a700 1 --2- 172.21.15.98:0/1130993049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f84107b20 0x7f4f84107f00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009990 tx=0x7f4f74009ca0 comp rx=0 tx=0).stop 2024-04-25T17:31:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.466+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1130993049 shutdown_connections 2024-04-25T17:31:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.466+0000 7f4f88b1a700 1 --2- 172.21.15.98:0/1130993049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f84101dc0 0x7f4f8410f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.466+0000 7f4f88b1a700 1 --2- 172.21.15.98:0/1130993049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f84107b20 0x7f4f84107f00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.466+0000 7f4f88b1a700 1 --2- 172.21.15.98:0/1130993049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f84101400 0x7f4f84101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.466+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1130993049 >> 172.21.15.98:0/1130993049 conn(0x7f4f84076c60 msgr2=0x7f4f84077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.466+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1130993049 shutdown_connections 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.466+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1130993049 wait complete. 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f88b1a700 1 Processor -- start 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f88b1a700 1 -- start start 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f88b1a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f84101400 0x7f4f841991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f88b1a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f84101dc0 0x7f4f84199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f88b1a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f84107b20 0x7f4f8419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f88b1a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f84112c70 con 0x7f4f84101dc0 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f88b1a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4f84112af0 con 0x7f4f84107b20 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f88b1a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4f84112df0 con 0x7f4f84101400 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f81d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f84101dc0 0x7f4f84199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f81d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f84101dc0 0x7f4f84199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39316/0 (socket says 172.21.15.98:39316) 2024-04-25T17:31:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f81d9b700 1 -- 172.21.15.98:0/1457663290 learned_addr learned my addr 172.21.15.98:0/1457663290 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f8259c700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f84101400 0x7f4f841991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f82d9d700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f84107b20 0x7f4f8419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f81d9b700 1 -- 172.21.15.98:0/1457663290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f84101400 msgr2=0x7f4f841991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f81d9b700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f84101400 0x7f4f841991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f81d9b700 1 -- 172.21.15.98:0/1457663290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f84107b20 msgr2=0x7f4f8419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f81d9b700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f84107b20 0x7f4f8419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f81d9b700 1 -- 172.21.15.98:0/1457663290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8419e1a0 con 0x7f4f84101dc0 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.467+0000 7f4f81d9b700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f84101dc0 0x7f4f84199720 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800c970 tx=0x7f4f7800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.468+0000 7f4f737fe700 1 -- 172.21.15.98:0/1457663290 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78007aa0 con 0x7f4f84101dc0 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.468+0000 7f4f737fe700 1 -- 172.21.15.98:0/1457663290 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f78004ce0 con 0x7f4f84101dc0 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.468+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1457663290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8419e490 con 0x7f4f84101dc0 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.468+0000 7f4f737fe700 1 -- 172.21.15.98:0/1457663290 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78005510 con 0x7f4f84101dc0 2024-04-25T17:31:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.468+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1457663290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f841a5d10 con 0x7f4f84101dc0 2024-04-25T17:31:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.469+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1457663290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f84069730 con 0x7f4f84101dc0 2024-04-25T17:31:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.470+0000 7f4f737fe700 1 -- 172.21.15.98:0/1457663290 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4f78007c40 con 0x7f4f84101dc0 2024-04-25T17:31:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.470+0000 7f4f737fe700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4f6c0a3c40 0x7f4f6c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:31:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.470+0000 7f4f8259c700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4f6c0a3c40 0x7f4f6c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:31:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.470+0000 7f4f737fe700 1 -- 172.21.15.98:0/1457663290 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4f780cff70 con 0x7f4f84101dc0 2024-04-25T17:31:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.471+0000 7f4f8259c700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4f6c0a3c40 0x7f4f6c0a6100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4f74006010 tx=0x7f4f74005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:31:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.474+0000 7f4f737fe700 1 -- 172.21.15.98:0/1457663290 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f7809cf10 con 0x7f4f84101dc0 2024-04-25T17:31:58.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.714+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1457663290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f8402df80 con 0x7f4f84101dc0 2024-04-25T17:31:58.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.715+0000 7f4f737fe700 1 -- 172.21.15.98:0/1457663290 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7f4f7809c810 con 0x7f4f84101dc0 2024-04-25T17:31:58.716 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:31:58.716 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T17:31:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.717+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1457663290 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4f6c0a3c40 msgr2=0x7f4f6c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.717+0000 7f4f88b1a700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4f6c0a3c40 0x7f4f6c0a6100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4f74006010 tx=0x7f4f74005960 comp rx=0 tx=0).stop 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.717+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1457663290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f84101dc0 msgr2=0x7f4f84199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.717+0000 7f4f88b1a700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f84101dc0 0x7f4f84199720 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800c970 tx=0x7f4f7800cc80 comp rx=0 tx=0).stop 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.717+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1457663290 shutdown_connections 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.717+0000 7f4f88b1a700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4f6c0a3c40 0x7f4f6c0a6100 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.717+0000 7f4f88b1a700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f84101400 0x7f4f841991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.717+0000 7f4f88b1a700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f84107b20 0x7f4f8419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.717+0000 7f4f88b1a700 1 --2- 172.21.15.98:0/1457663290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f84101dc0 0x7f4f84199720 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.717+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1457663290 >> 172.21.15.98:0/1457663290 conn(0x7f4f84076c60 msgr2=0x7f4f84100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.718+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1457663290 shutdown_connections 2024-04-25T17:31:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:31:58.718+0000 7f4f88b1a700 1 -- 172.21.15.98:0/1457663290 wait complete. 2024-04-25T17:31:58.720 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 6 2024-04-25T17:31:58.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:31:58 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1457663290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:59.061 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:31:58 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1457663290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:59.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:31:58 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1457663290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:31:59.213 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:31:59 smithi203 systemd[1]: Started Ceph mds.f for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:32:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:00.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:00.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: Deploying daemon mds.c on smithi203 2024-04-25T17:32:00.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.203:6834/2803298626,v1:172.21.15.203:6835/2803298626] up:boot 2024-04-25T17:32:00.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: fsmap 6 up:standby 2024-04-25T17:32:00.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:00 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T17:32:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: Deploying daemon mds.c on smithi203 2024-04-25T17:32:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.203:6834/2803298626,v1:172.21.15.203:6835/2803298626] up:boot 2024-04-25T17:32:00.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: fsmap 6 up:standby 2024-04-25T17:32:00.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:00 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T17:32:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: Deploying daemon mds.c on smithi203 2024-04-25T17:32:00.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.203:6834/2803298626,v1:172.21.15.203:6835/2803298626] up:boot 2024-04-25T17:32:00.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: fsmap 6 up:standby 2024-04-25T17:32:00.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:00 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T17:32:01.403 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:32:01.463 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:32:01 smithi203 systemd[1]: Starting Ceph mds.c for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:32:01.652 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:01.894 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:32:01 smithi203 podman[63547]: 2024-04-25 17:32:01.464219585 +0000 UTC m=+0.018499220 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:32:01.894 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:32:01 smithi203 podman[63547]: 2024-04-25 17:32:01.615591199 +0000 UTC m=+0.169870823 container create 9af6cac92cbc05fb57f0aa3fe60af037ab0a0fcc89d7e27cc53cb2d5a2c00bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T17:32:01.894 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:32:01 smithi203 podman[63547]: 2024-04-25 17:32:01.893620488 +0000 UTC m=+0.447900112 container init 9af6cac92cbc05fb57f0aa3fe60af037ab0a0fcc89d7e27cc53cb2d5a2c00bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T17:32:02.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:02 smithi203 ceph-mon[32834]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:02.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:02 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:02.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:02 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:02.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:32:01 smithi203 podman[63547]: 2024-04-25 17:32:01.896456353 +0000 UTC m=+0.450735977 container start 9af6cac92cbc05fb57f0aa3fe60af037ab0a0fcc89d7e27cc53cb2d5a2c00bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T17:32:02.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:32:01 smithi203 ceph-mds[63566]: -- 172.21.15.203:0/137425910 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5647491aba40 con 0x56474917f000 2024-04-25T17:32:02.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:32:01 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: starting mds.c at 2024-04-25T17:32:02.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:32:01 smithi203 bash[63547]: 9af6cac92cbc05fb57f0aa3fe60af037ab0a0fcc89d7e27cc53cb2d5a2c00bcb 2024-04-25T17:32:02.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:32:01 smithi203 systemd[1]: Started Ceph mds.c for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:32:02.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:02 smithi098 ceph-mon[29192]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:02.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:02 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:02.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:02 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:02.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:02 smithi173 ceph-mon[32856]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:02.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:02.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:02 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:03.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:03 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.203:6836/1057173981,v1:172.21.15.203:6837/1057173981] up:boot 2024-04-25T17:32:03.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:03 smithi098 ceph-mon[29192]: fsmap 7 up:standby 2024-04-25T17:32:03.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:32:03.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:03.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:03.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:03.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:03 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:03.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:03 smithi098 ceph-mon[29192]: Deploying daemon mds.h on smithi173 2024-04-25T17:32:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:03 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.203:6836/1057173981,v1:172.21.15.203:6837/1057173981] up:boot 2024-04-25T17:32:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:03 smithi173 ceph-mon[32856]: fsmap 7 up:standby 2024-04-25T17:32:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:32:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:03 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:03.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:03 smithi173 ceph-mon[32856]: Deploying daemon mds.h on smithi173 2024-04-25T17:32:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.457+0000 7f5859592700 1 -- 172.21.15.98:0/448173213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58541019d0 msgr2=0x7f585410f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.457+0000 7f5859592700 1 --2- 172.21.15.98:0/448173213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58541019d0 0x7f585410f910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f584c00b0a0 tx=0x7f584c00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.457+0000 7f5859592700 1 -- 172.21.15.98:0/448173213 shutdown_connections 2024-04-25T17:32:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.457+0000 7f5859592700 1 --2- 172.21.15.98:0/448173213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5854108ad0 0x7f5854108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.457+0000 7f5859592700 1 --2- 172.21.15.98:0/448173213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58541019d0 0x7f585410f910 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.457+0000 7f5859592700 1 --2- 172.21.15.98:0/448173213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5854101030 0x7f5854101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.457+0000 7f5859592700 1 -- 172.21.15.98:0/448173213 >> 172.21.15.98:0/448173213 conn(0x7f5854077fb0 msgr2=0x7f58540ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.457+0000 7f5859592700 1 -- 172.21.15.98:0/448173213 shutdown_connections 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.457+0000 7f5859592700 1 -- 172.21.15.98:0/448173213 wait complete. 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f5859592700 1 Processor -- start 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f5859592700 1 -- start start 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f5859592700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5854101030 0x7f5854102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f5859592700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58541019d0 0x7f5854102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f5859592700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5854108ad0 0x7f5854102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f5859592700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5854112c50 con 0x7f5854108ad0 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f5859592700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5854112ad0 con 0x7f5854101030 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f5859592700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5854112dd0 con 0x7f58541019d0 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f58537fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5854108ad0 0x7f5854102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f58537fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5854108ad0 0x7f5854102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39332/0 (socket says 172.21.15.98:39332) 2024-04-25T17:32:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f5852ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5854101030 0x7f5854102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f58537fe700 1 -- 172.21.15.98:0/1066950648 learned_addr learned my addr 172.21.15.98:0/1066950648 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f58527fc700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58541019d0 0x7f5854102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f58537fe700 1 -- 172.21.15.98:0/1066950648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58541019d0 msgr2=0x7f5854102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f58537fe700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58541019d0 0x7f5854102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f58537fe700 1 -- 172.21.15.98:0/1066950648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5854101030 msgr2=0x7f5854102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f58537fe700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5854101030 0x7f5854102210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f58537fe700 1 -- 172.21.15.98:0/1066950648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58541a7640 con 0x7f5854108ad0 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.458+0000 7f5852ffd700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5854101030 0x7f5854102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.459+0000 7f58537fe700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5854108ad0 0x7f5854102c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f584c009fd0 tx=0x7f584c004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.459+0000 7f583bfff700 1 -- 172.21.15.98:0/1066950648 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c021070 con 0x7f5854108ad0 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.459+0000 7f5859592700 1 -- 172.21.15.98:0/1066950648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58541a78d0 con 0x7f5854108ad0 2024-04-25T17:32:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.459+0000 7f583bfff700 1 -- 172.21.15.98:0/1066950648 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f584c007b80 con 0x7f5854108ad0 2024-04-25T17:32:03.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.459+0000 7f583bfff700 1 -- 172.21.15.98:0/1066950648 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c0052a0 con 0x7f5854108ad0 2024-04-25T17:32:03.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.459+0000 7f5859592700 1 -- 172.21.15.98:0/1066950648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58541a7dd0 con 0x7f5854108ad0 2024-04-25T17:32:03.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:03 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.203:6836/1057173981,v1:172.21.15.203:6837/1057173981] up:boot 2024-04-25T17:32:03.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:03 smithi203 ceph-mon[32834]: fsmap 7 up:standby 2024-04-25T17:32:03.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:32:03.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:03.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:03.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:03.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:03 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:03.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:03 smithi203 ceph-mon[32834]: Deploying daemon mds.h on smithi173 2024-04-25T17:32:03.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.460+0000 7f5859592700 1 -- 172.21.15.98:0/1066950648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5854069730 con 0x7f5854108ad0 2024-04-25T17:32:03.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.461+0000 7f583bfff700 1 -- 172.21.15.98:0/1066950648 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f584c024070 con 0x7f5854108ad0 2024-04-25T17:32:03.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.461+0000 7f583bfff700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f583c0a3cc0 0x7f583c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:03.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.462+0000 7f5852ffd700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f583c0a3cc0 0x7f583c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:03.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.462+0000 7f583bfff700 1 -- 172.21.15.98:0/1066950648 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f584c00f020 con 0x7f5854108ad0 2024-04-25T17:32:03.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.462+0000 7f5852ffd700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f583c0a3cc0 0x7f583c0a6180 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f58440103d0 tx=0x7f584400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:03.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.465+0000 7f583bfff700 1 -- 172.21.15.98:0/1066950648 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f584c02b030 con 0x7f5854108ad0 2024-04-25T17:32:03.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.705+0000 7f5859592700 1 -- 172.21.15.98:0/1066950648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f585402e140 con 0x7f5854108ad0 2024-04-25T17:32:03.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.706+0000 7f583bfff700 1 -- 172.21.15.98:0/1066950648 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7f584c09e4d0 con 0x7f5854108ad0 2024-04-25T17:32:03.707 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:32:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T17:32:03.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 -- 172.21.15.98:0/1066950648 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f583c0a3cc0 msgr2=0x7f583c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f583c0a3cc0 0x7f583c0a6180 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f58440103d0 tx=0x7f584400b3c0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 -- 172.21.15.98:0/1066950648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5854108ad0 msgr2=0x7f5854102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5854108ad0 0x7f5854102c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f584c009fd0 tx=0x7f584c004dc0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 -- 172.21.15.98:0/1066950648 shutdown_connections 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f583c0a3cc0 0x7f583c0a6180 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5854108ad0 0x7f5854102c90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5854101030 0x7f5854102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 --2- 172.21.15.98:0/1066950648 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58541019d0 0x7f5854102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 -- 172.21.15.98:0/1066950648 >> 172.21.15.98:0/1066950648 conn(0x7f5854077fb0 msgr2=0x7f58540ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 -- 172.21.15.98:0/1066950648 shutdown_connections 2024-04-25T17:32:03.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:03.708+0000 7f5859592700 1 -- 172.21.15.98:0/1066950648 wait complete. 2024-04-25T17:32:03.711 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-04-25T17:32:04.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:04 smithi098 ceph-mon[29192]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:04.404 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:04 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1066950648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:04 smithi203 ceph-mon[32834]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:04 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1066950648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:04.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:04 smithi173 ceph-mon[32856]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:04.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:04 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1066950648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:04.523 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:32:04 smithi173 systemd[1]: Starting Ceph mds.h for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:32:04.859 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:32:04 smithi173 podman[62813]: 2024-04-25 17:32:04.469128197 +0000 UTC m=+0.018099232 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:32:04.860 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:32:04 smithi173 podman[62813]: 2024-04-25 17:32:04.598328893 +0000 UTC m=+0.147299911 container create ab34c40f9d5861c64532b8f216cf5ed836600e762e42db09cf0381b3f6e2c4b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-25T17:32:04.860 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:32:04 smithi173 podman[62813]: 2024-04-25 17:32:04.859419719 +0000 UTC m=+0.408390740 container init ab34c40f9d5861c64532b8f216cf5ed836600e762e42db09cf0381b3f6e2c4b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-25T17:32:05.273 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:32:04 smithi173 podman[62813]: 2024-04-25 17:32:04.862119308 +0000 UTC m=+0.411090327 container start ab34c40f9d5861c64532b8f216cf5ed836600e762e42db09cf0381b3f6e2c4b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:32:05.273 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:32:04 smithi173 ceph-mds[62832]: -- 172.21.15.173:0/3752492339 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564753485a40 con 0x564753459000 2024-04-25T17:32:05.274 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:32:04 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h[62828]: starting mds.h at 2024-04-25T17:32:05.274 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:32:04 smithi173 bash[62813]: ab34c40f9d5861c64532b8f216cf5ed836600e762e42db09cf0381b3f6e2c4b8 2024-04-25T17:32:05.274 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:32:04 smithi173 systemd[1]: Started Ceph mds.h for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:32:06.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:06.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:06.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:06.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:06.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:06.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:06.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:06.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: Deploying daemon mds.i on smithi203 2024-04-25T17:32:06.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] up:boot 2024-04-25T17:32:06.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: fsmap 8 up:standby 2024-04-25T17:32:06.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:06 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T17:32:06.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:32:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:06.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:06.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: Deploying daemon mds.i on smithi203 2024-04-25T17:32:06.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] up:boot 2024-04-25T17:32:06.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: fsmap 8 up:standby 2024-04-25T17:32:06.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:06 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T17:32:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:06.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:06.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: Deploying daemon mds.i on smithi203 2024-04-25T17:32:06.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] up:boot 2024-04-25T17:32:06.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: fsmap 8 up:standby 2024-04-25T17:32:06.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:06 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T17:32:06.683 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:07.426 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:32:07 smithi203 systemd[1]: Starting Ceph mds.i for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:32:07.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:32:07 smithi203 podman[63856]: 2024-04-25 17:32:07.329012772 +0000 UTC m=+0.017996580 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:32:07.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:32:07 smithi203 podman[63856]: 2024-04-25 17:32:07.437195942 +0000 UTC m=+0.126179743 container create 8e4ed0f945d224be0531bc734f2839fefca641b1031ba0d1d04cb310be1c0b34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-04-25T17:32:07.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:32:07 smithi203 podman[63856]: 2024-04-25 17:32:07.688772531 +0000 UTC m=+0.377756329 container init 8e4ed0f945d224be0531bc734f2839fefca641b1031ba0d1d04cb310be1c0b34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T17:32:07.963 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:32:07 smithi203 podman[63856]: 2024-04-25 17:32:07.692017871 +0000 UTC m=+0.381001676 container start 8e4ed0f945d224be0531bc734f2839fefca641b1031ba0d1d04cb310be1c0b34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-25T17:32:07.963 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:32:07 smithi203 ceph-mds[63875]: -- 172.21.15.203:0/1010591809 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1d32bba40 con 0x55e1d328ec00 2024-04-25T17:32:07.964 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:32:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i[63871]: starting mds.i at 2024-04-25T17:32:07.964 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:32:07 smithi203 bash[63856]: 8e4ed0f945d224be0531bc734f2839fefca641b1031ba0d1d04cb310be1c0b34 2024-04-25T17:32:07.964 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:32:07 smithi203 systemd[1]: Started Ceph mds.i for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:32:08.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:08 smithi098 ceph-mon[29192]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:08.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:08.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:08.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:08.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:08.187 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:08.187 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:08 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.433+0000 7f4b74e98700 1 -- 172.21.15.98:0/665322500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b70107990 msgr2=0x7f4b70107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.433+0000 7f4b74e98700 1 --2- 172.21.15.98:0/665322500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b70107990 0x7f4b70107d70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800b0a0 tx=0x7f4b6800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:32:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.433+0000 7f4b74e98700 1 -- 172.21.15.98:0/665322500 shutdown_connections 2024-04-25T17:32:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.433+0000 7f4b74e98700 1 --2- 172.21.15.98:0/665322500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b70102300 0x7f4b7010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.433+0000 7f4b74e98700 1 --2- 172.21.15.98:0/665322500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b70107990 0x7f4b70107d70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.433+0000 7f4b74e98700 1 --2- 172.21.15.98:0/665322500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b70101940 0x7f4b70101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.433+0000 7f4b74e98700 1 -- 172.21.15.98:0/665322500 >> 172.21.15.98:0/665322500 conn(0x7f4b700788f0 msgr2=0x7f4b70078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.433+0000 7f4b74e98700 1 -- 172.21.15.98:0/665322500 shutdown_connections 2024-04-25T17:32:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.433+0000 7f4b74e98700 1 -- 172.21.15.98:0/665322500 wait complete. 2024-04-25T17:32:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b74e98700 1 Processor -- start 2024-04-25T17:32:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b74e98700 1 -- start start 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b74e98700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b70101940 0x7f4b701992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b74e98700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b70102300 0x7f4b70199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b74e98700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b70107990 0x7f4b7019db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b74e98700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70111bd0 con 0x7f4b70101940 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b74e98700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70111a50 con 0x7f4b70102300 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b74e98700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70111d50 con 0x7f4b70107990 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b6e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b70101940 0x7f4b701992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b6e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b70101940 0x7f4b701992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54924/0 (socket says 172.21.15.98:54924) 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b6ed9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b70107990 0x7f4b7019db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b6e59c700 1 -- 172.21.15.98:0/1426887892 learned_addr learned my addr 172.21.15.98:0/1426887892 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.434+0000 7f4b6dd9b700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b70102300 0x7f4b70199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b6e59c700 1 -- 172.21.15.98:0/1426887892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b70107990 msgr2=0x7f4b7019db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b6e59c700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b70107990 0x7f4b7019db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b6e59c700 1 -- 172.21.15.98:0/1426887892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b70102300 msgr2=0x7f4b70199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b6e59c700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b70102300 0x7f4b70199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b6e59c700 1 -- 172.21.15.98:0/1426887892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7019e2b0 con 0x7f4b70101940 2024-04-25T17:32:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b6e59c700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b70101940 0x7f4b701992f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006070 tx=0x7f4b68004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b5f7fe700 1 -- 172.21.15.98:0/1426887892 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68021070 con 0x7f4b70101940 2024-04-25T17:32:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b5f7fe700 1 -- 172.21.15.98:0/1426887892 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b68003c00 con 0x7f4b70101940 2024-04-25T17:32:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b5f7fe700 1 -- 172.21.15.98:0/1426887892 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b680047b0 con 0x7f4b70101940 2024-04-25T17:32:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b74e98700 1 -- 172.21.15.98:0/1426887892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7019e540 con 0x7f4b70101940 2024-04-25T17:32:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.435+0000 7f4b74e98700 1 -- 172.21.15.98:0/1426887892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b701a5e20 con 0x7f4b70101940 2024-04-25T17:32:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.436+0000 7f4b74e98700 1 -- 172.21.15.98:0/1426887892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b70069730 con 0x7f4b70101940 2024-04-25T17:32:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.437+0000 7f4b5f7fe700 1 -- 172.21.15.98:0/1426887892 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4b68007a00 con 0x7f4b70101940 2024-04-25T17:32:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.438+0000 7f4b5f7fe700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4b580a3c40 0x7f4b580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.438+0000 7f4b6dd9b700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4b580a3c40 0x7f4b580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.438+0000 7f4b5f7fe700 1 -- 172.21.15.98:0/1426887892 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4b6800f020 con 0x7f4b70101940 2024-04-25T17:32:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.438+0000 7f4b6dd9b700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4b580a3c40 0x7f4b580a6100 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4b70078f30 tx=0x7f4b64009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:08.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.442+0000 7f4b5f7fe700 1 -- 172.21.15.98:0/1426887892 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b6802a020 con 0x7f4b70101940 2024-04-25T17:32:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:08 smithi203 ceph-mon[32834]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:08.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:08 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:08 smithi173 ceph-mon[32856]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:08 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:08.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.693+0000 7f4b74e98700 1 -- 172.21.15.98:0/1426887892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b7019a600 con 0x7f4b70101940 2024-04-25T17:32:08.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.693+0000 7f4b5f7fe700 1 -- 172.21.15.98:0/1426887892 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7f4b7019a600 con 0x7f4b70101940 2024-04-25T17:32:08.695 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:32:08.695 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T17:32:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.695+0000 7f4b74e98700 1 -- 172.21.15.98:0/1426887892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4b580a3c40 msgr2=0x7f4b580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.695+0000 7f4b74e98700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4b580a3c40 0x7f4b580a6100 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4b70078f30 tx=0x7f4b64009190 comp rx=0 tx=0).stop 2024-04-25T17:32:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.695+0000 7f4b74e98700 1 -- 172.21.15.98:0/1426887892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b70101940 msgr2=0x7f4b701992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.695+0000 7f4b74e98700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b70101940 0x7f4b701992f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006070 tx=0x7f4b68004e40 comp rx=0 tx=0).stop 2024-04-25T17:32:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.696+0000 7f4b74e98700 1 -- 172.21.15.98:0/1426887892 shutdown_connections 2024-04-25T17:32:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.696+0000 7f4b74e98700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f4b580a3c40 0x7f4b580a6100 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.696+0000 7f4b74e98700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b70107990 0x7f4b7019db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.696+0000 7f4b74e98700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b70101940 0x7f4b701992f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.696+0000 7f4b74e98700 1 --2- 172.21.15.98:0/1426887892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b70102300 0x7f4b70199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.696+0000 7f4b74e98700 1 -- 172.21.15.98:0/1426887892 >> 172.21.15.98:0/1426887892 conn(0x7f4b700788f0 msgr2=0x7f4b7006a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.696+0000 7f4b74e98700 1 -- 172.21.15.98:0/1426887892 shutdown_connections 2024-04-25T17:32:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:08.696+0000 7f4b74e98700 1 -- 172.21.15.98:0/1426887892 wait complete. 2024-04-25T17:32:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 10 2024-04-25T17:32:09.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:09 smithi098 ceph-mon[29192]: Deploying daemon mds.g on smithi098 2024-04-25T17:32:09.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:09 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] up:boot 2024-04-25T17:32:09.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:09 smithi098 ceph-mon[29192]: fsmap 9 up:standby 2024-04-25T17:32:09.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T17:32:09.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:09 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1426887892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:09.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:09 smithi203 ceph-mon[32834]: Deploying daemon mds.g on smithi098 2024-04-25T17:32:09.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:09 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] up:boot 2024-04-25T17:32:09.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:09 smithi203 ceph-mon[32834]: fsmap 9 up:standby 2024-04-25T17:32:09.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T17:32:09.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:09 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1426887892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:09 smithi173 ceph-mon[32856]: Deploying daemon mds.g on smithi098 2024-04-25T17:32:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:09 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] up:boot 2024-04-25T17:32:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:09 smithi173 ceph-mon[32856]: fsmap 9 up:standby 2024-04-25T17:32:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T17:32:09.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:09 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1426887892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:10 smithi203 ceph-mon[32834]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:10.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:10 smithi098 ceph-mon[29192]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:10 smithi173 ceph-mon[32856]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:11.206 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:32:10 smithi098 systemd[1]: Starting Ceph mds.g for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:32:11.498 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:32:11 smithi098 podman[71408]: 2024-04-25 17:32:11.117160481 +0000 UTC m=+0.017944774 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:32:11.498 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:32:11 smithi098 podman[71408]: 2024-04-25 17:32:11.242656005 +0000 UTC m=+0.143440298 container create b13d255fce754143c699aa75e47599d7501a19ef036a62c8436f6103a29c6948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:32:11.498 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:32:11 smithi098 podman[71408]: 2024-04-25 17:32:11.497670764 +0000 UTC m=+0.398455062 container init b13d255fce754143c699aa75e47599d7501a19ef036a62c8436f6103a29c6948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T17:32:11.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:32:11.825 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:11.948 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:32:11 smithi098 podman[71408]: 2024-04-25 17:32:11.50022894 +0000 UTC m=+0.401013236 container start b13d255fce754143c699aa75e47599d7501a19ef036a62c8436f6103a29c6948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-g, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:32:11.948 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:32:11 smithi098 ceph-mds[71427]: -- 172.21.15.98:0/3641732780 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ee88033a40 con 0x55ee88007000 2024-04-25T17:32:11.948 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:32:11 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-g[71423]: starting mds.g at 2024-04-25T17:32:11.948 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:32:11 smithi098 bash[71408]: b13d255fce754143c699aa75e47599d7501a19ef036a62c8436f6103a29c6948 2024-04-25T17:32:11.948 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:32:11 smithi098 systemd[1]: Started Ceph mds.g for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:32:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:12 smithi098 ceph-mon[29192]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:12.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:12 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:12 smithi203 ceph-mon[32834]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:12.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:12 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:12.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:12 smithi173 ceph-mon[32856]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:12.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:12.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:12.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:12.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:12.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:12.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:12 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:13.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:13 smithi098 ceph-mon[29192]: Deploying daemon mds.a on smithi098 2024-04-25T17:32:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:13 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.98:6838/672802539,v1:172.21.15.98:6839/672802539] up:boot 2024-04-25T17:32:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:13 smithi098 ceph-mon[29192]: fsmap 10 up:standby 2024-04-25T17:32:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:13 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T17:32:13.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:13 smithi203 ceph-mon[32834]: Deploying daemon mds.a on smithi098 2024-04-25T17:32:13.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:13 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.98:6838/672802539,v1:172.21.15.98:6839/672802539] up:boot 2024-04-25T17:32:13.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:13 smithi203 ceph-mon[32834]: fsmap 10 up:standby 2024-04-25T17:32:13.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:13 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T17:32:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.486+0000 7f153b256700 1 -- 172.21.15.98:0/572961382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1534102ed0 msgr2=0x7f153410e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.486+0000 7f153b256700 1 --2- 172.21.15.98:0/572961382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1534102ed0 0x7f153410e8e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f152c00b0d0 tx=0x7f152c00b3e0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.486+0000 7f153b256700 1 -- 172.21.15.98:0/572961382 shutdown_connections 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.486+0000 7f153b256700 1 --2- 172.21.15.98:0/572961382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1534107c30 0x7f1534108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.486+0000 7f153b256700 1 --2- 172.21.15.98:0/572961382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1534102ed0 0x7f153410e8e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.486+0000 7f153b256700 1 --2- 172.21.15.98:0/572961382 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1534102510 0x7f1534102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.486+0000 7f153b256700 1 -- 172.21.15.98:0/572961382 >> 172.21.15.98:0/572961382 conn(0x7f1534076c60 msgr2=0x7f1534077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.486+0000 7f153b256700 1 -- 172.21.15.98:0/572961382 shutdown_connections 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.486+0000 7f153b256700 1 -- 172.21.15.98:0/572961382 wait complete. 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f153b256700 1 Processor -- start 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f153b256700 1 -- start start 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f153b256700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1534102510 0x7f1534198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f153b256700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1534102ed0 0x7f1534198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f153b256700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1534107c30 0x7f153419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f153b256700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1534111cd0 con 0x7f1534102ed0 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f153b256700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1534111b50 con 0x7f1534107c30 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f153b256700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1534111e50 con 0x7f1534102510 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f1533fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1534102ed0 0x7f1534198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f1533fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1534102ed0 0x7f1534198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54522/0 (socket says 172.21.15.98:54522) 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f1533fff700 1 -- 172.21.15.98:0/1875601276 learned_addr learned my addr 172.21.15.98:0/1875601276 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.487+0000 7f1538ff2700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1534102510 0x7f1534198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f15397f3700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1534107c30 0x7f153419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f1533fff700 1 -- 172.21.15.98:0/1875601276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1534102510 msgr2=0x7f1534198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f1533fff700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1534102510 0x7f1534198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f1533fff700 1 -- 172.21.15.98:0/1875601276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1534107c30 msgr2=0x7f153419c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f1533fff700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1534107c30 0x7f153419c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f1533fff700 1 -- 172.21.15.98:0/1875601276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153419d0c0 con 0x7f1534102ed0 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f1538ff2700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1534102510 0x7f1534198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:32:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f15397f3700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1534107c30 0x7f153419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:32:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f1533fff700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1534102ed0 0x7f1534198640 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f152400b530 tx=0x7f152400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f1531ffb700 1 -- 172.21.15.98:0/1875601276 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524015020 con 0x7f1534102ed0 2024-04-25T17:32:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f1531ffb700 1 -- 172.21.15.98:0/1875601276 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1524011ce0 con 0x7f1534102ed0 2024-04-25T17:32:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f1531ffb700 1 -- 172.21.15.98:0/1875601276 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524010d20 con 0x7f1534102ed0 2024-04-25T17:32:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f153b256700 1 -- 172.21.15.98:0/1875601276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153419d3b0 con 0x7f1534102ed0 2024-04-25T17:32:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.488+0000 7f153b256700 1 -- 172.21.15.98:0/1875601276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15341a4c30 con 0x7f1534102ed0 2024-04-25T17:32:13.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.489+0000 7f153b256700 1 -- 172.21.15.98:0/1875601276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1534069730 con 0x7f1534102ed0 2024-04-25T17:32:13.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.490+0000 7f1531ffb700 1 -- 172.21.15.98:0/1875601276 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1524018020 con 0x7f1534102ed0 2024-04-25T17:32:13.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.491+0000 7f1531ffb700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15200a3cc0 0x7f15200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:13.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.491+0000 7f1538ff2700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15200a3cc0 0x7f15200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:13.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.491+0000 7f1531ffb700 1 -- 172.21.15.98:0/1875601276 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f15240cf740 con 0x7f1534102ed0 2024-04-25T17:32:13.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.491+0000 7f1538ff2700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15200a3cc0 0x7f15200a6180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f151c005950 tx=0x7f151c005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.495+0000 7f1531ffb700 1 -- 172.21.15.98:0/1875601276 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f152401f020 con 0x7f1534102ed0 2024-04-25T17:32:13.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:13 smithi173 ceph-mon[32856]: Deploying daemon mds.a on smithi098 2024-04-25T17:32:13.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:13 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.98:6838/672802539,v1:172.21.15.98:6839/672802539] up:boot 2024-04-25T17:32:13.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:13 smithi173 ceph-mon[32856]: fsmap 10 up:standby 2024-04-25T17:32:13.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:13 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T17:32:13.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.747+0000 7f153b256700 1 -- 172.21.15.98:0/1875601276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1534199410 con 0x7f1534102ed0 2024-04-25T17:32:13.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.747+0000 7f1531ffb700 1 -- 172.21.15.98:0/1875601276 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7f152409c5c0 con 0x7f1534102ed0 2024-04-25T17:32:13.749 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:32:13.749 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T17:32:13.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 -- 172.21.15.98:0/1875601276 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15200a3cc0 msgr2=0x7f15200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15200a3cc0 0x7f15200a6180 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f151c005950 tx=0x7f151c005f30 comp rx=0 tx=0).stop 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 -- 172.21.15.98:0/1875601276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1534102ed0 msgr2=0x7f1534198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1534102ed0 0x7f1534198640 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f152400b530 tx=0x7f152400b8f0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 -- 172.21.15.98:0/1875601276 shutdown_connections 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15200a3cc0 0x7f15200a6180 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1534102510 0x7f1534198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1534107c30 0x7f153419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 --2- 172.21.15.98:0/1875601276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1534102ed0 0x7f1534198640 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 -- 172.21.15.98:0/1875601276 >> 172.21.15.98:0/1875601276 conn(0x7f1534076c60 msgr2=0x7f15340ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 -- 172.21.15.98:0/1875601276 shutdown_connections 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:13.750+0000 7f153b256700 1 -- 172.21.15.98:0/1875601276 wait complete. 2024-04-25T17:32:13.752 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 11 2024-04-25T17:32:14.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:14 smithi098 ceph-mon[29192]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:14.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:14 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1875601276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:14 smithi203 ceph-mon[32834]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:14 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1875601276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:14.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:14 smithi173 ceph-mon[32856]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:14.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:14 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1875601276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:16.049 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:32:16 smithi098 podman[71878]: 2024-04-25 17:32:15.926863639 +0000 UTC m=+0.017810814 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:32:16.050 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:32:16 smithi098 podman[71878]: 2024-04-25 17:32:16.048886478 +0000 UTC m=+0.139833639 container create f9496264578803285d00b0e04d960c1249a81b106a55671e0cc85581006131b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:32:16.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:16 smithi098 ceph-mon[29192]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:16.320 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:32:16 smithi098 podman[71878]: 2024-04-25 17:32:16.29349461 +0000 UTC m=+0.384441768 container init f9496264578803285d00b0e04d960c1249a81b106a55671e0cc85581006131b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-25T17:32:16.320 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:32:16 smithi098 podman[71878]: 2024-04-25 17:32:16.296059735 +0000 UTC m=+0.387006894 container start f9496264578803285d00b0e04d960c1249a81b106a55671e0cc85581006131b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T17:32:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:16 smithi203 ceph-mon[32834]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:16.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:16 smithi173 ceph-mon[32856]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:16.706 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:32:16 smithi098 ceph-mds[71897]: -- 172.21.15.98:0/2442407040 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d67ac71a40 con 0x55d67ac45000 2024-04-25T17:32:16.706 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:32:16 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-a[71893]: starting mds.a at 2024-04-25T17:32:16.706 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:32:16 smithi098 bash[71878]: f9496264578803285d00b0e04d960c1249a81b106a55671e0cc85581006131b5 2024-04-25T17:32:16.706 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:32:16 smithi098 systemd[1]: Started Ceph mds.a for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:32:16.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:32:17.112 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:17.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:17.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:17.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:17.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:17.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:17.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:17.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: Deploying daemon mds.b on smithi173 2024-04-25T17:32:17.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:17.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.98:6840/121413521,v1:172.21.15.98:6841/121413521] up:boot 2024-04-25T17:32:17.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: fsmap 11 up:standby 2024-04-25T17:32:17.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:17 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T17:32:17.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:17.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:17.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:17.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:17.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:17.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:17.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: Deploying daemon mds.b on smithi173 2024-04-25T17:32:17.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:17.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.98:6840/121413521,v1:172.21.15.98:6841/121413521] up:boot 2024-04-25T17:32:17.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: fsmap 11 up:standby 2024-04-25T17:32:17.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:17 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T17:32:17.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:17.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:17.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:17.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T17:32:17.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T17:32:17.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:17.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: Deploying daemon mds.b on smithi173 2024-04-25T17:32:17.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:17.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.98:6840/121413521,v1:172.21.15.98:6841/121413521] up:boot 2024-04-25T17:32:17.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: fsmap 11 up:standby 2024-04-25T17:32:17.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:17 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.865+0000 7fe52d9f3700 1 -- 172.21.15.98:0/1774885521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe528108a60 msgr2=0x7fe528108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.865+0000 7fe52d9f3700 1 --2- 172.21.15.98:0/1774885521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe528108a60 0x7fe528108e40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe518009990 tx=0x7fe518009ca0 comp rx=0 tx=0).stop 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.865+0000 7fe52d9f3700 1 -- 172.21.15.98:0/1774885521 shutdown_connections 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.865+0000 7fe52d9f3700 1 --2- 172.21.15.98:0/1774885521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52806bae0 0x7fe52810f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.865+0000 7fe52d9f3700 1 --2- 172.21.15.98:0/1774885521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe528108a60 0x7fe528108e40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.865+0000 7fe52d9f3700 1 --2- 172.21.15.98:0/1774885521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe52806b140 0x7fe52806b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.865+0000 7fe52d9f3700 1 -- 172.21.15.98:0/1774885521 >> 172.21.15.98:0/1774885521 conn(0x7fe528078980 msgr2=0x7fe528078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.865+0000 7fe52d9f3700 1 -- 172.21.15.98:0/1774885521 shutdown_connections 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.865+0000 7fe52d9f3700 1 -- 172.21.15.98:0/1774885521 wait complete. 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.865+0000 7fe52d9f3700 1 Processor -- start 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe52d9f3700 1 -- start start 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe52d9f3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe52806b140 0x7fe528199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:18.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe52d9f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe52806bae0 0x7fe528199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe52d9f3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528108a60 0x7fe52819db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe52d9f3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe528112ce0 con 0x7fe52806bae0 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe52d9f3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe528112b60 con 0x7fe52806b140 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe52d9f3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe528112e60 con 0x7fe528108a60 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe51e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe52806bae0 0x7fe528199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe51e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe52806bae0 0x7fe528199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54554/0 (socket says 172.21.15.98:54554) 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe51e7fc700 1 -- 172.21.15.98:0/3099449774 learned_addr learned my addr 172.21.15.98:0/3099449774 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe526ffd700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe52806b140 0x7fe528199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe5277fe700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528108a60 0x7fe52819db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe51e7fc700 1 -- 172.21.15.98:0/3099449774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528108a60 msgr2=0x7fe52819db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe51e7fc700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528108a60 0x7fe52819db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe51e7fc700 1 -- 172.21.15.98:0/3099449774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe52806b140 msgr2=0x7fe528199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe51e7fc700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe52806b140 0x7fe528199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:18.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.866+0000 7fe51e7fc700 1 -- 172.21.15.98:0/3099449774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52819e260 con 0x7fe52806bae0 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.867+0000 7fe51e7fc700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe52806bae0 0x7fe528199790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe51000b530 tx=0x7fe51000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.867+0000 7fe52c9f1700 1 -- 172.21.15.98:0/3099449774 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510015020 con 0x7fe52806bae0 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.867+0000 7fe52d9f3700 1 -- 172.21.15.98:0/3099449774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe52819e550 con 0x7fe52806bae0 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.867+0000 7fe52c9f1700 1 -- 172.21.15.98:0/3099449774 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe510011ce0 con 0x7fe52806bae0 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.867+0000 7fe52c9f1700 1 -- 172.21.15.98:0/3099449774 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510010e40 con 0x7fe52806bae0 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.867+0000 7fe52d9f3700 1 -- 172.21.15.98:0/3099449774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5281a5dd0 con 0x7fe52806bae0 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.868+0000 7fe52d9f3700 1 -- 172.21.15.98:0/3099449774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5281041a0 con 0x7fe52806bae0 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.869+0000 7fe52c9f1700 1 -- 172.21.15.98:0/3099449774 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe510018020 con 0x7fe52806bae0 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.869+0000 7fe52c9f1700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe5140a3c40 0x7fe5140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.870+0000 7fe526ffd700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe5140a3c40 0x7fe5140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.870+0000 7fe52c9f1700 1 -- 172.21.15.98:0/3099449774 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe5100cfd30 con 0x7fe52806bae0 2024-04-25T17:32:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.870+0000 7fe526ffd700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe5140a3c40 0x7fe5140a6100 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe51800b5c0 tx=0x7fe518005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:18.873+0000 7fe52c9f1700 1 -- 172.21.15.98:0/3099449774 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe51001f020 con 0x7fe52806bae0 2024-04-25T17:32:19.062 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:32:18 smithi173 podman[63120]: 2024-04-25 17:32:18.899272367 +0000 UTC m=+0.018177213 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:32:19.063 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:32:19 smithi173 podman[63120]: 2024-04-25 17:32:19.061775033 +0000 UTC m=+0.180679846 container create d433fc0d571006ea16003765d341c2e9833b29a4c7cbc2a59c844e0be79d288a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:32:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.120+0000 7fe52d9f3700 1 -- 172.21.15.98:0/3099449774 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe52802df80 con 0x7fe52806bae0 2024-04-25T17:32:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.121+0000 7fe52c9f1700 1 -- 172.21.15.98:0/3099449774 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9659 (secure 0 0 0) 0x7fe51009cbb0 con 0x7fe52806bae0 2024-04-25T17:32:19.122 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:32:19.122 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T17:32:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 -- 172.21.15.98:0/3099449774 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe5140a3c40 msgr2=0x7fe5140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe5140a3c40 0x7fe5140a6100 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe51800b5c0 tx=0x7fe518005960 comp rx=0 tx=0).stop 2024-04-25T17:32:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 -- 172.21.15.98:0/3099449774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe52806bae0 msgr2=0x7fe528199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe52806bae0 0x7fe528199790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe51000b530 tx=0x7fe51000b840 comp rx=0 tx=0).stop 2024-04-25T17:32:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 -- 172.21.15.98:0/3099449774 shutdown_connections 2024-04-25T17:32:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe5140a3c40 0x7fe5140a6100 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe528108a60 0x7fe52819db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe52806b140 0x7fe528199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:19.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 --2- 172.21.15.98:0/3099449774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe52806bae0 0x7fe528199790 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 -- 172.21.15.98:0/3099449774 >> 172.21.15.98:0/3099449774 conn(0x7fe528078980 msgr2=0x7fe5280ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 -- 172.21.15.98:0/3099449774 shutdown_connections 2024-04-25T17:32:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:19.123+0000 7fe52d9f3700 1 -- 172.21.15.98:0/3099449774 wait complete. 2024-04-25T17:32:19.125 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 12 2024-04-25T17:32:19.776 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:32:19 smithi173 podman[63120]: 2024-04-25 17:32:19.314709693 +0000 UTC m=+0.433614507 container init d433fc0d571006ea16003765d341c2e9833b29a4c7cbc2a59c844e0be79d288a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:32:19.777 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:32:19 smithi173 podman[63120]: 2024-04-25 17:32:19.317396695 +0000 UTC m=+0.436301519 container start d433fc0d571006ea16003765d341c2e9833b29a4c7cbc2a59c844e0be79d288a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:32:19.777 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:32:19 smithi173 ceph-mds[63139]: -- 172.21.15.173:0/1303997186 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d269123a40 con 0x55d2690f7000 2024-04-25T17:32:19.777 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:32:19 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b[63135]: starting mds.b at 2024-04-25T17:32:19.777 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:32:19 smithi173 bash[63120]: d433fc0d571006ea16003765d341c2e9833b29a4c7cbc2a59c844e0be79d288a 2024-04-25T17:32:19.777 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:32:19 smithi173 systemd[1]: Started Ceph mds.b for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:32:20.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:20 smithi098 ceph-mon[29192]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:20 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3099449774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:20 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:32:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:20 smithi173 ceph-mon[32856]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:20 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3099449774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:20 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:32:20.417 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:20 smithi203 ceph-mon[32834]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:20.418 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:20 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3099449774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:20.418 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.418 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.418 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.418 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.418 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:20.418 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:20 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:32:21.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:21 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] up:boot 2024-04-25T17:32:21.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:21 smithi173 ceph-mon[32856]: fsmap 12 up:standby 2024-04-25T17:32:21.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:21 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T17:32:21.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:21 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] up:boot 2024-04-25T17:32:21.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:21 smithi098 ceph-mon[29192]: fsmap 12 up:standby 2024-04-25T17:32:21.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:21 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T17:32:21.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:21 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] up:boot 2024-04-25T17:32:21.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:21 smithi203 ceph-mon[32834]: fsmap 12 up:standby 2024-04-25T17:32:21.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:21 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T17:32:21.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:32:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:22.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:22.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:22.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:32:22.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:22 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:32:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:32:22.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:22 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:32:22.402 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:22.402 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.402 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.402 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.402 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.402 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.403 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.403 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:22.403 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:32:22.403 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:22.403 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:22 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:32:24.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.022+0000 7f044a0ca700 1 -- 172.21.15.98:0/3555010677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0444101970 msgr2=0x7f0444101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:24.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.022+0000 7f044a0ca700 1 --2- 172.21.15.98:0/3555010677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0444101970 0x7f0444101df0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f042c009990 tx=0x7f042c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.022+0000 7f044a0ca700 1 -- 172.21.15.98:0/3555010677 shutdown_connections 2024-04-25T17:32:24.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.022+0000 7f044a0ca700 1 --2- 172.21.15.98:0/3555010677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444102330 0x7f044410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.022+0000 7f044a0ca700 1 --2- 172.21.15.98:0/3555010677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444107920 0x7f0444107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.022+0000 7f044a0ca700 1 --2- 172.21.15.98:0/3555010677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0444101970 0x7f0444101df0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.022+0000 7f044a0ca700 1 -- 172.21.15.98:0/3555010677 >> 172.21.15.98:0/3555010677 conn(0x7f0444078910 msgr2=0x7f0444078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.022+0000 7f044a0ca700 1 -- 172.21.15.98:0/3555010677 shutdown_connections 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f044a0ca700 1 -- 172.21.15.98:0/3555010677 wait complete. 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f044a0ca700 1 Processor -- start 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f044a0ca700 1 -- start start 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f044a0ca700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444101970 0x7f0444198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f044a0ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0444102330 0x7f0444198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f044a0ca700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444107920 0x7f044419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f044a0ca700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0444111b60 con 0x7f0444102330 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f044a0ca700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f04441119e0 con 0x7f0444101970 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f044a0ca700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0444111ce0 con 0x7f0444107920 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f043bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0444102330 0x7f0444198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.023+0000 7f043bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0444102330 0x7f0444198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53808/0 (socket says 172.21.15.98:53808) 2024-04-25T17:32:24.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f043bfff700 1 -- 172.21.15.98:0/3676773207 learned_addr learned my addr 172.21.15.98:0/3676773207 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f04437fe700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444101970 0x7f0444198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f0443fff700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444107920 0x7f044419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f043bfff700 1 -- 172.21.15.98:0/3676773207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444107920 msgr2=0x7f044419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f043bfff700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444107920 0x7f044419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f043bfff700 1 -- 172.21.15.98:0/3676773207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444101970 msgr2=0x7f0444198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f043bfff700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444101970 0x7f0444198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f043bfff700 1 -- 172.21.15.98:0/3676773207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044419d040 con 0x7f0444102330 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f043bfff700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0444102330 0x7f0444198570 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f042c000c00 tx=0x7f042c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f04417fa700 1 -- 172.21.15.98:0/3676773207 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c027070 con 0x7f0444102330 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f04417fa700 1 -- 172.21.15.98:0/3676773207 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f042c003f20 con 0x7f0444102330 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f044a0ca700 1 -- 172.21.15.98:0/3676773207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044419d2d0 con 0x7f0444102330 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f04417fa700 1 -- 172.21.15.98:0/3676773207 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c022ca0 con 0x7f0444102330 2024-04-25T17:32:24.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.024+0000 7f044a0ca700 1 -- 172.21.15.98:0/3676773207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04441a4bb0 con 0x7f0444102330 2024-04-25T17:32:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.025+0000 7f044a0ca700 1 -- 172.21.15.98:0/3676773207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0444069730 con 0x7f0444102330 2024-04-25T17:32:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.026+0000 7f04417fa700 1 -- 172.21.15.98:0/3676773207 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f042c02a070 con 0x7f0444102330 2024-04-25T17:32:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.027+0000 7f04417fa700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f04300a3cc0 0x7f04300a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.027+0000 7f04437fe700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f04300a3cc0 0x7f04300a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.027+0000 7f04417fa700 1 -- 172.21.15.98:0/3676773207 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f042c0d1410 con 0x7f0444102330 2024-04-25T17:32:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.027+0000 7f04437fe700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f04300a3cc0 0x7f04300a6180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0434005950 tx=0x7f043400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.030+0000 7f04417fa700 1 -- 172.21.15.98:0/3676773207 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f042c031070 con 0x7f0444102330 2024-04-25T17:32:24.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:24 smithi098 ceph-mon[29192]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:24.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:24 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.281+0000 7f044a0ca700 1 -- 172.21.15.98:0/3676773207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0444199390 con 0x7f0444102330 2024-04-25T17:32:24.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.281+0000 7f04417fa700 1 -- 172.21.15.98:0/3676773207 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10488 (secure 0 0 0) 0x7f042c013070 con 0x7f0444102330 2024-04-25T17:32:24.283 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:32:24.283 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24440,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24445,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 -- 172.21.15.98:0/3676773207 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f04300a3cc0 msgr2=0x7f04300a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f04300a3cc0 0x7f04300a6180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0434005950 tx=0x7f043400b410 comp rx=0 tx=0).stop 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 -- 172.21.15.98:0/3676773207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0444102330 msgr2=0x7f0444198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0444102330 0x7f0444198570 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f042c000c00 tx=0x7f042c0047b0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 -- 172.21.15.98:0/3676773207 shutdown_connections 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f04300a3cc0 0x7f04300a6180 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0444107920 0x7f044419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0444101970 0x7f0444198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 --2- 172.21.15.98:0/3676773207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0444102330 0x7f0444198570 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 -- 172.21.15.98:0/3676773207 >> 172.21.15.98:0/3676773207 conn(0x7f0444078910 msgr2=0x7f04440ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 -- 172.21.15.98:0/3676773207 shutdown_connections 2024-04-25T17:32:24.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:24.284+0000 7f044a0ca700 1 -- 172.21.15.98:0/3676773207 wait complete. 2024-04-25T17:32:24.286 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-04-25T17:32:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:24 smithi203 ceph-mon[32834]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:24 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:24 smithi173 ceph-mon[32856]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:24 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:25.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-04-25T17:32:25.406 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:25.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:25 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3676773207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:25.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:25 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3676773207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:25.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:25 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3676773207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:32:26.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:26 smithi098 ceph-mon[29192]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:26.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:26 smithi203 ceph-mon[32834]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:26.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:26 smithi173 ceph-mon[32856]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:27.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.116+0000 7faa84277700 1 -- 172.21.15.98:0/80423001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c101da0 msgr2=0x7faa7c10df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:27.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.116+0000 7faa84277700 1 --2- 172.21.15.98:0/80423001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c101da0 0x7faa7c10df70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7faa7400b0d0 tx=0x7faa7400b3e0 comp rx=0 tx=0).stop 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.116+0000 7faa84277700 1 -- 172.21.15.98:0/80423001 shutdown_connections 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.116+0000 7faa84277700 1 --2- 172.21.15.98:0/80423001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa7c1071c0 0x7faa7c1075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.116+0000 7faa84277700 1 --2- 172.21.15.98:0/80423001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c101da0 0x7faa7c10df70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.116+0000 7faa84277700 1 --2- 172.21.15.98:0/80423001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7c1013e0 0x7faa7c101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.116+0000 7faa84277700 1 -- 172.21.15.98:0/80423001 >> 172.21.15.98:0/80423001 conn(0x7faa7c076ce0 msgr2=0x7faa7c0770f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.116+0000 7faa84277700 1 -- 172.21.15.98:0/80423001 shutdown_connections 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.116+0000 7faa84277700 1 -- 172.21.15.98:0/80423001 wait complete. 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa84277700 1 Processor -- start 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa84277700 1 -- start start 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa84277700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa7c1013e0 0x7faa7c198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa84277700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7c101da0 0x7faa7c199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa84277700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c1071c0 0x7faa7c19d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa84277700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c1109f0 con 0x7faa7c1071c0 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa84277700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c110870 con 0x7faa7c101da0 2024-04-25T17:32:27.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa84277700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c110b70 con 0x7faa7c1013e0 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa82814700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c1071c0 0x7faa7c19d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa82814700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c1071c0 0x7faa7c19d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53834/0 (socket says 172.21.15.98:53834) 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa82814700 1 -- 172.21.15.98:0/4017002821 learned_addr learned my addr 172.21.15.98:0/4017002821 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa82013700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa7c1013e0 0x7faa7c198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.117+0000 7faa81812700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7c101da0 0x7faa7c199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa82814700 1 -- 172.21.15.98:0/4017002821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa7c1013e0 msgr2=0x7faa7c198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa82814700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa7c1013e0 0x7faa7c198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa82814700 1 -- 172.21.15.98:0/4017002821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7c101da0 msgr2=0x7faa7c199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa82814700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7c101da0 0x7faa7c199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa82814700 1 -- 172.21.15.98:0/4017002821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7c19dfe0 con 0x7faa7c1071c0 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa81812700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7c101da0 0x7faa7c199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa82814700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c1071c0 0x7faa7c19d8a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa74009850 tx=0x7faa74007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa737fe700 1 -- 172.21.15.98:0/4017002821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7401f070 con 0x7faa7c1071c0 2024-04-25T17:32:27.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa737fe700 1 -- 172.21.15.98:0/4017002821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa74004c50 con 0x7faa7c1071c0 2024-04-25T17:32:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa84277700 1 -- 172.21.15.98:0/4017002821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa7c19e270 con 0x7faa7c1071c0 2024-04-25T17:32:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa737fe700 1 -- 172.21.15.98:0/4017002821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7401b7b0 con 0x7faa7c1071c0 2024-04-25T17:32:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.118+0000 7faa84277700 1 -- 172.21.15.98:0/4017002821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa7c1a5b50 con 0x7faa7c1071c0 2024-04-25T17:32:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.119+0000 7faa84277700 1 -- 172.21.15.98:0/4017002821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa7c1027e0 con 0x7faa7c1071c0 2024-04-25T17:32:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.120+0000 7faa737fe700 1 -- 172.21.15.98:0/4017002821 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faa74022070 con 0x7faa7c1071c0 2024-04-25T17:32:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.121+0000 7faa737fe700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa680a3c40 0x7faa680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.121+0000 7faa82013700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa680a3c40 0x7faa680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.121+0000 7faa737fe700 1 -- 172.21.15.98:0/4017002821 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7faa7400f020 con 0x7faa7c1071c0 2024-04-25T17:32:27.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.121+0000 7faa82013700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa680a3c40 0x7faa680a6100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faa7800ac50 tx=0x7faa78010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:27.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.126+0000 7faa737fe700 1 -- 172.21.15.98:0/4017002821 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa74029020 con 0x7faa7c1071c0 2024-04-25T17:32:27.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.335+0000 7faa84277700 1 -- 172.21.15.98:0/4017002821 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7faa7c19a330 con 0x7faa7c1071c0 2024-04-25T17:32:27.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.335+0000 7faa737fe700 1 -- 172.21.15.98:0/4017002821 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7faa740233b0 con 0x7faa7c1071c0 2024-04-25T17:32:27.337 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:32:27.337 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2024-04-25T17:32:27.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:32:27.337 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2024-04-25T17:32:27.337 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2024-04-25T17:32:27.337 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:32:27.337 INFO:teuthology.orchestra.run.smithi098.stdout:] 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.337+0000 7faa84277700 1 -- 172.21.15.98:0/4017002821 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa680a3c40 msgr2=0x7faa680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.337+0000 7faa84277700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa680a3c40 0x7faa680a6100 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faa7800ac50 tx=0x7faa78010040 comp rx=0 tx=0).stop 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.337+0000 7faa84277700 1 -- 172.21.15.98:0/4017002821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c1071c0 msgr2=0x7faa7c19d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.337+0000 7faa84277700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c1071c0 0x7faa7c19d8a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faa74009850 tx=0x7faa74007a20 comp rx=0 tx=0).stop 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.337+0000 7faa84277700 1 -- 172.21.15.98:0/4017002821 shutdown_connections 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.337+0000 7faa84277700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7faa680a3c40 0x7faa680a6100 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.337+0000 7faa84277700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa7c1013e0 0x7faa7c198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.337+0000 7faa84277700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa7c1071c0 0x7faa7c19d8a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.337+0000 7faa84277700 1 --2- 172.21.15.98:0/4017002821 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7c101da0 0x7faa7c199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.337+0000 7faa84277700 1 -- 172.21.15.98:0/4017002821 >> 172.21.15.98:0/4017002821 conn(0x7faa7c076ce0 msgr2=0x7faa7c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.338+0000 7faa84277700 1 -- 172.21.15.98:0/4017002821 shutdown_connections 2024-04-25T17:32:27.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:27.338+0000 7faa84277700 1 -- 172.21.15.98:0/4017002821 wait complete. 2024-04-25T17:32:27.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:27.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:27.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:27 smithi098 ceph-mon[29192]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:27.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:27.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:32:27.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:27 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:27.957 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:27 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4017002821' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T17:32:27.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:27.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:27.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:27 smithi203 ceph-mon[32834]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:27.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:27.963 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:32:27.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:27 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:27.964 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:27 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/4017002821' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T17:32:28.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:28.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:28.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:27 smithi173 ceph-mon[32856]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:28.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:32:28.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:32:28.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:27 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:32:28.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:27 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/4017002821' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T17:32:28.036 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T17:32:28.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T17:32:28.277 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:30.057 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:30 smithi098 ceph-mon[29192]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:30.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.056+0000 7f6ff1e44700 1 -- 172.21.15.98:0/2264658792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fec108a30 msgr2=0x7f6fec108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:30.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.056+0000 7f6ff1e44700 1 --2- 172.21.15.98:0/2264658792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fec108a30 0x7f6fec108e10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009990 tx=0x7f6fe0009ca0 comp rx=0 tx=0).stop 2024-04-25T17:32:30.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.056+0000 7f6ff1e44700 1 -- 172.21.15.98:0/2264658792 shutdown_connections 2024-04-25T17:32:30.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.056+0000 7f6ff1e44700 1 --2- 172.21.15.98:0/2264658792 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fec06bb10 0x7f6fec10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.056+0000 7f6ff1e44700 1 --2- 172.21.15.98:0/2264658792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fec108a30 0x7f6fec108e10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.056+0000 7f6ff1e44700 1 --2- 172.21.15.98:0/2264658792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fec06b170 0x7f6fec06b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.056+0000 7f6ff1e44700 1 -- 172.21.15.98:0/2264658792 >> 172.21.15.98:0/2264658792 conn(0x7f6fec0788f0 msgr2=0x7f6fec078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.056+0000 7f6ff1e44700 1 -- 172.21.15.98:0/2264658792 shutdown_connections 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.056+0000 7f6ff1e44700 1 -- 172.21.15.98:0/2264658792 wait complete. 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.057+0000 7f6ff1e44700 1 Processor -- start 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.057+0000 7f6ff1e44700 1 -- start start 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.057+0000 7f6ff1e44700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fec06b170 0x7f6fec19b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.057+0000 7f6ff1e44700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fec06bb10 0x7f6fec19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.057+0000 7f6ff1e44700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fec108a30 0x7f6fec19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.057+0000 7f6ff1e44700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec112c80 con 0x7f6fec06b170 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.057+0000 7f6ff1e44700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec112b00 con 0x7f6fec06bb10 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.057+0000 7f6ff1e44700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec112e00 con 0x7f6fec108a30 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fec06b170 0x7f6fec19b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:30.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fec06b170 0x7f6fec19b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53854/0 (socket says 172.21.15.98:53854) 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feb7fe700 1 -- 172.21.15.98:0/3462817873 learned_addr learned my addr 172.21.15.98:0/3462817873 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feaffd700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fec06bb10 0x7f6fec19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6febfff700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fec108a30 0x7f6fec19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feb7fe700 1 -- 172.21.15.98:0/3462817873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fec108a30 msgr2=0x7f6fec19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feb7fe700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fec108a30 0x7f6fec19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feb7fe700 1 -- 172.21.15.98:0/3462817873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fec06bb10 msgr2=0x7f6fec19b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feb7fe700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fec06bb10 0x7f6fec19b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feb7fe700 1 -- 172.21.15.98:0/3462817873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fec1a03b0 con 0x7f6fec06b170 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feaffd700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fec06bb10 0x7f6fec19b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6feb7fe700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fec06b170 0x7f6fec19b3f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0005910 tx=0x7f6fe00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6fe8ff9700 1 -- 172.21.15.98:0/3462817873 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0027070 con 0x7f6fec06b170 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6ff1e44700 1 -- 172.21.15.98:0/3462817873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fec1a0640 con 0x7f6fec06b170 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6fe8ff9700 1 -- 172.21.15.98:0/3462817873 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fe0004020 con 0x7f6fec06b170 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6fe8ff9700 1 -- 172.21.15.98:0/3462817873 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0022e00 con 0x7f6fec06b170 2024-04-25T17:32:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.058+0000 7f6ff1e44700 1 -- 172.21.15.98:0/3462817873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fec1a7f20 con 0x7f6fec06b170 2024-04-25T17:32:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.060+0000 7f6ff1e44700 1 -- 172.21.15.98:0/3462817873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fec069730 con 0x7f6fec06b170 2024-04-25T17:32:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.060+0000 7f6fe8ff9700 1 -- 172.21.15.98:0/3462817873 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6fe002a070 con 0x7f6fec06b170 2024-04-25T17:32:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.061+0000 7f6fe8ff9700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6fd40a3c40 0x7f6fd40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.061+0000 7f6feaffd700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6fd40a3c40 0x7f6fd40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.061+0000 7f6fe8ff9700 1 -- 172.21.15.98:0/3462817873 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6fe00d0e20 con 0x7f6fec06b170 2024-04-25T17:32:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.061+0000 7f6feaffd700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6fd40a3c40 0x7f6fd40a6100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc005fd0 tx=0x7f6fdc005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:30.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.066+0000 7f6fe8ff9700 1 -- 172.21.15.98:0/3462817873 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6fe005eff0 con 0x7f6fec06b170 2024-04-25T17:32:30.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:30.262+0000 7f6ff1e44700 1 -- 172.21.15.98:0/3462817873 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f6fec02df80 con 0x7f6fec06b170 2024-04-25T17:32:30.755 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:30 smithi203 ceph-mon[32834]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:30.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:30 smithi173 ceph-mon[32856]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:31.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.023+0000 7f6fe8ff9700 1 -- 172.21.15.98:0/3462817873 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f6fe0013070 con 0x7f6fec06b170 2024-04-25T17:32:31.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 -- 172.21.15.98:0/3462817873 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6fd40a3c40 msgr2=0x7f6fd40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:31.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6fd40a3c40 0x7f6fd40a6100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc005fd0 tx=0x7f6fdc005e30 comp rx=0 tx=0).stop 2024-04-25T17:32:31.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 -- 172.21.15.98:0/3462817873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fec06b170 msgr2=0x7f6fec19b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:31.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fec06b170 0x7f6fec19b3f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0005910 tx=0x7f6fe00047b0 comp rx=0 tx=0).stop 2024-04-25T17:32:31.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 -- 172.21.15.98:0/3462817873 shutdown_connections 2024-04-25T17:32:31.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6fd40a3c40 0x7f6fd40a6100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:31.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6fec108a30 0x7f6fec19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:31.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6fec06b170 0x7f6fec19b3f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:31.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 --2- 172.21.15.98:0/3462817873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6fec06bb10 0x7f6fec19b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:31.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 -- 172.21.15.98:0/3462817873 >> 172.21.15.98:0/3462817873 conn(0x7f6fec0788f0 msgr2=0x7f6fec10de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:31.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 -- 172.21.15.98:0/3462817873 shutdown_connections 2024-04-25T17:32:31.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:31.027+0000 7f6ff1e44700 1 -- 172.21.15.98:0/3462817873 wait complete. 2024-04-25T17:32:31.031 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2024-04-25T17:32:31.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:31 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3462817873' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T17:32:31.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:31 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3462817873' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T17:32:31.473 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:31 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3462817873' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T17:32:31.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T17:32:32.012 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:32.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:32 smithi098 ceph-mon[29192]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:32.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:32 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3462817873' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T17:32:32.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:32 smithi098 ceph-mon[29192]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T17:32:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:32 smithi203 ceph-mon[32834]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:32 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3462817873' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T17:32:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:32 smithi203 ceph-mon[32834]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:32 smithi173 ceph-mon[32856]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:32 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3462817873' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T17:32:32.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:32 smithi173 ceph-mon[32856]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T17:32:33.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:33 smithi098 ceph-mon[29192]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T17:32:33.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:33 smithi203 ceph-mon[32834]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T17:32:33.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:33 smithi173 ceph-mon[32856]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.741+0000 7f0efa7ce700 1 -- 172.21.15.98:0/2046277138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef4107990 msgr2=0x7f0ef4107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.741+0000 7f0efa7ce700 1 --2- 172.21.15.98:0/2046277138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef4107990 0x7f0ef4107d70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009990 tx=0x7f0ee4009ca0 comp rx=0 tx=0).stop 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.741+0000 7f0efa7ce700 1 -- 172.21.15.98:0/2046277138 shutdown_connections 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.741+0000 7f0efa7ce700 1 --2- 172.21.15.98:0/2046277138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ef4102300 0x7f0ef410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.741+0000 7f0efa7ce700 1 --2- 172.21.15.98:0/2046277138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef4107990 0x7f0ef4107d70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.741+0000 7f0efa7ce700 1 --2- 172.21.15.98:0/2046277138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ef4101940 0x7f0ef4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.741+0000 7f0efa7ce700 1 -- 172.21.15.98:0/2046277138 >> 172.21.15.98:0/2046277138 conn(0x7f0ef40788f0 msgr2=0x7f0ef4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.741+0000 7f0efa7ce700 1 -- 172.21.15.98:0/2046277138 shutdown_connections 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.742+0000 7f0efa7ce700 1 -- 172.21.15.98:0/2046277138 wait complete. 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.742+0000 7f0efa7ce700 1 Processor -- start 2024-04-25T17:32:33.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.742+0000 7f0efa7ce700 1 -- start start 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.742+0000 7f0efa7ce700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ef4101940 0x7f0ef4198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.742+0000 7f0efa7ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef4102300 0x7f0ef4198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.742+0000 7f0efa7ce700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ef4107990 0x7f0ef419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.742+0000 7f0efa7ce700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4111bb0 con 0x7f0ef4102300 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.742+0000 7f0efa7ce700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4111a30 con 0x7f0ef4101940 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.742+0000 7f0efa7ce700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4111d30 con 0x7f0ef4107990 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef4102300 0x7f0ef4198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef4102300 0x7f0ef4198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47240/0 (socket says 172.21.15.98:47240) 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef37fe700 1 -- 172.21.15.98:0/3918600680 learned_addr learned my addr 172.21.15.98:0/3918600680 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef3fff700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ef4101940 0x7f0ef4198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef8d6b700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ef4107990 0x7f0ef419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef37fe700 1 -- 172.21.15.98:0/3918600680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ef4107990 msgr2=0x7f0ef419ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:33.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef37fe700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ef4107990 0x7f0ef419ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef37fe700 1 -- 172.21.15.98:0/3918600680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ef4101940 msgr2=0x7f0ef4198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef37fe700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ef4101940 0x7f0ef4198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef37fe700 1 -- 172.21.15.98:0/3918600680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef419d150 con 0x7f0ef4102300 2024-04-25T17:32:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef37fe700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef4102300 0x7f0ef4198680 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009970 tx=0x7f0ee8009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef17fa700 1 -- 172.21.15.98:0/3918600680 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee801c070 con 0x7f0ef4102300 2024-04-25T17:32:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef17fa700 1 -- 172.21.15.98:0/3918600680 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ee800d260 con 0x7f0ef4102300 2024-04-25T17:32:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0efa7ce700 1 -- 172.21.15.98:0/3918600680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef419d440 con 0x7f0ef4102300 2024-04-25T17:32:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0ef17fa700 1 -- 172.21.15.98:0/3918600680 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8017ab0 con 0x7f0ef4102300 2024-04-25T17:32:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.743+0000 7f0efa7ce700 1 -- 172.21.15.98:0/3918600680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef406a720 con 0x7f0ef4102300 2024-04-25T17:32:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.744+0000 7f0efa7ce700 1 -- 172.21.15.98:0/3918600680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef4069730 con 0x7f0ef4102300 2024-04-25T17:32:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.745+0000 7f0ef17fa700 1 -- 172.21.15.98:0/3918600680 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0ee801f070 con 0x7f0ef4102300 2024-04-25T17:32:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.746+0000 7f0ef17fa700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0edc0a3cc0 0x7f0edc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.746+0000 7f0ef3fff700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0edc0a3cc0 0x7f0edc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.746+0000 7f0ef17fa700 1 -- 172.21.15.98:0/3918600680 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7497+0+0 (secure 0 0 0) 0x7f0ee80d15e0 con 0x7f0ef4102300 2024-04-25T17:32:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.746+0000 7f0ef3fff700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0edc0a3cc0 0x7f0edc0a6180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400b5c0 tx=0x7f0ee401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:33.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.750+0000 7f0ef17fa700 1 -- 172.21.15.98:0/3918600680 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ee809e2f0 con 0x7f0ef4102300 2024-04-25T17:32:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:33.952+0000 7f0efa7ce700 1 -- 172.21.15.98:0/3918600680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0ef41994a0 con 0x7f0ef4102300 2024-04-25T17:32:34.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.040+0000 7f0ef17fa700 1 -- 172.21.15.98:0/3918600680 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v86) ==== 170+0+0 (secure 0 0 0) 0x7f0ee809dbf0 con 0x7f0ef4102300 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.043+0000 7f0efa7ce700 1 -- 172.21.15.98:0/3918600680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0edc0a3cc0 msgr2=0x7f0edc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.043+0000 7f0efa7ce700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0edc0a3cc0 0x7f0edc0a6180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400b5c0 tx=0x7f0ee401a040 comp rx=0 tx=0).stop 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.043+0000 7f0efa7ce700 1 -- 172.21.15.98:0/3918600680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef4102300 msgr2=0x7f0ef4198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.043+0000 7f0efa7ce700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef4102300 0x7f0ef4198680 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009970 tx=0x7f0ee8009d30 comp rx=0 tx=0).stop 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.044+0000 7f0efa7ce700 1 -- 172.21.15.98:0/3918600680 shutdown_connections 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.044+0000 7f0efa7ce700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0edc0a3cc0 0x7f0edc0a6180 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.044+0000 7f0efa7ce700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ef4107990 0x7f0ef419ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.044+0000 7f0efa7ce700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ef4101940 0x7f0ef4198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.044+0000 7f0efa7ce700 1 --2- 172.21.15.98:0/3918600680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ef4102300 0x7f0ef4198680 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.044+0000 7f0efa7ce700 1 -- 172.21.15.98:0/3918600680 >> 172.21.15.98:0/3918600680 conn(0x7f0ef40788f0 msgr2=0x7f0ef41007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:34.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.044+0000 7f0efa7ce700 1 -- 172.21.15.98:0/3918600680 shutdown_connections 2024-04-25T17:32:34.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:34.044+0000 7f0efa7ce700 1 -- 172.21.15.98:0/3918600680 wait complete. 2024-04-25T17:32:34.047 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2024-04-25T17:32:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:34 smithi098 ceph-mon[29192]: pgmap v296: 33 pgs: 2 creating+activating, 5 creating+peering, 11 active+clean, 15 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:34 smithi098 ceph-mon[29192]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T17:32:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:34 smithi098 ceph-mon[29192]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T17:32:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:34 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3918600680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T17:32:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:34 smithi203 ceph-mon[32834]: pgmap v296: 33 pgs: 2 creating+activating, 5 creating+peering, 11 active+clean, 15 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:34 smithi203 ceph-mon[32834]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T17:32:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:34 smithi203 ceph-mon[32834]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T17:32:34.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:34 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3918600680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T17:32:34.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:34 smithi173 ceph-mon[32856]: pgmap v296: 33 pgs: 2 creating+activating, 5 creating+peering, 11 active+clean, 15 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:34.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:34 smithi173 ceph-mon[32856]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T17:32:34.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:34 smithi173 ceph-mon[32856]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T17:32:34.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:34 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3918600680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T17:32:34.875 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T17:32:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:35.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:35 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3918600680' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T17:32:35.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:35 smithi098 ceph-mon[29192]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T17:32:35.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:35 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3918600680' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T17:32:35.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:35 smithi203 ceph-mon[32834]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T17:32:35.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:35 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3918600680' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T17:32:35.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:35 smithi173 ceph-mon[32856]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T17:32:36.376 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:36 smithi098 ceph-mon[29192]: pgmap v299: 97 pgs: 2 creating+activating, 24 creating+peering, 23 active+clean, 48 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:36.376 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:36 smithi098 ceph-mon[29192]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T17:32:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:36 smithi203 ceph-mon[32834]: pgmap v299: 97 pgs: 2 creating+activating, 24 creating+peering, 23 active+clean, 48 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:36 smithi203 ceph-mon[32834]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T17:32:36.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:36 smithi173 ceph-mon[32856]: pgmap v299: 97 pgs: 2 creating+activating, 24 creating+peering, 23 active+clean, 48 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:36.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:36 smithi173 ceph-mon[32856]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T17:32:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 -- 172.21.15.98:0/1051038860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa950102620 msgr2=0x7fa95010e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 --2- 172.21.15.98:0/1051038860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa950102620 0x7fa95010e790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa94800b0a0 tx=0x7fa94800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:32:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 -- 172.21.15.98:0/1051038860 shutdown_connections 2024-04-25T17:32:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 --2- 172.21.15.98:0/1051038860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa950102620 0x7fa95010e790 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 --2- 172.21.15.98:0/1051038860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa950107a20 0x7fa950107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 --2- 172.21.15.98:0/1051038860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa950101c60 0x7fa9501020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 -- 172.21.15.98:0/1051038860 >> 172.21.15.98:0/1051038860 conn(0x7fa9500786a0 msgr2=0x7fa950078ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 -- 172.21.15.98:0/1051038860 shutdown_connections 2024-04-25T17:32:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 -- 172.21.15.98:0/1051038860 wait complete. 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 Processor -- start 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.916+0000 7fa95794c700 1 -- start start 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa95794c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa950101c60 0x7fa950199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa95794c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa950102620 0x7fa95019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa95794c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa950107a20 0x7fa9501a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa95794c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa950111b80 con 0x7fa950102620 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa95794c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa950111a00 con 0x7fa950107a20 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa95794c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa950111d00 con 0x7fa950101c60 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa954ee7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa950102620 0x7fa95019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa954ee7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa950102620 0x7fa95019a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47268/0 (socket says 172.21.15.98:47268) 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa954ee7700 1 -- 172.21.15.98:0/4114940503 learned_addr learned my addr 172.21.15.98:0/4114940503 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa9556e8700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa950101c60 0x7fa950199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa955ee9700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa950107a20 0x7fa9501a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa954ee7700 1 -- 172.21.15.98:0/4114940503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa950101c60 msgr2=0x7fa950199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa954ee7700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa950101c60 0x7fa950199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa954ee7700 1 -- 172.21.15.98:0/4114940503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa950107a20 msgr2=0x7fa9501a4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa954ee7700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa950107a20 0x7fa9501a4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa954ee7700 1 -- 172.21.15.98:0/4114940503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9501a5380 con 0x7fa950102620 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa9556e8700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa950101c60 0x7fa950199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa955ee9700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa950107a20 0x7fa9501a4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa954ee7700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa950102620 0x7fa95019a300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa94000c920 tx=0x7fa94000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa9467fc700 1 -- 172.21.15.98:0/4114940503 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940007b00 con 0x7fa950102620 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa95794c700 1 -- 172.21.15.98:0/4114940503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9501a55b0 con 0x7fa950102620 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa9467fc700 1 -- 172.21.15.98:0/4114940503 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa940004ce0 con 0x7fa950102620 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa9467fc700 1 -- 172.21.15.98:0/4114940503 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940005650 con 0x7fa950102620 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.917+0000 7fa95794c700 1 -- 172.21.15.98:0/4114940503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9501a5b40 con 0x7fa950102620 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.918+0000 7fa95794c700 1 -- 172.21.15.98:0/4114940503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa950069730 con 0x7fa950102620 2024-04-25T17:32:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.919+0000 7fa9467fc700 1 -- 172.21.15.98:0/4114940503 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa940007ca0 con 0x7fa950102620 2024-04-25T17:32:36.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.920+0000 7fa9467fc700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa93c0a3c40 0x7fa93c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:36.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.920+0000 7fa9556e8700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa93c0a3c40 0x7fa93c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:36.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.920+0000 7fa9467fc700 1 -- 172.21.15.98:0/4114940503 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7879+0+0 (secure 0 0 0) 0x7fa9400d01e0 con 0x7fa950102620 2024-04-25T17:32:36.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.921+0000 7fa9556e8700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa93c0a3c40 0x7fa93c0a6100 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa94c005fd0 tx=0x7fa94c00a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:36.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:36.924+0000 7fa9467fc700 1 -- 172.21.15.98:0/4114940503 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa940014030 con 0x7fa950102620 2024-04-25T17:32:37.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:37.172+0000 7fa95794c700 1 -- 172.21.15.98:0/4114940503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa9501a5f10 con 0x7fa950102620 2024-04-25T17:32:37.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:37 smithi098 ceph-mon[29192]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T17:32:37.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:37 smithi203 ceph-mon[32834]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T17:32:37.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:37 smithi173 ceph-mon[32856]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T17:32:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.067+0000 7fa9467fc700 1 -- 172.21.15.98:0/4114940503 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fa94009cd70 con 0x7fa950102620 2024-04-25T17:32:38.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 -- 172.21.15.98:0/4114940503 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa93c0a3c40 msgr2=0x7fa93c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa93c0a3c40 0x7fa93c0a6100 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa94c005fd0 tx=0x7fa94c00a5e0 comp rx=0 tx=0).stop 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 -- 172.21.15.98:0/4114940503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa950102620 msgr2=0x7fa95019a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa950102620 0x7fa95019a300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa94000c920 tx=0x7fa94000cce0 comp rx=0 tx=0).stop 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 -- 172.21.15.98:0/4114940503 shutdown_connections 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa93c0a3c40 0x7fa93c0a6100 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa950101c60 0x7fa950199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa950107a20 0x7fa9501a4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 --2- 172.21.15.98:0/4114940503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa950102620 0x7fa95019a300 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 -- 172.21.15.98:0/4114940503 >> 172.21.15.98:0/4114940503 conn(0x7fa9500786a0 msgr2=0x7fa9501006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 -- 172.21.15.98:0/4114940503 shutdown_connections 2024-04-25T17:32:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:38.070+0000 7fa95794c700 1 -- 172.21.15.98:0/4114940503 wait complete. 2024-04-25T17:32:38.073 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T17:32:38.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:38 smithi098 ceph-mon[29192]: pgmap v302: 97 pgs: 2 creating+activating, 21 creating+peering, 35 active+clean, 39 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:38.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:38 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4114940503' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T17:32:38.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:38 smithi098 ceph-mon[29192]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T17:32:38.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:38 smithi203 ceph-mon[32834]: pgmap v302: 97 pgs: 2 creating+activating, 21 creating+peering, 35 active+clean, 39 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:38.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:38 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/4114940503' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T17:32:38.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:38 smithi203 ceph-mon[32834]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T17:32:38.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:38 smithi173 ceph-mon[32856]: pgmap v302: 97 pgs: 2 creating+activating, 21 creating+peering, 35 active+clean, 39 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:38.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:38 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/4114940503' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T17:32:38.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:38 smithi173 ceph-mon[32856]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T17:32:38.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T17:32:39.014 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:39.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:39 smithi098 ceph-mon[29192]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T17:32:39.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:39 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4114940503' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T17:32:39.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:39 smithi098 ceph-mon[29192]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T17:32:39.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:39 smithi098 ceph-mon[29192]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T17:32:39.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:39 smithi203 ceph-mon[32834]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T17:32:39.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:39 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/4114940503' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T17:32:39.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:39 smithi203 ceph-mon[32834]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T17:32:39.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:39 smithi203 ceph-mon[32834]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T17:32:39.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:39 smithi173 ceph-mon[32856]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T17:32:39.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:39 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/4114940503' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T17:32:39.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:39 smithi173 ceph-mon[32856]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T17:32:39.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:39 smithi173 ceph-mon[32856]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T17:32:40.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:40 smithi098 ceph-mon[29192]: pgmap v304: 97 pgs: 19 creating+peering, 78 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:40.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:40 smithi098 ceph-mon[29192]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T17:32:40.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:40 smithi098 ceph-mon[29192]: Cluster is now healthy 2024-04-25T17:32:40.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:40 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] up:active 2024-04-25T17:32:40.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:40 smithi098 ceph-mon[29192]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:40.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:40 smithi098 ceph-mon[29192]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:40 smithi203 ceph-mon[32834]: pgmap v304: 97 pgs: 19 creating+peering, 78 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:40 smithi203 ceph-mon[32834]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T17:32:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:40 smithi203 ceph-mon[32834]: Cluster is now healthy 2024-04-25T17:32:40.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:40 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] up:active 2024-04-25T17:32:40.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:40 smithi203 ceph-mon[32834]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:40.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:40 smithi203 ceph-mon[32834]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:40.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:32:40 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b[63135]: 2024-04-25T17:32:40.028+0000 7f00986b2700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T17:32:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:40 smithi173 ceph-mon[32856]: pgmap v304: 97 pgs: 19 creating+peering, 78 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:32:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:40 smithi173 ceph-mon[32856]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T17:32:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:40 smithi173 ceph-mon[32856]: Cluster is now healthy 2024-04-25T17:32:40.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:40 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] up:active 2024-04-25T17:32:40.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:40 smithi173 ceph-mon[32856]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:40.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:40 smithi173 ceph-mon[32856]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.990+0000 7f3abfc15700 1 -- 172.21.15.98:0/2160232582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8107990 msgr2=0x7f3ab8107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.990+0000 7f3abfc15700 1 --2- 172.21.15.98:0/2160232582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8107990 0x7f3ab8107d70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009990 tx=0x7f3aa8009ca0 comp rx=0 tx=0).stop 2024-04-25T17:32:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.990+0000 7f3abfc15700 1 -- 172.21.15.98:0/2160232582 shutdown_connections 2024-04-25T17:32:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.990+0000 7f3abfc15700 1 --2- 172.21.15.98:0/2160232582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ab8102300 0x7f3ab810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.990+0000 7f3abfc15700 1 --2- 172.21.15.98:0/2160232582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8107990 0x7f3ab8107d70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.990+0000 7f3abfc15700 1 --2- 172.21.15.98:0/2160232582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab8101940 0x7f3ab8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.990+0000 7f3abfc15700 1 -- 172.21.15.98:0/2160232582 >> 172.21.15.98:0/2160232582 conn(0x7f3ab80788f0 msgr2=0x7f3ab8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abfc15700 1 -- 172.21.15.98:0/2160232582 shutdown_connections 2024-04-25T17:32:40.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abfc15700 1 -- 172.21.15.98:0/2160232582 wait complete. 2024-04-25T17:32:40.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abfc15700 1 Processor -- start 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abfc15700 1 -- start start 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abfc15700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab8101940 0x7f3ab819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abfc15700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ab8102300 0x7f3ab819b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abfc15700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8107990 0x7f3ab819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abfc15700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab8111b00 con 0x7f3ab8107990 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abfc15700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab8111980 con 0x7f3ab8101940 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abfc15700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab8111c80 con 0x7f3ab8102300 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abe1b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8107990 0x7f3ab819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abe1b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8107990 0x7f3ab819fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47282/0 (socket says 172.21.15.98:47282) 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.991+0000 7f3abe1b2700 1 -- 172.21.15.98:0/4019021171 learned_addr learned my addr 172.21.15.98:0/4019021171 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abd1b0700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ab8102300 0x7f3ab819b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abd9b1700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab8101940 0x7f3ab819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abe1b2700 1 -- 172.21.15.98:0/4019021171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ab8102300 msgr2=0x7f3ab819b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abe1b2700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ab8102300 0x7f3ab819b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:40.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abe1b2700 1 -- 172.21.15.98:0/4019021171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab8101940 msgr2=0x7f3ab819b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abe1b2700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab8101940 0x7f3ab819b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abe1b2700 1 -- 172.21.15.98:0/4019021171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab81a02f0 con 0x7f3ab8107990 2024-04-25T17:32:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abd9b1700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab8101940 0x7f3ab819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:32:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abe1b2700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8107990 0x7f3ab819fbb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0007ee0 tx=0x7f3ab000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3aaeffd700 1 -- 172.21.15.98:0/4019021171 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab000dbd0 con 0x7f3ab8107990 2024-04-25T17:32:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abfc15700 1 -- 172.21.15.98:0/4019021171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab81a05e0 con 0x7f3ab8107990 2024-04-25T17:32:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3aaeffd700 1 -- 172.21.15.98:0/4019021171 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ab000dd70 con 0x7f3ab8107990 2024-04-25T17:32:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3aaeffd700 1 -- 172.21.15.98:0/4019021171 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0005990 con 0x7f3ab8107990 2024-04-25T17:32:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.992+0000 7f3abfc15700 1 -- 172.21.15.98:0/4019021171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab81a7e60 con 0x7f3ab8107990 2024-04-25T17:32:40.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.993+0000 7f3abfc15700 1 -- 172.21.15.98:0/4019021171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab8069730 con 0x7f3ab8107990 2024-04-25T17:32:40.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.994+0000 7f3aaeffd700 1 -- 172.21.15.98:0/4019021171 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3ab001a020 con 0x7f3ab8107990 2024-04-25T17:32:40.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.995+0000 7f3aaeffd700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3aa40a3cc0 0x7f3aa40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:40.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.995+0000 7f3abd9b1700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3aa40a3cc0 0x7f3aa40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:40.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.995+0000 7f3aaeffd700 1 -- 172.21.15.98:0/4019021171 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f3ab00cf930 con 0x7f3ab8107990 2024-04-25T17:32:40.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.995+0000 7f3abd9b1700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3aa40a3cc0 0x7f3aa40a6180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800b2e0 tx=0x7f3aa801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:41.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:40.999+0000 7f3aaeffd700 1 -- 172.21.15.98:0/4019021171 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ab0010070 con 0x7f3ab8107990 2024-04-25T17:32:41.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.200+0000 7f3abfc15700 1 -- 172.21.15.98:0/4019021171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ab81a80f0 con 0x7f3ab8107990 2024-04-25T17:32:41.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.202+0000 7f3aaeffd700 1 -- 172.21.15.98:0/4019021171 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22255 (secure 0 0 0) 0x7f3ab009c460 con 0x7f3ab8107990 2024-04-25T17:32:41.203 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:32:41.204 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:32:37.173499+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T17:32:30.267079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T17:32:33.956902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:32:41.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 -- 172.21.15.98:0/4019021171 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3aa40a3cc0 msgr2=0x7f3aa40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3aa40a3cc0 0x7f3aa40a6180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800b2e0 tx=0x7f3aa801a040 comp rx=0 tx=0).stop 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 -- 172.21.15.98:0/4019021171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8107990 msgr2=0x7f3ab819fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8107990 0x7f3ab819fbb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0007ee0 tx=0x7f3ab000d3b0 comp rx=0 tx=0).stop 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 -- 172.21.15.98:0/4019021171 shutdown_connections 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3aa40a3cc0 0x7f3aa40a6180 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ab8107990 0x7f3ab819fbb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ab8101940 0x7f3ab819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 --2- 172.21.15.98:0/4019021171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ab8102300 0x7f3ab819b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 -- 172.21.15.98:0/4019021171 >> 172.21.15.98:0/4019021171 conn(0x7f3ab80788f0 msgr2=0x7f3ab806a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 -- 172.21.15.98:0/4019021171 shutdown_connections 2024-04-25T17:32:41.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:41.204+0000 7f3abfc15700 1 -- 172.21.15.98:0/4019021171 wait complete. 2024-04-25T17:32:41.916 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T17:32:42.156 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:42.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:42 smithi098 ceph-mon[29192]: pgmap v305: 97 pgs: 97 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2024-04-25T17:32:42.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:42 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4019021171' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:32:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:42 smithi203 ceph-mon[32834]: pgmap v305: 97 pgs: 97 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2024-04-25T17:32:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:42 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/4019021171' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:32:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:42 smithi173 ceph-mon[32856]: pgmap v305: 97 pgs: 97 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2024-04-25T17:32:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:42 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/4019021171' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:32:43.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 -- 172.21.15.98:0/338478167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12601025e0 msgr2=0x7f126010e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:43.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 --2- 172.21.15.98:0/338478167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12601025e0 0x7f126010e7d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f125800b0a0 tx=0x7f125800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:32:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 -- 172.21.15.98:0/338478167 shutdown_connections 2024-04-25T17:32:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 --2- 172.21.15.98:0/338478167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12601025e0 0x7f126010e7d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 --2- 172.21.15.98:0/338478167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1260106940 0x7f1260106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 --2- 172.21.15.98:0/338478167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1260101c20 0x7f12601020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 -- 172.21.15.98:0/338478167 >> 172.21.15.98:0/338478167 conn(0x7f1260078830 msgr2=0x7f1260078c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 -- 172.21.15.98:0/338478167 shutdown_connections 2024-04-25T17:32:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 -- 172.21.15.98:0/338478167 wait complete. 2024-04-25T17:32:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 Processor -- start 2024-04-25T17:32:43.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.881+0000 7f1266104700 1 -- start start 2024-04-25T17:32:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1266104700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1260101c20 0x7f12601033f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1266104700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12601025e0 0x7f1260103930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1266104700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1260106940 0x7f1260103e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1266104700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1260111bf0 con 0x7f12601025e0 2024-04-25T17:32:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1266104700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1260111a70 con 0x7f1260106940 2024-04-25T17:32:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1266104700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1260111d70 con 0x7f1260101c20 2024-04-25T17:32:43.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1256ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12601025e0 0x7f1260103930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1256ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12601025e0 0x7f1260103930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55454/0 (socket says 172.21.15.98:55454) 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1256ffd700 1 -- 172.21.15.98:0/1356289184 learned_addr learned my addr 172.21.15.98:0/1356289184 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f125f7fe700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1260101c20 0x7f12601033f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f125ffff700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1260106940 0x7f1260103e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1256ffd700 1 -- 172.21.15.98:0/1356289184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1260101c20 msgr2=0x7f12601033f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1256ffd700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1260101c20 0x7f12601033f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1256ffd700 1 -- 172.21.15.98:0/1356289184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1260106940 msgr2=0x7f1260103e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1256ffd700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1260106940 0x7f1260103e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1256ffd700 1 -- 172.21.15.98:0/1356289184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12601a7790 con 0x7f12601025e0 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f125ffff700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1260106940 0x7f1260103e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.882+0000 7f1256ffd700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12601025e0 0x7f1260103930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f124800b530 tx=0x7f124800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.883+0000 7f125d7fa700 1 -- 172.21.15.98:0/1356289184 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248015020 con 0x7f12601025e0 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.883+0000 7f1266104700 1 -- 172.21.15.98:0/1356289184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12601a7a80 con 0x7f12601025e0 2024-04-25T17:32:43.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.883+0000 7f125d7fa700 1 -- 172.21.15.98:0/1356289184 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1248011ce0 con 0x7f12601025e0 2024-04-25T17:32:43.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.883+0000 7f125d7fa700 1 -- 172.21.15.98:0/1356289184 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248010e40 con 0x7f12601025e0 2024-04-25T17:32:43.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.883+0000 7f1266104700 1 -- 172.21.15.98:0/1356289184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12601a7fc0 con 0x7f12601025e0 2024-04-25T17:32:43.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.884+0000 7f1266104700 1 -- 172.21.15.98:0/1356289184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12601030b0 con 0x7f12601025e0 2024-04-25T17:32:43.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.885+0000 7f125d7fa700 1 -- 172.21.15.98:0/1356289184 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1248018020 con 0x7f12601025e0 2024-04-25T17:32:43.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.885+0000 7f125d7fa700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f124c0a3c40 0x7f124c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:43.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.885+0000 7f125d7fa700 1 -- 172.21.15.98:0/1356289184 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f12480cfad0 con 0x7f12601025e0 2024-04-25T17:32:43.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.885+0000 7f125f7fe700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f124c0a3c40 0x7f124c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:43.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.886+0000 7f125f7fe700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f124c0a3c40 0x7f124c0a6100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1250009a10 tx=0x7f1250009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:43.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:43.890+0000 7f125d7fa700 1 -- 172.21.15.98:0/1356289184 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f124801f020 con 0x7f12601025e0 2024-04-25T17:32:44.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.089+0000 7f1266104700 1 -- 172.21.15.98:0/1356289184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f126002df80 con 0x7f12601025e0 2024-04-25T17:32:44.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.090+0000 7f125d7fa700 1 -- 172.21.15.98:0/1356289184 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22255 (secure 0 0 0) 0x7f124809c600 con 0x7f12601025e0 2024-04-25T17:32:44.092 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:32:44.092 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":89,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:32:37.173499+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T17:32:30.267079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T17:32:33.956902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:32:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.092+0000 7f1266104700 1 -- 172.21.15.98:0/1356289184 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f124c0a3c40 msgr2=0x7f124c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.092+0000 7f1266104700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f124c0a3c40 0x7f124c0a6100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1250009a10 tx=0x7f1250009190 comp rx=0 tx=0).stop 2024-04-25T17:32:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.092+0000 7f1266104700 1 -- 172.21.15.98:0/1356289184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12601025e0 msgr2=0x7f1260103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.092+0000 7f1266104700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12601025e0 0x7f1260103930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f124800b530 tx=0x7f124800b840 comp rx=0 tx=0).stop 2024-04-25T17:32:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.093+0000 7f1266104700 1 -- 172.21.15.98:0/1356289184 shutdown_connections 2024-04-25T17:32:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.093+0000 7f1266104700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f124c0a3c40 0x7f124c0a6100 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.093+0000 7f1266104700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1260101c20 0x7f12601033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.093+0000 7f1266104700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1260106940 0x7f1260103e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:44.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.093+0000 7f1266104700 1 --2- 172.21.15.98:0/1356289184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12601025e0 0x7f1260103930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:44.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.093+0000 7f1266104700 1 -- 172.21.15.98:0/1356289184 >> 172.21.15.98:0/1356289184 conn(0x7f1260078830 msgr2=0x7f126006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:44.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.093+0000 7f1266104700 1 -- 172.21.15.98:0/1356289184 shutdown_connections 2024-04-25T17:32:44.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:44.093+0000 7f1266104700 1 -- 172.21.15.98:0/1356289184 wait complete. 2024-04-25T17:32:44.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:44 smithi098 ceph-mon[29192]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-25T17:32:44.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:44 smithi203 ceph-mon[32834]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-25T17:32:44.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:44 smithi173 ceph-mon[32856]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-25T17:32:44.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-04-25T17:32:45.146 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:45.369 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:45 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1356289184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:32:45.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:45 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1356289184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:32:45.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:45 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1356289184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:32:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:46 smithi098 ceph-mon[29192]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-25T17:32:46.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:46 smithi203 ceph-mon[32834]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-25T17:32:46.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:46 smithi173 ceph-mon[32856]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-04-25T17:32:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 -- 172.21.15.98:0/4260666368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a68103450 msgr2=0x7f0a6810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 --2- 172.21.15.98:0/4260666368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a68103450 0x7f0a6810f8b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000b0a0 tx=0x7f0a6000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:32:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 -- 172.21.15.98:0/4260666368 shutdown_connections 2024-04-25T17:32:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 --2- 172.21.15.98:0/4260666368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a68103450 0x7f0a6810f8b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 --2- 172.21.15.98:0/4260666368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a68108a50 0x7f0a68108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 --2- 172.21.15.98:0/4260666368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a68102a90 0x7f0a68102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 -- 172.21.15.98:0/4260666368 >> 172.21.15.98:0/4260666368 conn(0x7f0a68079ba0 msgr2=0x7f0a68079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 -- 172.21.15.98:0/4260666368 shutdown_connections 2024-04-25T17:32:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 -- 172.21.15.98:0/4260666368 wait complete. 2024-04-25T17:32:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 Processor -- start 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.923+0000 7f0a6ed85700 1 -- start start 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a6ed85700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a68102a90 0x7f0a681990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a6ed85700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a68103450 0x7f0a681995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a6ed85700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a68108a50 0x7f0a6819d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a6ed85700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a68112c60 con 0x7f0a68103450 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a6ed85700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a68112ae0 con 0x7f0a68108a50 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a6ed85700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a68112de0 con 0x7f0a68102a90 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a67fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a68103450 0x7f0a681995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a67fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a68103450 0x7f0a681995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55474/0 (socket says 172.21.15.98:55474) 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a67fff700 1 -- 172.21.15.98:0/3679368411 learned_addr learned my addr 172.21.15.98:0/3679368411 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a6cb21700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a68102a90 0x7f0a681990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a6d322700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a68108a50 0x7f0a6819d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a67fff700 1 -- 172.21.15.98:0/3679368411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a68102a90 msgr2=0x7f0a681990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:46.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a67fff700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a68102a90 0x7f0a681990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a67fff700 1 -- 172.21.15.98:0/3679368411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a68108a50 msgr2=0x7f0a6819d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a67fff700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a68108a50 0x7f0a6819d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a67fff700 1 -- 172.21.15.98:0/3679368411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6819e0c0 con 0x7f0a68103450 2024-04-25T17:32:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a67fff700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a68103450 0x7f0a681995f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400c9b0 tx=0x7f0a5400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a65ffb700 1 -- 172.21.15.98:0/3679368411 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54007b00 con 0x7f0a68103450 2024-04-25T17:32:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a6ed85700 1 -- 172.21.15.98:0/3679368411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6819e3b0 con 0x7f0a68103450 2024-04-25T17:32:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a65ffb700 1 -- 172.21.15.98:0/3679368411 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a54007ca0 con 0x7f0a68103450 2024-04-25T17:32:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.924+0000 7f0a65ffb700 1 -- 172.21.15.98:0/3679368411 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a54018660 con 0x7f0a68103450 2024-04-25T17:32:46.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.925+0000 7f0a6ed85700 1 -- 172.21.15.98:0/3679368411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a681a5c30 con 0x7f0a68103450 2024-04-25T17:32:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.926+0000 7f0a6ed85700 1 -- 172.21.15.98:0/3679368411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a68069730 con 0x7f0a68103450 2024-04-25T17:32:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.927+0000 7f0a65ffb700 1 -- 172.21.15.98:0/3679368411 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0a5400f450 con 0x7f0a68103450 2024-04-25T17:32:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.927+0000 7f0a65ffb700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a580a3c40 0x7f0a580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.927+0000 7f0a6cb21700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a580a3c40 0x7f0a580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.928+0000 7f0a65ffb700 1 -- 172.21.15.98:0/3679368411 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f0a540d0270 con 0x7f0a68103450 2024-04-25T17:32:46.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.928+0000 7f0a6cb21700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a580a3c40 0x7f0a580a6100 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009a10 tx=0x7f0a5c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:46.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:46.932+0000 7f0a65ffb700 1 -- 172.21.15.98:0/3679368411 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a54014030 con 0x7f0a68103450 2024-04-25T17:32:47.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:47.179+0000 7f0a6ed85700 1 -- 172.21.15.98:0/3679368411 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0a6802df80 con 0x7f0a68103450 2024-04-25T17:32:48.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.084+0000 7f0a65ffb700 1 -- 172.21.15.98:0/3679368411 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f0a5409cda0 con 0x7f0a68103450 2024-04-25T17:32:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 -- 172.21.15.98:0/3679368411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a580a3c40 msgr2=0x7f0a580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:48.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a580a3c40 0x7f0a580a6100 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009a10 tx=0x7f0a5c009190 comp rx=0 tx=0).stop 2024-04-25T17:32:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 -- 172.21.15.98:0/3679368411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a68103450 msgr2=0x7f0a681995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a68103450 0x7f0a681995f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400c9b0 tx=0x7f0a5400ccc0 comp rx=0 tx=0).stop 2024-04-25T17:32:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 -- 172.21.15.98:0/3679368411 shutdown_connections 2024-04-25T17:32:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a580a3c40 0x7f0a580a6100 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a68102a90 0x7f0a681990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a68108a50 0x7f0a6819d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 --2- 172.21.15.98:0/3679368411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a68103450 0x7f0a681995f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 -- 172.21.15.98:0/3679368411 >> 172.21.15.98:0/3679368411 conn(0x7f0a68079ba0 msgr2=0x7f0a68101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 -- 172.21.15.98:0/3679368411 shutdown_connections 2024-04-25T17:32:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:48.086+0000 7f0a6ed85700 1 -- 172.21.15.98:0/3679368411 wait complete. 2024-04-25T17:32:48.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:48 smithi098 ceph-mon[29192]: pgmap v308: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-25T17:32:48.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:48 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3679368411' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T17:32:48.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:48 smithi203 ceph-mon[32834]: pgmap v308: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-25T17:32:48.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:48 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3679368411' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T17:32:48.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:48 smithi173 ceph-mon[32856]: pgmap v308: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-25T17:32:48.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:48 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3679368411' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T17:32:48.921 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-25T17:32:48.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-04-25T17:32:49.163 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:49.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:49 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3679368411' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T17:32:49.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:49 smithi098 ceph-mon[29192]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:49.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:49 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3679368411' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T17:32:49.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:49 smithi203 ceph-mon[32834]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:49.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:49 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3679368411' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T17:32:49.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:49 smithi173 ceph-mon[32856]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:50.364 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:50 smithi203 ceph-mon[32834]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-25T17:32:50.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:50 smithi173 ceph-mon[32856]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-25T17:32:50.437 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:50 smithi098 ceph-mon[29192]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-25T17:32:51.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.004+0000 7f26395b7700 1 -- 172.21.15.98:0/1563250408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f263406bb70 msgr2=0x7f263410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.004+0000 7f26395b7700 1 --2- 172.21.15.98:0/1563250408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f263406bb70 0x7f263410f900 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f262c00b0d0 tx=0x7f262c00b3e0 comp rx=0 tx=0).stop 2024-04-25T17:32:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.004+0000 7f26395b7700 1 -- 172.21.15.98:0/1563250408 shutdown_connections 2024-04-25T17:32:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.004+0000 7f26395b7700 1 --2- 172.21.15.98:0/1563250408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f263406bb70 0x7f263410f900 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.004+0000 7f26395b7700 1 --2- 172.21.15.98:0/1563250408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2634108a50 0x7f2634108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.004+0000 7f26395b7700 1 --2- 172.21.15.98:0/1563250408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f263406b1b0 0x7f263406b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.004+0000 7f26395b7700 1 -- 172.21.15.98:0/1563250408 >> 172.21.15.98:0/1563250408 conn(0x7f2634077990 msgr2=0x7f2634077da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.004+0000 7f26395b7700 1 -- 172.21.15.98:0/1563250408 shutdown_connections 2024-04-25T17:32:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.004+0000 7f26395b7700 1 -- 172.21.15.98:0/1563250408 wait complete. 2024-04-25T17:32:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26395b7700 1 Processor -- start 2024-04-25T17:32:51.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26395b7700 1 -- start start 2024-04-25T17:32:51.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26395b7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f263406b1b0 0x7f26341991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:51.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26395b7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f263406bb70 0x7f26341996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:51.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26395b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2634108a50 0x7f263419da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:51.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26395b7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2634112c90 con 0x7f2634108a50 2024-04-25T17:32:51.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26395b7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2634112b10 con 0x7f263406b1b0 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26395b7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2634112e10 con 0x7f263406bb70 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26337fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2634108a50 0x7f263419da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26337fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2634108a50 0x7f263419da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55490/0 (socket says 172.21.15.98:55490) 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26337fe700 1 -- 172.21.15.98:0/3548295677 learned_addr learned my addr 172.21.15.98:0/3548295677 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f2632ffd700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f263406b1b0 0x7f26341991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f262a7fc700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f263406bb70 0x7f26341996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26337fe700 1 -- 172.21.15.98:0/3548295677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f263406bb70 msgr2=0x7f26341996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26337fe700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f263406bb70 0x7f26341996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26337fe700 1 -- 172.21.15.98:0/3548295677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f263406b1b0 msgr2=0x7f26341991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26337fe700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f263406b1b0 0x7f26341991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.005+0000 7f26337fe700 1 -- 172.21.15.98:0/3548295677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263419e160 con 0x7f2634108a50 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.006+0000 7f26337fe700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2634108a50 0x7f263419da20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f262c009fd0 tx=0x7f262c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.006+0000 7f2630ff9700 1 -- 172.21.15.98:0/3548295677 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262c01f070 con 0x7f2634108a50 2024-04-25T17:32:51.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.006+0000 7f26395b7700 1 -- 172.21.15.98:0/3548295677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263419e3f0 con 0x7f2634108a50 2024-04-25T17:32:51.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.006+0000 7f2630ff9700 1 -- 172.21.15.98:0/3548295677 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f262c003c00 con 0x7f2634108a50 2024-04-25T17:32:51.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.006+0000 7f2630ff9700 1 -- 172.21.15.98:0/3548295677 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262c01b7b0 con 0x7f2634108a50 2024-04-25T17:32:51.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.006+0000 7f26395b7700 1 -- 172.21.15.98:0/3548295677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26341a5cd0 con 0x7f2634108a50 2024-04-25T17:32:51.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.007+0000 7f26395b7700 1 -- 172.21.15.98:0/3548295677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2634069730 con 0x7f2634108a50 2024-04-25T17:32:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.008+0000 7f2630ff9700 1 -- 172.21.15.98:0/3548295677 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f262c022070 con 0x7f2634108a50 2024-04-25T17:32:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.008+0000 7f2630ff9700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26200a3c40 0x7f26200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.008+0000 7f2632ffd700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26200a3c40 0x7f26200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.008+0000 7f2630ff9700 1 -- 172.21.15.98:0/3548295677 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f262c00f020 con 0x7f2634108a50 2024-04-25T17:32:51.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.009+0000 7f2632ffd700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26200a3c40 0x7f26200a6100 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2624009a10 tx=0x7f2624009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:51.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.013+0000 7f2630ff9700 1 -- 172.21.15.98:0/3548295677 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f262c029020 con 0x7f2634108a50 2024-04-25T17:32:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:51.268+0000 7f26395b7700 1 -- 172.21.15.98:0/3548295677 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f263402df80 con 0x7f2634108a50 2024-04-25T17:32:52.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.102+0000 7f2630ff9700 1 -- 172.21.15.98:0/3548295677 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f262c09e070 con 0x7f2634108a50 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 -- 172.21.15.98:0/3548295677 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26200a3c40 msgr2=0x7f26200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26200a3c40 0x7f26200a6100 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2624009a10 tx=0x7f2624009190 comp rx=0 tx=0).stop 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 -- 172.21.15.98:0/3548295677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2634108a50 msgr2=0x7f263419da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2634108a50 0x7f263419da20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f262c009fd0 tx=0x7f262c007a20 comp rx=0 tx=0).stop 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 -- 172.21.15.98:0/3548295677 shutdown_connections 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26200a3c40 0x7f26200a6100 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2634108a50 0x7f263419da20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f263406b1b0 0x7f26341991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 --2- 172.21.15.98:0/3548295677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f263406bb70 0x7f26341996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 -- 172.21.15.98:0/3548295677 >> 172.21.15.98:0/3548295677 conn(0x7f2634077990 msgr2=0x7f26341006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 -- 172.21.15.98:0/3548295677 shutdown_connections 2024-04-25T17:32:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:52.105+0000 7f26395b7700 1 -- 172.21.15.98:0/3548295677 wait complete. 2024-04-25T17:32:52.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:52 smithi098 ceph-mon[29192]: pgmap v310: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-25T17:32:52.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:52 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3548295677' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T17:32:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:52 smithi203 ceph-mon[32834]: pgmap v310: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-25T17:32:52.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:52 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3548295677' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T17:32:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:52 smithi173 ceph-mon[32856]: pgmap v310: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-25T17:32:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:52 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3548295677' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T17:32:52.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-04-25T17:32:53.142 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:53.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:53 smithi098 ceph-mon[29192]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T17:32:53.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:53 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3548295677' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T17:32:53.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:53 smithi098 ceph-mon[29192]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T17:32:53.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:53 smithi098 ceph-mon[29192]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:53.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:53 smithi098 ceph-mon[29192]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T17:32:53.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:53 smithi098 ceph-mon[29192]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T17:32:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:53 smithi203 ceph-mon[32834]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T17:32:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:53 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3548295677' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T17:32:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:53 smithi203 ceph-mon[32834]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T17:32:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:53 smithi203 ceph-mon[32834]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:53 smithi203 ceph-mon[32834]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T17:32:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:53 smithi203 ceph-mon[32834]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T17:32:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:53 smithi173 ceph-mon[32856]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T17:32:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:53 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3548295677' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T17:32:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:53 smithi173 ceph-mon[32856]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T17:32:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:53 smithi173 ceph-mon[32856]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T17:32:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:53 smithi173 ceph-mon[32856]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T17:32:53.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:53 smithi173 ceph-mon[32856]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T17:32:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:54 smithi098 ceph-mon[29192]: pgmap v311: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 3 op/s 2024-04-25T17:32:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:54 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] up:active 2024-04-25T17:32:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:54 smithi098 ceph-mon[29192]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T17:32:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:54 smithi098 ceph-mon[29192]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T17:32:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:54 smithi098 ceph-mon[29192]: Cluster is now healthy 2024-04-25T17:32:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:54 smithi098 ceph-mon[29192]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T17:32:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:54 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T17:32:54.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:54 smithi098 ceph-mon[29192]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T17:32:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:54 smithi203 ceph-mon[32834]: pgmap v311: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 3 op/s 2024-04-25T17:32:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:54 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] up:active 2024-04-25T17:32:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:54 smithi203 ceph-mon[32834]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T17:32:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:54 smithi203 ceph-mon[32834]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T17:32:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:54 smithi203 ceph-mon[32834]: Cluster is now healthy 2024-04-25T17:32:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:54 smithi203 ceph-mon[32834]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T17:32:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:54 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T17:32:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:54 smithi203 ceph-mon[32834]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T17:32:54.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:54 smithi173 ceph-mon[32856]: pgmap v311: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 3 op/s 2024-04-25T17:32:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:54 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] up:active 2024-04-25T17:32:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:54 smithi173 ceph-mon[32856]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T17:32:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:54 smithi173 ceph-mon[32856]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T17:32:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:54 smithi173 ceph-mon[32856]: Cluster is now healthy 2024-04-25T17:32:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:54 smithi173 ceph-mon[32856]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T17:32:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:54 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T17:32:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:54 smithi173 ceph-mon[32856]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T17:32:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.927+0000 7fba005e4700 1 -- 172.21.15.98:0/2383148689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9f806b110 msgr2=0x7fb9f806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.927+0000 7fba005e4700 1 --2- 172.21.15.98:0/2383148689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9f806b110 0x7fb9f806b590 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009990 tx=0x7fb9e8009ca0 comp rx=0 tx=0).stop 2024-04-25T17:32:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.927+0000 7fba005e4700 1 -- 172.21.15.98:0/2383148689 shutdown_connections 2024-04-25T17:32:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.927+0000 7fba005e4700 1 --2- 172.21.15.98:0/2383148689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9f8108c20 0x7fb9f8109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.927+0000 7fba005e4700 1 --2- 172.21.15.98:0/2383148689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f806bad0 0x7fb9f810f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.927+0000 7fba005e4700 1 --2- 172.21.15.98:0/2383148689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9f806b110 0x7fb9f806b590 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.927+0000 7fba005e4700 1 -- 172.21.15.98:0/2383148689 >> 172.21.15.98:0/2383148689 conn(0x7fb9f80788f0 msgr2=0x7fb9f8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.927+0000 7fba005e4700 1 -- 172.21.15.98:0/2383148689 shutdown_connections 2024-04-25T17:32:54.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.927+0000 7fba005e4700 1 -- 172.21.15.98:0/2383148689 wait complete. 2024-04-25T17:32:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fba005e4700 1 Processor -- start 2024-04-25T17:32:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fba005e4700 1 -- start start 2024-04-25T17:32:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fba005e4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9f806b110 0x7fb9f8199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fba005e4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9f806bad0 0x7fb9f81995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fba005e4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8108c20 0x7fb9f819d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fba005e4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f8112c60 con 0x7fb9f806bad0 2024-04-25T17:32:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fba005e4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f8112ae0 con 0x7fb9f8108c20 2024-04-25T17:32:54.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fba005e4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f8112de0 con 0x7fb9f806b110 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fb9fdb7f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9f806bad0 0x7fb9f81995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fb9fdb7f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9f806bad0 0x7fb9f81995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47948/0 (socket says 172.21.15.98:47948) 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fb9fdb7f700 1 -- 172.21.15.98:0/3557169668 learned_addr learned my addr 172.21.15.98:0/3557169668 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fb9feb81700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8108c20 0x7fb9f819d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.928+0000 7fb9fe380700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9f806b110 0x7fb9f8199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fb9fdb7f700 1 -- 172.21.15.98:0/3557169668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9f806b110 msgr2=0x7fb9f8199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fb9fdb7f700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9f806b110 0x7fb9f8199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fb9fdb7f700 1 -- 172.21.15.98:0/3557169668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8108c20 msgr2=0x7fb9f819d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fb9fdb7f700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8108c20 0x7fb9f819d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fb9fdb7f700 1 -- 172.21.15.98:0/3557169668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f819e080 con 0x7fb9f806bad0 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fb9fdb7f700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9f806bad0 0x7fb9f81995b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8000c00 tx=0x7fb9e8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fb9ef7fe700 1 -- 172.21.15.98:0/3557169668 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8027070 con 0x7fb9f806bad0 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fba005e4700 1 -- 172.21.15.98:0/3557169668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f819e310 con 0x7fb9f806bad0 2024-04-25T17:32:54.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fb9ef7fe700 1 -- 172.21.15.98:0/3557169668 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9e8022470 con 0x7fb9f806bad0 2024-04-25T17:32:54.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fb9ef7fe700 1 -- 172.21.15.98:0/3557169668 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8017650 con 0x7fb9f806bad0 2024-04-25T17:32:54.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.929+0000 7fba005e4700 1 -- 172.21.15.98:0/3557169668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f81a5bf0 con 0x7fb9f806bad0 2024-04-25T17:32:54.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.930+0000 7fba005e4700 1 -- 172.21.15.98:0/3557169668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9f8104190 con 0x7fb9f806bad0 2024-04-25T17:32:54.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.932+0000 7fb9ef7fe700 1 -- 172.21.15.98:0/3557169668 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb9e802a070 con 0x7fb9f806bad0 2024-04-25T17:32:54.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.932+0000 7fb9ef7fe700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb9e40a3c40 0x7fb9e40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:54.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.933+0000 7fb9fe380700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb9e40a3c40 0x7fb9e40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:54.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.933+0000 7fb9ef7fe700 1 -- 172.21.15.98:0/3557169668 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fb9e80d11e0 con 0x7fb9f806bad0 2024-04-25T17:32:54.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.933+0000 7fb9fe380700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb9e40a3c40 0x7fb9e40a6100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4005950 tx=0x7fb9f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:54.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:54.936+0000 7fb9ef7fe700 1 -- 172.21.15.98:0/3557169668 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9e8013070 con 0x7fb9f806bad0 2024-04-25T17:32:55.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:55.184+0000 7fba005e4700 1 -- 172.21.15.98:0/3557169668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fb9f819a3d0 con 0x7fb9f806bad0 2024-04-25T17:32:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:55 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] up:active 2024-04-25T17:32:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:55 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T17:32:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:55 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] up:active 2024-04-25T17:32:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:55 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T17:32:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:55 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] up:active 2024-04-25T17:32:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:55 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T17:32:55.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:32:55 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b[63135]: 2024-04-25T17:32:55.029+0000 7f00986b2700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T17:32:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.133+0000 7fb9ef7fe700 1 -- 172.21.15.98:0/3557169668 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7fb9e809dda0 con 0x7fb9f806bad0 2024-04-25T17:32:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 -- 172.21.15.98:0/3557169668 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb9e40a3c40 msgr2=0x7fb9e40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb9e40a3c40 0x7fb9e40a6100 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4005950 tx=0x7fb9f400b410 comp rx=0 tx=0).stop 2024-04-25T17:32:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 -- 172.21.15.98:0/3557169668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9f806bad0 msgr2=0x7fb9f81995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9f806bad0 0x7fb9f81995b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8000c00 tx=0x7fb9e8009580 comp rx=0 tx=0).stop 2024-04-25T17:32:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 -- 172.21.15.98:0/3557169668 shutdown_connections 2024-04-25T17:32:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb9e40a3c40 0x7fb9e40a6100 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9f806b110 0x7fb9f8199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8108c20 0x7fb9f819d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 --2- 172.21.15.98:0/3557169668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9f806bad0 0x7fb9f81995b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 -- 172.21.15.98:0/3557169668 >> 172.21.15.98:0/3557169668 conn(0x7fb9f80788f0 msgr2=0x7fb9f8100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 -- 172.21.15.98:0/3557169668 shutdown_connections 2024-04-25T17:32:56.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:56.136+0000 7fba005e4700 1 -- 172.21.15.98:0/3557169668 wait complete. 2024-04-25T17:32:56.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:56 smithi098 ceph-mon[29192]: pgmap v312: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-04-25T17:32:56.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:56 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3557169668' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T17:32:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:56 smithi203 ceph-mon[32834]: pgmap v312: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-04-25T17:32:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:56 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3557169668' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T17:32:56.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:56 smithi173 ceph-mon[32856]: pgmap v312: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 2 op/s 2024-04-25T17:32:56.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:56 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3557169668' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T17:32:56.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-04-25T17:32:57.176 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:32:57.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:57 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3557169668' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T17:32:57.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:57 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T17:32:57.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:57 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:32:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:57 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3557169668' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T17:32:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:57 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T17:32:57.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:57 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:32:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:57 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3557169668' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T17:32:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:57 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T17:32:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:57 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:32:58.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:32:58 smithi098 ceph-mon[29192]: pgmap v313: 97 pgs: 97 active+clean; 580 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-25T17:32:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:32:58 smithi203 ceph-mon[32834]: pgmap v313: 97 pgs: 97 active+clean; 580 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-25T17:32:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:32:58 smithi173 ceph-mon[32856]: pgmap v313: 97 pgs: 97 active+clean; 580 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-25T17:32:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.058+0000 7f6e2c756700 1 -- 172.21.15.98:0/2993103798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24108a10 msgr2=0x7f6e24108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.058+0000 7f6e2c756700 1 --2- 172.21.15.98:0/2993103798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24108a10 0x7f6e24108df0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009990 tx=0x7f6e20009ca0 comp rx=0 tx=0).stop 2024-04-25T17:32:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.058+0000 7f6e2c756700 1 -- 172.21.15.98:0/2993103798 shutdown_connections 2024-04-25T17:32:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.058+0000 7f6e2c756700 1 --2- 172.21.15.98:0/2993103798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e241034c0 0x7f6e2410f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.058+0000 7f6e2c756700 1 --2- 172.21.15.98:0/2993103798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24108a10 0x7f6e24108df0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.058+0000 7f6e2c756700 1 --2- 172.21.15.98:0/2993103798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e24102b00 0x7f6e24102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.058+0000 7f6e2c756700 1 -- 172.21.15.98:0/2993103798 >> 172.21.15.98:0/2993103798 conn(0x7f6e24079c00 msgr2=0x7f6e2407a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.058+0000 7f6e2c756700 1 -- 172.21.15.98:0/2993103798 shutdown_connections 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.058+0000 7f6e2c756700 1 -- 172.21.15.98:0/2993103798 wait complete. 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2c756700 1 Processor -- start 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2c756700 1 -- start start 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2c756700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e24102b00 0x7f6e24074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2c756700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e241034c0 0x7f6e24075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2c756700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24108a10 0x7f6e240757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2c756700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e24112bf0 con 0x7f6e24108a10 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2c756700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e24112a70 con 0x7f6e241034c0 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2c756700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e24112d70 con 0x7f6e24102b00 2024-04-25T17:32:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2acf3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24108a10 0x7f6e240757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2acf3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24108a10 0x7f6e240757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47968/0 (socket says 172.21.15.98:47968) 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2acf3700 1 -- 172.21.15.98:0/824959469 learned_addr learned my addr 172.21.15.98:0/824959469 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e29cf1700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e241034c0 0x7f6e24075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2a4f2700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e24102b00 0x7f6e24074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2acf3700 1 -- 172.21.15.98:0/824959469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e24102b00 msgr2=0x7f6e24074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2acf3700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e24102b00 0x7f6e24074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2acf3700 1 -- 172.21.15.98:0/824959469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e241034c0 msgr2=0x7f6e24075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2acf3700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e241034c0 0x7f6e24075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.059+0000 7f6e2acf3700 1 -- 172.21.15.98:0/824959469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e241a7570 con 0x7f6e24108a10 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.060+0000 7f6e2acf3700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24108a10 0x7f6e240757c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c007d70 tx=0x7f6e1c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.060+0000 7f6e1b7fe700 1 -- 172.21.15.98:0/824959469 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c016020 con 0x7f6e24108a10 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.060+0000 7f6e2c756700 1 -- 172.21.15.98:0/824959469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e241a7860 con 0x7f6e24108a10 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.060+0000 7f6e1b7fe700 1 -- 172.21.15.98:0/824959469 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e1c00ac90 con 0x7f6e24108a10 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.060+0000 7f6e1b7fe700 1 -- 172.21.15.98:0/824959469 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c0115e0 con 0x7f6e24108a10 2024-04-25T17:32:59.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.060+0000 7f6e2c756700 1 -- 172.21.15.98:0/824959469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e241a7d50 con 0x7f6e24108a10 2024-04-25T17:32:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.061+0000 7f6e2c756700 1 -- 172.21.15.98:0/824959469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e2410afa0 con 0x7f6e24108a10 2024-04-25T17:32:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.062+0000 7f6e1b7fe700 1 -- 172.21.15.98:0/824959469 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6e1c019020 con 0x7f6e24108a10 2024-04-25T17:32:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.063+0000 7f6e1b7fe700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e100a3c40 0x7f6e100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:32:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.063+0000 7f6e2a4f2700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e100a3c40 0x7f6e100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:32:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.063+0000 7f6e1b7fe700 1 -- 172.21.15.98:0/824959469 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f6e1c0d0620 con 0x7f6e24108a10 2024-04-25T17:32:59.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.064+0000 7f6e2a4f2700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e100a3c40 0x7f6e100a6100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6e200098e0 tx=0x7f6e20005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:32:59.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.067+0000 7f6e1b7fe700 1 -- 172.21.15.98:0/824959469 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e1c020020 con 0x7f6e24108a10 2024-04-25T17:32:59.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:32:59.313+0000 7f6e2c756700 1 -- 172.21.15.98:0/824959469 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f6e24076270 con 0x7f6e24108a10 2024-04-25T17:33:00.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.154+0000 7f6e1b7fe700 1 -- 172.21.15.98:0/824959469 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7f6e1c01ae40 con 0x7f6e24108a10 2024-04-25T17:33:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.156+0000 7f6e2c756700 1 -- 172.21.15.98:0/824959469 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e100a3c40 msgr2=0x7f6e100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.156+0000 7f6e2c756700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e100a3c40 0x7f6e100a6100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6e200098e0 tx=0x7f6e20005960 comp rx=0 tx=0).stop 2024-04-25T17:33:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.156+0000 7f6e2c756700 1 -- 172.21.15.98:0/824959469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24108a10 msgr2=0x7f6e240757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.156+0000 7f6e2c756700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24108a10 0x7f6e240757c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c007d70 tx=0x7f6e1c00a340 comp rx=0 tx=0).stop 2024-04-25T17:33:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.157+0000 7f6e2c756700 1 -- 172.21.15.98:0/824959469 shutdown_connections 2024-04-25T17:33:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.157+0000 7f6e2c756700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e100a3c40 0x7f6e100a6100 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.157+0000 7f6e2c756700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e24102b00 0x7f6e24074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.157+0000 7f6e2c756700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e24108a10 0x7f6e240757c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.157+0000 7f6e2c756700 1 --2- 172.21.15.98:0/824959469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e241034c0 0x7f6e24075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.157+0000 7f6e2c756700 1 -- 172.21.15.98:0/824959469 >> 172.21.15.98:0/824959469 conn(0x7f6e24079c00 msgr2=0x7f6e24100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.157+0000 7f6e2c756700 1 -- 172.21.15.98:0/824959469 shutdown_connections 2024-04-25T17:33:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:00.157+0000 7f6e2c756700 1 -- 172.21.15.98:0/824959469 wait complete. 2024-04-25T17:33:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:00 smithi098 ceph-mon[29192]: pgmap v314: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-25T17:33:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:00 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/824959469' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T17:33:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:00 smithi203 ceph-mon[32834]: pgmap v314: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-25T17:33:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:00 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/824959469' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T17:33:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:00 smithi173 ceph-mon[32856]: pgmap v314: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-25T17:33:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:00 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/824959469' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T17:33:00.948 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-25T17:33:00.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-04-25T17:33:01.201 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:01.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:01 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/824959469' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T17:33:01.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:01 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:33:01.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:01 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/824959469' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T17:33:01.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:01 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:33:01.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:01 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/824959469' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T17:33:01.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:01 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:33:02.361 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:02 smithi098 ceph-mon[29192]: pgmap v315: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-25T17:33:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:02 smithi203 ceph-mon[32834]: pgmap v315: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-25T17:33:02.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:02 smithi173 ceph-mon[32856]: pgmap v315: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-25T17:33:02.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.902+0000 7f7351ca1700 1 -- 172.21.15.98:0/3818631663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f734c107920 msgr2=0x7f734c107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:02.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.902+0000 7f7351ca1700 1 --2- 172.21.15.98:0/3818631663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f734c107920 0x7f734c107d00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f733c009990 tx=0x7f733c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:33:02.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.903+0000 7f7351ca1700 1 -- 172.21.15.98:0/3818631663 shutdown_connections 2024-04-25T17:33:02.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.903+0000 7f7351ca1700 1 --2- 172.21.15.98:0/3818631663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c102330 0x7f734c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:02.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.903+0000 7f7351ca1700 1 --2- 172.21.15.98:0/3818631663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f734c107920 0x7f734c107d00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:02.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.903+0000 7f7351ca1700 1 --2- 172.21.15.98:0/3818631663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f734c101970 0x7f734c101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:02.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.903+0000 7f7351ca1700 1 -- 172.21.15.98:0/3818631663 >> 172.21.15.98:0/3818631663 conn(0x7f734c0788f0 msgr2=0x7f734c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:02.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.903+0000 7f7351ca1700 1 -- 172.21.15.98:0/3818631663 shutdown_connections 2024-04-25T17:33:02.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.903+0000 7f7351ca1700 1 -- 172.21.15.98:0/3818631663 wait complete. 2024-04-25T17:33:02.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.903+0000 7f7351ca1700 1 Processor -- start 2024-04-25T17:33:02.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.903+0000 7f7351ca1700 1 -- start start 2024-04-25T17:33:02.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f7351ca1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f734c101970 0x7f734c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:02.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f7351ca1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f734c102330 0x7f734c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:02.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f7351ca1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c107920 0x7f734c19c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:02.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f7351ca1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f734c111b00 con 0x7f734c101970 2024-04-25T17:33:02.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f7351ca1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f734c111980 con 0x7f734c102330 2024-04-25T17:33:02.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f7351ca1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f734c111c80 con 0x7f734c107920 2024-04-25T17:33:02.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f734c101970 0x7f734c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:02.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f734c101970 0x7f734c198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47982/0 (socket says 172.21.15.98:47982) 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734b7fe700 1 -- 172.21.15.98:0/2095053672 learned_addr learned my addr 172.21.15.98:0/2095053672 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734affd700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f734c102330 0x7f734c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734bfff700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c107920 0x7f734c19c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734b7fe700 1 -- 172.21.15.98:0/2095053672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c107920 msgr2=0x7f734c19c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734b7fe700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c107920 0x7f734c19c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734b7fe700 1 -- 172.21.15.98:0/2095053672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f734c102330 msgr2=0x7f734c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734b7fe700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f734c102330 0x7f734c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734b7fe700 1 -- 172.21.15.98:0/2095053672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734c19d040 con 0x7f734c101970 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734bfff700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c107920 0x7f734c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.904+0000 7f734b7fe700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f734c101970 0x7f734c198030 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f733c000c00 tx=0x7f733c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.905+0000 7f7348ff9700 1 -- 172.21.15.98:0/2095053672 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c027070 con 0x7f734c101970 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.905+0000 7f7351ca1700 1 -- 172.21.15.98:0/2095053672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f734c19d2d0 con 0x7f734c101970 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.905+0000 7f7348ff9700 1 -- 172.21.15.98:0/2095053672 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f733c0040e0 con 0x7f734c101970 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.905+0000 7f7348ff9700 1 -- 172.21.15.98:0/2095053672 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c00f460 con 0x7f734c101970 2024-04-25T17:33:02.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.905+0000 7f7351ca1700 1 -- 172.21.15.98:0/2095053672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f734c1a4bb0 con 0x7f734c101970 2024-04-25T17:33:02.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.906+0000 7f7351ca1700 1 -- 172.21.15.98:0/2095053672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f734c103060 con 0x7f734c101970 2024-04-25T17:33:02.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.907+0000 7f7348ff9700 1 -- 172.21.15.98:0/2095053672 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f733c02a070 con 0x7f734c101970 2024-04-25T17:33:02.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.907+0000 7f7348ff9700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73340a3c40 0x7f73340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:02.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.908+0000 7f734affd700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73340a3c40 0x7f73340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:02.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.908+0000 7f7348ff9700 1 -- 172.21.15.98:0/2095053672 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f733c0d16c0 con 0x7f734c101970 2024-04-25T17:33:02.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.908+0000 7f734affd700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73340a3c40 0x7f73340a6100 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f734c078f60 tx=0x7f734000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:02.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:02.912+0000 7f7348ff9700 1 -- 172.21.15.98:0/2095053672 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f733c031070 con 0x7f734c101970 2024-04-25T17:33:03.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.142+0000 7f7351ca1700 1 -- 172.21.15.98:0/2095053672 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f734c199390 con 0x7f73340a3c40 2024-04-25T17:33:03.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.165+0000 7f7348ff9700 1 -- 172.21.15.98:0/2095053672 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f734c199390 con 0x7f73340a3c40 2024-04-25T17:33:03.167 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:33:03 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:33:03.146+0000 7f0d6f47b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:33:03.167 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:33:03 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:33:03.146+0000 7f0d6f47b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:33:03.167 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:33:03 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:33:03.146+0000 7f0d6f47b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:33:03.167 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:33:03 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:33:03.146+0000 7f0d6f47b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:33:03.168 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:33:03 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:33:03.146+0000 7f0d6f47b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:33:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 -- 172.21.15.98:0/2095053672 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73340a3c40 msgr2=0x7f73340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73340a3c40 0x7f73340a6100 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f734c078f60 tx=0x7f734000a300 comp rx=0 tx=0).stop 2024-04-25T17:33:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 -- 172.21.15.98:0/2095053672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f734c101970 msgr2=0x7f734c198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f734c101970 0x7f734c198030 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f733c000c00 tx=0x7f733c0047b0 comp rx=0 tx=0).stop 2024-04-25T17:33:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 -- 172.21.15.98:0/2095053672 shutdown_connections 2024-04-25T17:33:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f73340a3c40 0x7f73340a6100 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c107920 0x7f734c19c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f734c101970 0x7f734c198030 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:03.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 --2- 172.21.15.98:0/2095053672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f734c102330 0x7f734c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 -- 172.21.15.98:0/2095053672 >> 172.21.15.98:0/2095053672 conn(0x7f734c0788f0 msgr2=0x7f734c0ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 -- 172.21.15.98:0/2095053672 shutdown_connections 2024-04-25T17:33:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:03.168+0000 7f7351ca1700 1 -- 172.21.15.98:0/2095053672 wait complete. 2024-04-25T17:33:03.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T17:33:04.066 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:04.442 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:04 smithi098 ceph-mon[29192]: pgmap v316: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-25T17:33:04.442 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:04 smithi098 ceph-mon[29192]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:04 smithi203 ceph-mon[32834]: pgmap v316: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-25T17:33:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:04 smithi203 ceph-mon[32834]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:04.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:04 smithi173 ceph-mon[32856]: pgmap v316: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-25T17:33:04.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:04 smithi173 ceph-mon[32856]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.814+0000 7f0156cfe700 1 -- 172.21.15.98:0/2248471026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0150100f90 msgr2=0x7f0150101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.814+0000 7f0156cfe700 1 --2- 172.21.15.98:0/2248471026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0150100f90 0x7f0150101410 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f013c0099c0 tx=0x7f013c009cd0 comp rx=0 tx=0).stop 2024-04-25T17:33:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.815+0000 7f0156cfe700 1 -- 172.21.15.98:0/2248471026 shutdown_connections 2024-04-25T17:33:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.815+0000 7f0156cfe700 1 --2- 172.21.15.98:0/2248471026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0150101950 0x7f015010f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.815+0000 7f0156cfe700 1 --2- 172.21.15.98:0/2248471026 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0150107a60 0x7f0150107e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.815+0000 7f0156cfe700 1 --2- 172.21.15.98:0/2248471026 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0150100f90 0x7f0150101410 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.815+0000 7f0156cfe700 1 -- 172.21.15.98:0/2248471026 >> 172.21.15.98:0/2248471026 conn(0x7f0150077f40 msgr2=0x7f01500ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.815+0000 7f0156cfe700 1 -- 172.21.15.98:0/2248471026 shutdown_connections 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.815+0000 7f0156cfe700 1 -- 172.21.15.98:0/2248471026 wait complete. 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.815+0000 7f0156cfe700 1 Processor -- start 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.815+0000 7f0156cfe700 1 -- start start 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.815+0000 7f0156cfe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0150100f90 0x7f0150102170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f0156cfe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0150101950 0x7f01501026b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f0156cfe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0150107a60 0x7f0150102bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f0156cfe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0150112d70 con 0x7f0150107a60 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f0156cfe700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0150112bf0 con 0x7f0150101950 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f0156cfe700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0150112ef0 con 0x7f0150100f90 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f015529b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0150107a60 0x7f0150102bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f0154a9a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0150100f90 0x7f0150102170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f015529b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0150107a60 0x7f0150102bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42970/0 (socket says 172.21.15.98:42970) 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f015529b700 1 -- 172.21.15.98:0/1530827504 learned_addr learned my addr 172.21.15.98:0/1530827504 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f014ffff700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0150101950 0x7f01501026b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f015529b700 1 -- 172.21.15.98:0/1530827504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0150100f90 msgr2=0x7f0150102170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f015529b700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0150100f90 0x7f0150102170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f015529b700 1 -- 172.21.15.98:0/1530827504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0150101950 msgr2=0x7f01501026b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f015529b700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0150101950 0x7f01501026b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f015529b700 1 -- 172.21.15.98:0/1530827504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01501a7700 con 0x7f0150107a60 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f015529b700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0150107a60 0x7f0150102bf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f014800b9e0 tx=0x7f014800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f014dffb700 1 -- 172.21.15.98:0/1530827504 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014800d820 con 0x7f0150107a60 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f0156cfe700 1 -- 172.21.15.98:0/1530827504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01501a79f0 con 0x7f0150107a60 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.816+0000 7f014dffb700 1 -- 172.21.15.98:0/1530827504 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0148015420 con 0x7f0150107a60 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.817+0000 7f014dffb700 1 -- 172.21.15.98:0/1530827504 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0148014610 con 0x7f0150107a60 2024-04-25T17:33:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.817+0000 7f0156cfe700 1 -- 172.21.15.98:0/1530827504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01501a7eb0 con 0x7f0150107a60 2024-04-25T17:33:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.818+0000 7f0156cfe700 1 -- 172.21.15.98:0/1530827504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0150103650 con 0x7f0150107a60 2024-04-25T17:33:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.819+0000 7f014dffb700 1 -- 172.21.15.98:0/1530827504 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f01480147b0 con 0x7f0150107a60 2024-04-25T17:33:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.820+0000 7f014dffb700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01400a3cd0 0x7f01400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.820+0000 7f0154a9a700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01400a3cd0 0x7f01400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.820+0000 7f014dffb700 1 -- 172.21.15.98:0/1530827504 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f01480cfc30 con 0x7f0150107a60 2024-04-25T17:33:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.820+0000 7f0154a9a700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01400a3cd0 0x7f01400a6190 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0144005950 tx=0x7f014400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:05.824+0000 7f014dffb700 1 -- 172.21.15.98:0/1530827504 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f014809c660 con 0x7f0150107a60 2024-04-25T17:33:06.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.055+0000 7f0156cfe700 1 -- 172.21.15.98:0/1530827504 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0150062e80 con 0x7f01400a3cd0 2024-04-25T17:33:06.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.080+0000 7f014dffb700 1 -- 172.21.15.98:0/1530827504 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0150062e80 con 0x7f01400a3cd0 2024-04-25T17:33:06.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 -- 172.21.15.98:0/1530827504 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01400a3cd0 msgr2=0x7f01400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:06.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01400a3cd0 0x7f01400a6190 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0144005950 tx=0x7f014400b410 comp rx=0 tx=0).stop 2024-04-25T17:33:06.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 -- 172.21.15.98:0/1530827504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0150107a60 msgr2=0x7f0150102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:06.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0150107a60 0x7f0150102bf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f014800b9e0 tx=0x7f014800bcf0 comp rx=0 tx=0).stop 2024-04-25T17:33:06.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 -- 172.21.15.98:0/1530827504 shutdown_connections 2024-04-25T17:33:06.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f01400a3cd0 0x7f01400a6190 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:06.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0150100f90 0x7f0150102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:06.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0150107a60 0x7f0150102bf0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:06.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 --2- 172.21.15.98:0/1530827504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0150101950 0x7f01501026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:06.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 -- 172.21.15.98:0/1530827504 >> 172.21.15.98:0/1530827504 conn(0x7f0150077f40 msgr2=0x7f0150105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:06.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 -- 172.21.15.98:0/1530827504 shutdown_connections 2024-04-25T17:33:06.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:06.082+0000 7f0156cfe700 1 -- 172.21.15.98:0/1530827504 wait complete. 2024-04-25T17:33:06.373 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:06 smithi098 ceph-mon[29192]: pgmap v317: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T17:33:06.373 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:06 smithi098 ceph-mon[29192]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T17:33:06.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:06 smithi203 ceph-mon[32834]: pgmap v317: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T17:33:06.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:06 smithi203 ceph-mon[32834]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T17:33:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:06 smithi173 ceph-mon[32856]: pgmap v317: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T17:33:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:06 smithi173 ceph-mon[32856]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T17:33:06.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T17:33:07.019 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:07.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:07 smithi098 ceph-mon[29192]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:07.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:07 smithi203 ceph-mon[32834]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:07.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:07 smithi173 ceph-mon[32856]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:08.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:08 smithi098 ceph-mon[29192]: pgmap v318: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-25T17:33:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:08 smithi203 ceph-mon[32834]: pgmap v318: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-25T17:33:08.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:08 smithi173 ceph-mon[32856]: pgmap v318: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-25T17:33:08.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.783+0000 7fafdf5e5700 1 -- 172.21.15.98:0/1163806150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd8107990 msgr2=0x7fafd8107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.783+0000 7fafdf5e5700 1 --2- 172.21.15.98:0/1163806150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd8107990 0x7fafd8107d70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009990 tx=0x7fafc8009ca0 comp rx=0 tx=0).stop 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.783+0000 7fafdf5e5700 1 -- 172.21.15.98:0/1163806150 shutdown_connections 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.783+0000 7fafdf5e5700 1 --2- 172.21.15.98:0/1163806150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafd8102300 0x7fafd810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.783+0000 7fafdf5e5700 1 --2- 172.21.15.98:0/1163806150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd8107990 0x7fafd8107d70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.783+0000 7fafdf5e5700 1 --2- 172.21.15.98:0/1163806150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafd8101940 0x7fafd8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.783+0000 7fafdf5e5700 1 -- 172.21.15.98:0/1163806150 >> 172.21.15.98:0/1163806150 conn(0x7fafd80788f0 msgr2=0x7fafd8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.783+0000 7fafdf5e5700 1 -- 172.21.15.98:0/1163806150 shutdown_connections 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.783+0000 7fafdf5e5700 1 -- 172.21.15.98:0/1163806150 wait complete. 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdf5e5700 1 Processor -- start 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdf5e5700 1 -- start start 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdf5e5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd8101940 0x7fafd819b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdf5e5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafd8102300 0x7fafd819b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:08.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdf5e5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafd8107990 0x7fafd819fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdf5e5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8111bd0 con 0x7fafd8101940 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdf5e5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8111a50 con 0x7fafd8107990 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdf5e5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8111d50 con 0x7fafd8102300 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdd381700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd8101940 0x7fafd819b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdd381700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd8101940 0x7fafd819b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42988/0 (socket says 172.21.15.98:42988) 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdd381700 1 -- 172.21.15.98:0/758616105 learned_addr learned my addr 172.21.15.98:0/758616105 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafddb82700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafd8107990 0x7fafd819fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdcb80700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafd8102300 0x7fafd819b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdd381700 1 -- 172.21.15.98:0/758616105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafd8102300 msgr2=0x7fafd819b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdd381700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafd8102300 0x7fafd819b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdd381700 1 -- 172.21.15.98:0/758616105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafd8107990 msgr2=0x7fafd819fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdd381700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafd8107990 0x7fafd819fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdd381700 1 -- 172.21.15.98:0/758616105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd81a0480 con 0x7fafd8101940 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafddb82700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafd8107990 0x7fafd819fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:33:08.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.784+0000 7fafdcb80700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafd8102300 0x7fafd819b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:33:08.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.785+0000 7fafdd381700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd8101940 0x7fafd819b470 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fafc8000c00 tx=0x7fafc80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:08.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.785+0000 7fafce7fc700 1 -- 172.21.15.98:0/758616105 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8027070 con 0x7fafd8101940 2024-04-25T17:33:08.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.785+0000 7fafdf5e5700 1 -- 172.21.15.98:0/758616105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd81a0710 con 0x7fafd8101940 2024-04-25T17:33:08.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.785+0000 7fafce7fc700 1 -- 172.21.15.98:0/758616105 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc80041d0 con 0x7fafd8101940 2024-04-25T17:33:08.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.785+0000 7fafce7fc700 1 -- 172.21.15.98:0/758616105 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc800f460 con 0x7fafd8101940 2024-04-25T17:33:08.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.785+0000 7fafdf5e5700 1 -- 172.21.15.98:0/758616105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd81a7ff0 con 0x7fafd8101940 2024-04-25T17:33:08.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.786+0000 7fafdf5e5700 1 -- 172.21.15.98:0/758616105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd8069730 con 0x7fafd8101940 2024-04-25T17:33:08.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.787+0000 7fafce7fc700 1 -- 172.21.15.98:0/758616105 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fafc802a070 con 0x7fafd8101940 2024-04-25T17:33:08.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.788+0000 7fafce7fc700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fafc40a3cd0 0x7fafc40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.788+0000 7fafdcb80700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fafc40a3cd0 0x7fafc40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.788+0000 7fafce7fc700 1 -- 172.21.15.98:0/758616105 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7fafc80d13c0 con 0x7fafd8101940 2024-04-25T17:33:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.788+0000 7fafdcb80700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fafc40a3cd0 0x7fafc40a6190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fafd8078da0 tx=0x7fafd4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:08.792+0000 7fafce7fc700 1 -- 172.21.15.98:0/758616105 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafc805eff0 con 0x7fafd8101940 2024-04-25T17:33:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.027+0000 7fafdf5e5700 1 -- 172.21.15.98:0/758616105 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fafd806aef0 con 0x7fafc40a3cd0 2024-04-25T17:33:09.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.033+0000 7fafce7fc700 1 -- 172.21.15.98:0/758616105 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fafd806aef0 con 0x7fafc40a3cd0 2024-04-25T17:33:09.034 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_0/c68f0dc3-9ccb-47d0-b8e2-9ded53b212f7 2024-04-25T17:33:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 -- 172.21.15.98:0/758616105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fafc40a3cd0 msgr2=0x7fafc40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:09.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fafc40a3cd0 0x7fafc40a6190 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fafd8078da0 tx=0x7fafd4008040 comp rx=0 tx=0).stop 2024-04-25T17:33:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 -- 172.21.15.98:0/758616105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd8101940 msgr2=0x7fafd819b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd8101940 0x7fafd819b470 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fafc8000c00 tx=0x7fafc80047b0 comp rx=0 tx=0).stop 2024-04-25T17:33:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 -- 172.21.15.98:0/758616105 shutdown_connections 2024-04-25T17:33:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fafc40a3cd0 0x7fafc40a6190 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafd8107990 0x7fafd819fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd8101940 0x7fafd819b470 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 --2- 172.21.15.98:0/758616105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafd8102300 0x7fafd819b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 -- 172.21.15.98:0/758616105 >> 172.21.15.98:0/758616105 conn(0x7fafd80788f0 msgr2=0x7fafd806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 -- 172.21.15.98:0/758616105 shutdown_connections 2024-04-25T17:33:09.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:09.035+0000 7fafdf5e5700 1 -- 172.21.15.98:0/758616105 wait complete. 2024-04-25T17:33:09.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]: dispatch 2024-04-25T17:33:09.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T17:33:09.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-25T17:33:09.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-25T17:33:09.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T17:33:09.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-25T17:33:09.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-25T17:33:09.408 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:09 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-25T17:33:09.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]: dispatch 2024-04-25T17:33:09.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T17:33:09.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-25T17:33:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-25T17:33:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T17:33:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-25T17:33:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-25T17:33:09.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:09 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-25T17:33:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]: dispatch 2024-04-25T17:33:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T17:33:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-25T17:33:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-25T17:33:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T17:33:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-25T17:33:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-25T17:33:09.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:09 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-25T17:33:09.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T17:33:09.999 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:10.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: pgmap v319: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-04-25T17:33:10.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:10.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]': finished 2024-04-25T17:33:10.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T17:33:10.447 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-25T17:33:10.447 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-25T17:33:10.447 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T17:33:10.447 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-25T17:33:10.447 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-25T17:33:10.447 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-25T17:33:10.447 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:10 smithi098 ceph-mon[29192]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T17:33:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: pgmap v319: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-04-25T17:33:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]': finished 2024-04-25T17:33:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T17:33:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-25T17:33:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-25T17:33:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T17:33:10.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-25T17:33:10.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-25T17:33:10.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-25T17:33:10.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:10 smithi203 ceph-mon[32834]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T17:33:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: pgmap v319: 97 pgs: 97 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-04-25T17:33:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]': finished 2024-04-25T17:33:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T17:33:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-25T17:33:10.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-25T17:33:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T17:33:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-25T17:33:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-25T17:33:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-25T17:33:10.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:10 smithi173 ceph-mon[32856]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T17:33:11.437 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:11 smithi098 ceph-mon[29192]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T17:33:11.437 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:11 smithi098 ceph-mon[29192]: pgmap v322: 97 pgs: 4 peering, 93 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-04-25T17:33:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:11 smithi203 ceph-mon[32834]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T17:33:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:11 smithi203 ceph-mon[32834]: pgmap v322: 97 pgs: 4 peering, 93 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-04-25T17:33:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:11 smithi173 ceph-mon[32856]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T17:33:11.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:11 smithi173 ceph-mon[32856]: pgmap v322: 97 pgs: 4 peering, 93 active+clean; 602 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-04-25T17:33:11.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.841+0000 7f569f0d7700 1 -- 172.21.15.98:0/2624695957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5698102a90 msgr2=0x7f5698102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.841+0000 7f569f0d7700 1 --2- 172.21.15.98:0/2624695957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5698102a90 0x7f5698102f10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5684009990 tx=0x7f5684009ca0 comp rx=0 tx=0).stop 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.841+0000 7f569f0d7700 1 -- 172.21.15.98:0/2624695957 shutdown_connections 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.841+0000 7f569f0d7700 1 --2- 172.21.15.98:0/2624695957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5698108a50 0x7f5698108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.841+0000 7f569f0d7700 1 --2- 172.21.15.98:0/2624695957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5698103450 0x7f569810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.841+0000 7f569f0d7700 1 --2- 172.21.15.98:0/2624695957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5698102a90 0x7f5698102f10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.841+0000 7f569f0d7700 1 -- 172.21.15.98:0/2624695957 >> 172.21.15.98:0/2624695957 conn(0x7f5698079ba0 msgr2=0x7f5698079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.841+0000 7f569f0d7700 1 -- 172.21.15.98:0/2624695957 shutdown_connections 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.841+0000 7f569f0d7700 1 -- 172.21.15.98:0/2624695957 wait complete. 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569f0d7700 1 Processor -- start 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569f0d7700 1 -- start start 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569f0d7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5698102a90 0x7f5698199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:11.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569f0d7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5698103450 0x7f56981995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569f0d7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5698108a50 0x7f569819d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569f0d7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5698112c00 con 0x7f5698102a90 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569f0d7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5698112a80 con 0x7f5698108a50 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569f0d7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5698112d80 con 0x7f5698103450 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569ce73700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5698102a90 0x7f5698199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569ce73700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5698102a90 0x7f5698199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43008/0 (socket says 172.21.15.98:43008) 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.842+0000 7f569ce73700 1 -- 172.21.15.98:0/1496651924 learned_addr learned my addr 172.21.15.98:0/1496651924 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f5697fff700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5698103450 0x7f56981995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f569d674700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5698108a50 0x7f569819d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f569ce73700 1 -- 172.21.15.98:0/1496651924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5698103450 msgr2=0x7f56981995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f569ce73700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5698103450 0x7f56981995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f569ce73700 1 -- 172.21.15.98:0/1496651924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5698108a50 msgr2=0x7f569819d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f569ce73700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5698108a50 0x7f569819d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:11.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f569ce73700 1 -- 172.21.15.98:0/1496651924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569819e0a0 con 0x7f5698102a90 2024-04-25T17:33:11.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f5697fff700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5698103450 0x7f56981995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:33:11.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f569d674700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5698108a50 0x7f569819d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:33:11.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f569ce73700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5698102a90 0x7f5698199090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f568c00c970 tx=0x7f568c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:11.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f5695ffb700 1 -- 172.21.15.98:0/1496651924 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c007aa0 con 0x7f5698102a90 2024-04-25T17:33:11.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f569f0d7700 1 -- 172.21.15.98:0/1496651924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f569819e390 con 0x7f5698102a90 2024-04-25T17:33:11.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f5695ffb700 1 -- 172.21.15.98:0/1496651924 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f568c004ce0 con 0x7f5698102a90 2024-04-25T17:33:11.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f5695ffb700 1 -- 172.21.15.98:0/1496651924 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c0055c0 con 0x7f5698102a90 2024-04-25T17:33:11.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.843+0000 7f569f0d7700 1 -- 172.21.15.98:0/1496651924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56981a5c10 con 0x7f5698102a90 2024-04-25T17:33:11.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.844+0000 7f569f0d7700 1 -- 172.21.15.98:0/1496651924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5698069730 con 0x7f5698102a90 2024-04-25T17:33:11.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.845+0000 7f5695ffb700 1 -- 172.21.15.98:0/1496651924 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f568c007c40 con 0x7f5698102a90 2024-04-25T17:33:11.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.846+0000 7f5695ffb700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f56880a3cd0 0x7f56880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:11.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.846+0000 7f5697fff700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f56880a3cd0 0x7f56880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:11.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.846+0000 7f5695ffb700 1 -- 172.21.15.98:0/1496651924 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f568c0d0320 con 0x7f5698102a90 2024-04-25T17:33:11.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.846+0000 7f5697fff700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f56880a3cd0 0x7f56880a6190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f568400b5c0 tx=0x7f5684005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:11.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:11.850+0000 7f5695ffb700 1 -- 172.21.15.98:0/1496651924 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f568c09cc40 con 0x7f5698102a90 2024-04-25T17:33:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.079+0000 7f569f0d7700 1 -- 172.21.15.98:0/1496651924 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f5698062e80 con 0x7f56880a3cd0 2024-04-25T17:33:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.107+0000 7f5695ffb700 1 -- 172.21.15.98:0/1496651924 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5698062e80 con 0x7f56880a3cd0 2024-04-25T17:33:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 -- 172.21.15.98:0/1496651924 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f56880a3cd0 msgr2=0x7f56880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f56880a3cd0 0x7f56880a6190 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f568400b5c0 tx=0x7f5684005fb0 comp rx=0 tx=0).stop 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 -- 172.21.15.98:0/1496651924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5698102a90 msgr2=0x7f5698199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5698102a90 0x7f5698199090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f568c00c970 tx=0x7f568c00cd30 comp rx=0 tx=0).stop 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 -- 172.21.15.98:0/1496651924 shutdown_connections 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f56880a3cd0 0x7f56880a6190 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5698108a50 0x7f569819d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5698102a90 0x7f5698199090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 --2- 172.21.15.98:0/1496651924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5698103450 0x7f56981995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 -- 172.21.15.98:0/1496651924 >> 172.21.15.98:0/1496651924 conn(0x7f5698079ba0 msgr2=0x7f5698101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 -- 172.21.15.98:0/1496651924 shutdown_connections 2024-04-25T17:33:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:12.110+0000 7f569f0d7700 1 -- 172.21.15.98:0/1496651924 wait complete. 2024-04-25T17:33:12.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:12 smithi098 ceph-mon[29192]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:12 smithi203 ceph-mon[32834]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:12.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:12 smithi173 ceph-mon[32856]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:12.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T17:33:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:13.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:13 smithi098 ceph-mon[29192]: pgmap v323: 97 pgs: 4 peering, 93 active+clean; 622 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-04-25T17:33:13.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:13 smithi203 ceph-mon[32834]: pgmap v323: 97 pgs: 4 peering, 93 active+clean; 622 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-04-25T17:33:13.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:13 smithi173 ceph-mon[32856]: pgmap v323: 97 pgs: 4 peering, 93 active+clean; 622 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2024-04-25T17:33:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.936+0000 7f26cc620700 1 -- 172.21.15.98:0/805191355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26c4103480 msgr2=0x7f26c410f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.936+0000 7f26cc620700 1 --2- 172.21.15.98:0/805191355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26c4103480 0x7f26c410f8d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00b0a0 tx=0x7f26bc00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:33:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.936+0000 7f26cc620700 1 -- 172.21.15.98:0/805191355 shutdown_connections 2024-04-25T17:33:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.936+0000 7f26cc620700 1 --2- 172.21.15.98:0/805191355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26c4103480 0x7f26c410f8d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.936+0000 7f26cc620700 1 --2- 172.21.15.98:0/805191355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26c4108a70 0x7f26c4108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.936+0000 7f26cc620700 1 --2- 172.21.15.98:0/805191355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26c4102ac0 0x7f26c4102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.936+0000 7f26cc620700 1 -- 172.21.15.98:0/805191355 >> 172.21.15.98:0/805191355 conn(0x7f26c4079bb0 msgr2=0x7f26c4079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.936+0000 7f26cc620700 1 -- 172.21.15.98:0/805191355 shutdown_connections 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.936+0000 7f26cc620700 1 -- 172.21.15.98:0/805191355 wait complete. 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26cc620700 1 Processor -- start 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26cc620700 1 -- start start 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26cc620700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26c4102ac0 0x7f26c41991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26cc620700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26c4103480 0x7f26c4199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26cc620700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26c4108a70 0x7f26c419dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26cc620700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f26c4112d00 con 0x7f26c4102ac0 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26cc620700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f26c4112b80 con 0x7f26c4103480 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26cc620700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f26c4112e80 con 0x7f26c4108a70 2024-04-25T17:33:14.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26ca3bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26c4102ac0 0x7f26c41991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26ca3bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26c4102ac0 0x7f26c41991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43566/0 (socket says 172.21.15.98:43566) 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26ca3bc700 1 -- 172.21.15.98:0/1909865756 learned_addr learned my addr 172.21.15.98:0/1909865756 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26c9bbb700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26c4103480 0x7f26c4199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26ca3bc700 1 -- 172.21.15.98:0/1909865756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26c4108a70 msgr2=0x7f26c419dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26ca3bc700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26c4108a70 0x7f26c419dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26ca3bc700 1 -- 172.21.15.98:0/1909865756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26c4103480 msgr2=0x7f26c4199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26ca3bc700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26c4103480 0x7f26c4199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26ca3bc700 1 -- 172.21.15.98:0/1909865756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c419e200 con 0x7f26c4102ac0 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.937+0000 7f26c9bbb700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26c4103480 0x7f26c4199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.938+0000 7f26ca3bc700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26c4102ac0 0x7f26c41991f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f26b400b530 tx=0x7f26b400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:14.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.938+0000 7f26bb7fe700 1 -- 172.21.15.98:0/1909865756 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4015020 con 0x7f26c4102ac0 2024-04-25T17:33:14.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.938+0000 7f26bb7fe700 1 -- 172.21.15.98:0/1909865756 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26b4011ce0 con 0x7f26c4102ac0 2024-04-25T17:33:14.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.938+0000 7f26cc620700 1 -- 172.21.15.98:0/1909865756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c419e4f0 con 0x7f26c4102ac0 2024-04-25T17:33:14.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.938+0000 7f26bb7fe700 1 -- 172.21.15.98:0/1909865756 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4010d20 con 0x7f26c4102ac0 2024-04-25T17:33:14.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.938+0000 7f26cc620700 1 -- 172.21.15.98:0/1909865756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c41a5d70 con 0x7f26c4102ac0 2024-04-25T17:33:14.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.939+0000 7f26cc620700 1 -- 172.21.15.98:0/1909865756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26c4069730 con 0x7f26c4102ac0 2024-04-25T17:33:14.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.940+0000 7f26bb7fe700 1 -- 172.21.15.98:0/1909865756 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f26b4018020 con 0x7f26c4102ac0 2024-04-25T17:33:14.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.940+0000 7f26bb7fe700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26b00a3cd0 0x7f26b00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:14.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.940+0000 7f26c9bbb700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26b00a3cd0 0x7f26b00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:14.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.940+0000 7f26bb7fe700 1 -- 172.21.15.98:0/1909865756 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f26b40cfcb0 con 0x7f26c4102ac0 2024-04-25T17:33:14.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.941+0000 7f26c9bbb700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26b00a3cd0 0x7f26b00a6190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f26c407a220 tx=0x7f26c0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:14.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:14.945+0000 7f26bb7fe700 1 -- 172.21.15.98:0/1909865756 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f26b401f020 con 0x7f26c4102ac0 2024-04-25T17:33:15.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.182+0000 7f26cc620700 1 -- 172.21.15.98:0/1909865756 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f26c4062e80 con 0x7f26b00a3cd0 2024-04-25T17:33:15.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.187+0000 7f26bb7fe700 1 -- 172.21.15.98:0/1909865756 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f26c4062e80 con 0x7f26b00a3cd0 2024-04-25T17:33:15.189 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_1/8b073cd4-d70f-47d4-966e-81c60974204b 2024-04-25T17:33:15.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 -- 172.21.15.98:0/1909865756 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26b00a3cd0 msgr2=0x7f26b00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:15.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26b00a3cd0 0x7f26b00a6190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f26c407a220 tx=0x7f26c0008040 comp rx=0 tx=0).stop 2024-04-25T17:33:15.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 -- 172.21.15.98:0/1909865756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26c4102ac0 msgr2=0x7f26c41991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:15.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26c4102ac0 0x7f26c41991f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f26b400b530 tx=0x7f26b400b8f0 comp rx=0 tx=0).stop 2024-04-25T17:33:15.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 -- 172.21.15.98:0/1909865756 shutdown_connections 2024-04-25T17:33:15.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f26b00a3cd0 0x7f26b00a6190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26c4108a70 0x7f26c419dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26c4102ac0 0x7f26c41991f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 --2- 172.21.15.98:0/1909865756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f26c4103480 0x7f26c4199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 -- 172.21.15.98:0/1909865756 >> 172.21.15.98:0/1909865756 conn(0x7f26c4079bb0 msgr2=0x7f26c41017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 -- 172.21.15.98:0/1909865756 shutdown_connections 2024-04-25T17:33:15.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:15.190+0000 7f26cc620700 1 -- 172.21.15.98:0/1909865756 wait complete. 2024-04-25T17:33:15.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:33:16.173 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:16.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:16 smithi098 ceph-mon[29192]: pgmap v324: 97 pgs: 97 active+clean; 622 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.2 KiB/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:16 smithi203 ceph-mon[32834]: pgmap v324: 97 pgs: 97 active+clean; 622 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.2 KiB/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:16.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:16 smithi173 ceph-mon[32856]: pgmap v324: 97 pgs: 97 active+clean; 622 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.2 KiB/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:17.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:17 smithi098 ceph-mon[29192]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:17.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:17 smithi203 ceph-mon[32834]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:17.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:17 smithi173 ceph-mon[32856]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:33:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.911+0000 7f2ac8492700 1 -- 172.21.15.98:0/545821462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac0108a80 msgr2=0x7f2ac0108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.911+0000 7f2ac8492700 1 --2- 172.21.15.98:0/545821462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac0108a80 0x7f2ac0108e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009990 tx=0x7f2ab0009ca0 comp rx=0 tx=0).stop 2024-04-25T17:33:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.911+0000 7f2ac8492700 1 -- 172.21.15.98:0/545821462 shutdown_connections 2024-04-25T17:33:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.911+0000 7f2ac8492700 1 --2- 172.21.15.98:0/545821462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ac01034b0 0x7f2ac010f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.911+0000 7f2ac8492700 1 --2- 172.21.15.98:0/545821462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac0108a80 0x7f2ac0108e60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.911+0000 7f2ac8492700 1 --2- 172.21.15.98:0/545821462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ac0102af0 0x7f2ac0102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.911+0000 7f2ac8492700 1 -- 172.21.15.98:0/545821462 >> 172.21.15.98:0/545821462 conn(0x7f2ac0079ad0 msgr2=0x7f2ac0079ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.911+0000 7f2ac8492700 1 -- 172.21.15.98:0/545821462 shutdown_connections 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.911+0000 7f2ac8492700 1 -- 172.21.15.98:0/545821462 wait complete. 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac8492700 1 Processor -- start 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac8492700 1 -- start start 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac8492700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ac0102af0 0x7f2ac0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac8492700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ac01034b0 0x7f2ac0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac8492700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac0108a80 0x7f2ac00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac8492700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac0112c90 con 0x7f2ac0108a80 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac8492700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac0112b10 con 0x7f2ac0102af0 2024-04-25T17:33:17.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac8492700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac0112e10 con 0x7f2ac01034b0 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac6a2f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac0108a80 0x7f2ac00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac622e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ac0102af0 0x7f2ac0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac6a2f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac0108a80 0x7f2ac00757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43586/0 (socket says 172.21.15.98:43586) 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac6a2f700 1 -- 172.21.15.98:0/309803394 learned_addr learned my addr 172.21.15.98:0/309803394 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac5a2d700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ac01034b0 0x7f2ac0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac6a2f700 1 -- 172.21.15.98:0/309803394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ac01034b0 msgr2=0x7f2ac0075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac6a2f700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ac01034b0 0x7f2ac0075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac6a2f700 1 -- 172.21.15.98:0/309803394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ac0102af0 msgr2=0x7f2ac0074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac6a2f700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ac0102af0 0x7f2ac0074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.912+0000 7f2ac6a2f700 1 -- 172.21.15.98:0/309803394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac01a7630 con 0x7f2ac0108a80 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.913+0000 7f2ac6a2f700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac0108a80 0x7f2ac00757c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8007d70 tx=0x7f2ab800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.913+0000 7f2ab77fe700 1 -- 172.21.15.98:0/309803394 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8016020 con 0x7f2ac0108a80 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.913+0000 7f2ab77fe700 1 -- 172.21.15.98:0/309803394 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ab800ad40 con 0x7f2ac0108a80 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.913+0000 7f2ac8492700 1 -- 172.21.15.98:0/309803394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac01a7920 con 0x7f2ac0108a80 2024-04-25T17:33:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.913+0000 7f2ab77fe700 1 -- 172.21.15.98:0/309803394 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab80115e0 con 0x7f2ac0108a80 2024-04-25T17:33:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.913+0000 7f2ac8492700 1 -- 172.21.15.98:0/309803394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac01a7e60 con 0x7f2ac0108a80 2024-04-25T17:33:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.914+0000 7f2ac8492700 1 -- 172.21.15.98:0/309803394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ac0069730 con 0x7f2ac0108a80 2024-04-25T17:33:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.915+0000 7f2ab77fe700 1 -- 172.21.15.98:0/309803394 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2ab8019020 con 0x7f2ac0108a80 2024-04-25T17:33:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.916+0000 7f2ab77fe700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2aac0a3d50 0x7f2aac0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.916+0000 7f2ac622e700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2aac0a3d50 0x7f2aac0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.916+0000 7f2ab77fe700 1 -- 172.21.15.98:0/309803394 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f2ab80d0920 con 0x7f2ac0108a80 2024-04-25T17:33:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.916+0000 7f2ac622e700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2aac0a3d50 0x7f2aac0a6210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000b5c0 tx=0x7f2ab0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:17.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:17.919+0000 7f2ab77fe700 1 -- 172.21.15.98:0/309803394 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ab8020020 con 0x7f2ac0108a80 2024-04-25T17:33:18.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.177+0000 7f2ac8492700 1 -- 172.21.15.98:0/309803394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ac002df80 con 0x7f2ac0108a80 2024-04-25T17:33:18.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.178+0000 7f2ab77fe700 1 -- 172.21.15.98:0/309803394 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11877 (secure 0 0 0) 0x7f2ab809d2c0 con 0x7f2ac0108a80 2024-04-25T17:33:18.180 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:33:18.180 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:32:59.314849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:33:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 -- 172.21.15.98:0/309803394 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2aac0a3d50 msgr2=0x7f2aac0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2aac0a3d50 0x7f2aac0a6210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000b5c0 tx=0x7f2ab0005960 comp rx=0 tx=0).stop 2024-04-25T17:33:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 -- 172.21.15.98:0/309803394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac0108a80 msgr2=0x7f2ac00757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac0108a80 0x7f2ac00757c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8007d70 tx=0x7f2ab800a340 comp rx=0 tx=0).stop 2024-04-25T17:33:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 -- 172.21.15.98:0/309803394 shutdown_connections 2024-04-25T17:33:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f2aac0a3d50 0x7f2aac0a6210 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ac0108a80 0x7f2ac00757c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ac0102af0 0x7f2ac0074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 --2- 172.21.15.98:0/309803394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ac01034b0 0x7f2ac0075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 -- 172.21.15.98:0/309803394 >> 172.21.15.98:0/309803394 conn(0x7f2ac0079ad0 msgr2=0x7f2ac01016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:18.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 -- 172.21.15.98:0/309803394 shutdown_connections 2024-04-25T17:33:18.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:18.180+0000 7f2ac8492700 1 -- 172.21.15.98:0/309803394 wait complete. 2024-04-25T17:33:18.183 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 25 2024-04-25T17:33:18.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:18 smithi098 ceph-mon[29192]: pgmap v325: 97 pgs: 97 active+clean; 622 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 3.4 KiB/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:18.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:18 smithi203 ceph-mon[32834]: pgmap v325: 97 pgs: 97 active+clean; 622 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 3.4 KiB/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:18 smithi173 ceph-mon[32856]: pgmap v325: 97 pgs: 97 active+clean; 622 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 3.4 KiB/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:18.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T17:33:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:19.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:19 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/309803394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:33:19.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:19 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/309803394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:33:19.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:19 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/309803394' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:33:20.316 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:20 smithi098 ceph-mon[29192]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 4.4 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:20.365 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:20 smithi203 ceph-mon[32834]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 4.4 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:20.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:20 smithi173 ceph-mon[32856]: pgmap v326: 97 pgs: 97 active+clean; 636 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 4.4 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.861+0000 7f8190f51700 1 -- 172.21.15.98:0/574998961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f818c107b00 msgr2=0x7f818c107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.861+0000 7f8190f51700 1 --2- 172.21.15.98:0/574998961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f818c107b00 0x7f818c107ee0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8180009990 tx=0x7f8180009ca0 comp rx=0 tx=0).stop 2024-04-25T17:33:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.861+0000 7f8190f51700 1 -- 172.21.15.98:0/574998961 shutdown_connections 2024-04-25T17:33:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.861+0000 7f8190f51700 1 --2- 172.21.15.98:0/574998961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c101da0 0x7f818c10f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.861+0000 7f8190f51700 1 --2- 172.21.15.98:0/574998961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f818c107b00 0x7f818c107ee0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.861+0000 7f8190f51700 1 --2- 172.21.15.98:0/574998961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f818c1013e0 0x7f818c101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.861+0000 7f8190f51700 1 -- 172.21.15.98:0/574998961 >> 172.21.15.98:0/574998961 conn(0x7f818c076c60 msgr2=0x7f818c077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:20.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.861+0000 7f8190f51700 1 -- 172.21.15.98:0/574998961 shutdown_connections 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.861+0000 7f8190f51700 1 -- 172.21.15.98:0/574998961 wait complete. 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.862+0000 7f8190f51700 1 Processor -- start 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.862+0000 7f8190f51700 1 -- start start 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.862+0000 7f8190f51700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f818c1013e0 0x7f818c199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.862+0000 7f8190f51700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c101da0 0x7f818c1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.862+0000 7f8190f51700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f818c107b00 0x7f818c19da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.862+0000 7f8190f51700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f818c112c50 con 0x7f818c1013e0 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.862+0000 7f8190f51700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f818c112ad0 con 0x7f818c107b00 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.862+0000 7f8190f51700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f818c112dd0 con 0x7f818c101da0 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f818a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f818c1013e0 0x7f818c199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f818a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f818c1013e0 0x7f818c199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43602/0 (socket says 172.21.15.98:43602) 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f818a59c700 1 -- 172.21.15.98:0/784572759 learned_addr learned my addr 172.21.15.98:0/784572759 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:20.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f8189d9b700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c101da0 0x7f818c1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f818ad9d700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f818c107b00 0x7f818c19da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f818a59c700 1 -- 172.21.15.98:0/784572759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c101da0 msgr2=0x7f818c1996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f818a59c700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c101da0 0x7f818c1996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f818a59c700 1 -- 172.21.15.98:0/784572759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f818c107b00 msgr2=0x7f818c19da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f818a59c700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f818c107b00 0x7f818c19da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f818a59c700 1 -- 172.21.15.98:0/784572759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818c19e140 con 0x7f818c1013e0 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f818a59c700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f818c1013e0 0x7f818c199180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8180000c00 tx=0x7f8180005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f817b7fe700 1 -- 172.21.15.98:0/784572759 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8180027070 con 0x7f818c1013e0 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f8190f51700 1 -- 172.21.15.98:0/784572759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818c19e3d0 con 0x7f818c1013e0 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f817b7fe700 1 -- 172.21.15.98:0/784572759 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8180003fe0 con 0x7f818c1013e0 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f817b7fe700 1 -- 172.21.15.98:0/784572759 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8180022bc0 con 0x7f818c1013e0 2024-04-25T17:33:20.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.863+0000 7f8190f51700 1 -- 172.21.15.98:0/784572759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f818c1a5cb0 con 0x7f818c1013e0 2024-04-25T17:33:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.864+0000 7f8190f51700 1 -- 172.21.15.98:0/784572759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f818c069730 con 0x7f818c1013e0 2024-04-25T17:33:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.865+0000 7f817b7fe700 1 -- 172.21.15.98:0/784572759 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f818002a070 con 0x7f818c1013e0 2024-04-25T17:33:20.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.866+0000 7f817b7fe700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f81740a3cd0 0x7f81740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.866+0000 7f8189d9b700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f81740a3cd0 0x7f81740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.866+0000 7f817b7fe700 1 -- 172.21.15.98:0/784572759 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f81800d1410 con 0x7f818c1013e0 2024-04-25T17:33:20.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.866+0000 7f8189d9b700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f81740a3cd0 0x7f81740a6190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f818c0772b0 tx=0x7f817c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:20.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:20.870+0000 7f817b7fe700 1 -- 172.21.15.98:0/784572759 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f818005eff0 con 0x7f818c1013e0 2024-04-25T17:33:21.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.073+0000 7f8190f51700 1 -- 172.21.15.98:0/784572759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f818c02df80 con 0x7f818c1013e0 2024-04-25T17:33:21.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.074+0000 7f817b7fe700 1 -- 172.21.15.98:0/784572759 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22686 (secure 0 0 0) 0x7f8180013070 con 0x7f818c1013e0 2024-04-25T17:33:21.076 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:33:21.076 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:33:10.182926+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T17:32:30.267079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T17:32:33.956902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:33:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.076+0000 7f8190f51700 1 -- 172.21.15.98:0/784572759 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f81740a3cd0 msgr2=0x7f81740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.076+0000 7f8190f51700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f81740a3cd0 0x7f81740a6190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f818c0772b0 tx=0x7f817c00b410 comp rx=0 tx=0).stop 2024-04-25T17:33:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.076+0000 7f8190f51700 1 -- 172.21.15.98:0/784572759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f818c1013e0 msgr2=0x7f818c199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.076+0000 7f8190f51700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f818c1013e0 0x7f818c199180 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8180000c00 tx=0x7f8180005ef0 comp rx=0 tx=0).stop 2024-04-25T17:33:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.076+0000 7f8190f51700 1 -- 172.21.15.98:0/784572759 shutdown_connections 2024-04-25T17:33:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.076+0000 7f8190f51700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f81740a3cd0 0x7f81740a6190 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.076+0000 7f8190f51700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f818c107b00 0x7f818c19da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.076+0000 7f8190f51700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f818c1013e0 0x7f818c199180 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.076+0000 7f8190f51700 1 --2- 172.21.15.98:0/784572759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c101da0 0x7f818c1996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.076+0000 7f8190f51700 1 -- 172.21.15.98:0/784572759 >> 172.21.15.98:0/784572759 conn(0x7f818c076c60 msgr2=0x7f818c0ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.077+0000 7f8190f51700 1 -- 172.21.15.98:0/784572759 shutdown_connections 2024-04-25T17:33:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:21.077+0000 7f8190f51700 1 -- 172.21.15.98:0/784572759 wait complete. 2024-04-25T17:33:21.929 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T17:33:21.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T17:33:22.176 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:22.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:22 smithi098 ceph-mon[29192]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.0 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:22.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:22 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/784572759' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:33:22.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:22 smithi203 ceph-mon[32834]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.0 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:22.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:22 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/784572759' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:33:22.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:22 smithi173 ceph-mon[32856]: pgmap v327: 97 pgs: 97 active+clean; 636 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.0 KiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:22.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:22 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/784572759' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:33:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 -- 172.21.15.98:0/2193715306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbb8107b00 msgr2=0x7ffbb8107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 --2- 172.21.15.98:0/2193715306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbb8107b00 0x7ffbb8107ee0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009990 tx=0x7ffbb4009ca0 comp rx=0 tx=0).stop 2024-04-25T17:33:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 -- 172.21.15.98:0/2193715306 shutdown_connections 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 --2- 172.21.15.98:0/2193715306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbb8101da0 0x7ffbb810f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 --2- 172.21.15.98:0/2193715306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbb8107b00 0x7ffbb8107ee0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 --2- 172.21.15.98:0/2193715306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbb81013e0 0x7ffbb8101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 -- 172.21.15.98:0/2193715306 >> 172.21.15.98:0/2193715306 conn(0x7ffbb8076c60 msgr2=0x7ffbb8077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 -- 172.21.15.98:0/2193715306 shutdown_connections 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 -- 172.21.15.98:0/2193715306 wait complete. 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 Processor -- start 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.879+0000 7ffbbfec7700 1 -- start start 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbfec7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbb81013e0 0x7ffbb81025c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbfec7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbb8101da0 0x7ffbb8102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbfec7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbb8107b00 0x7ffbb8103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbfec7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8112c50 con 0x7ffbb81013e0 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbfec7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8112ad0 con 0x7ffbb8107b00 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbfec7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8112dd0 con 0x7ffbb8101da0 2024-04-25T17:33:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbdc63700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbb81013e0 0x7ffbb81025c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbdc63700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbb81013e0 0x7ffbb81025c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40328/0 (socket says 172.21.15.98:40328) 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbdc63700 1 -- 172.21.15.98:0/33997680 learned_addr learned my addr 172.21.15.98:0/33997680 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbd462700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbb8101da0 0x7ffbb8102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbe464700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbb8107b00 0x7ffbb8103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbdc63700 1 -- 172.21.15.98:0/33997680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbb8101da0 msgr2=0x7ffbb8102b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbdc63700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbb8101da0 0x7ffbb8102b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbdc63700 1 -- 172.21.15.98:0/33997680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbb8107b00 msgr2=0x7ffbb8103040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbdc63700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbb8107b00 0x7ffbb8103040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbdc63700 1 -- 172.21.15.98:0/33997680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb81a75f0 con 0x7ffbb81013e0 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbdc63700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbb81013e0 0x7ffbb81025c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4000c00 tx=0x7ffbb40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbaeffd700 1 -- 172.21.15.98:0/33997680 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4027070 con 0x7ffbb81013e0 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbfec7700 1 -- 172.21.15.98:0/33997680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb81a7880 con 0x7ffbb81013e0 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbaeffd700 1 -- 172.21.15.98:0/33997680 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbb4003f20 con 0x7ffbb81013e0 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbaeffd700 1 -- 172.21.15.98:0/33997680 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4022c70 con 0x7ffbb81013e0 2024-04-25T17:33:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.880+0000 7ffbbfec7700 1 -- 172.21.15.98:0/33997680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb81a7cd0 con 0x7ffbb81013e0 2024-04-25T17:33:23.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.881+0000 7ffbbfec7700 1 -- 172.21.15.98:0/33997680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbb8069730 con 0x7ffbb81013e0 2024-04-25T17:33:23.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.883+0000 7ffbaeffd700 1 -- 172.21.15.98:0/33997680 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffbb402a070 con 0x7ffbb81013e0 2024-04-25T17:33:23.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.884+0000 7ffbaeffd700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ffba40a3cd0 0x7ffba40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:23.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.884+0000 7ffbbd462700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ffba40a3cd0 0x7ffba40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:23.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.884+0000 7ffbaeffd700 1 -- 172.21.15.98:0/33997680 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7ffbb40d1790 con 0x7ffbb81013e0 2024-04-25T17:33:23.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.884+0000 7ffbbd462700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ffba40a3cd0 0x7ffba40a6190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8077210 tx=0x7ffba800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:23.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:23.887+0000 7ffbaeffd700 1 -- 172.21.15.98:0/33997680 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffbb4031070 con 0x7ffbb81013e0 2024-04-25T17:33:24.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.085+0000 7ffbbfec7700 1 -- 172.21.15.98:0/33997680 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffbb8062e80 con 0x7ffba40a3cd0 2024-04-25T17:33:24.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.092+0000 7ffbaeffd700 1 -- 172.21.15.98:0/33997680 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258889 (secure 0 0 0) 0x7ffbb8062e80 con 0x7ffba40a3cd0 2024-04-25T17:33:24.097 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:33:24.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.095+0000 7ffbbfec7700 1 -- 172.21.15.98:0/33997680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ffba40a3cd0 msgr2=0x7ffba40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.095+0000 7ffbbfec7700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ffba40a3cd0 0x7ffba40a6190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8077210 tx=0x7ffba800b410 comp rx=0 tx=0).stop 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.095+0000 7ffbbfec7700 1 -- 172.21.15.98:0/33997680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbb81013e0 msgr2=0x7ffbb81025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.095+0000 7ffbbfec7700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbb81013e0 0x7ffbb81025c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4000c00 tx=0x7ffbb40047b0 comp rx=0 tx=0).stop 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.095+0000 7ffbbfec7700 1 -- 172.21.15.98:0/33997680 shutdown_connections 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.095+0000 7ffbbfec7700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ffba40a3cd0 0x7ffba40a6190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.095+0000 7ffbbfec7700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbb8107b00 0x7ffbb8103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.095+0000 7ffbbfec7700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbb81013e0 0x7ffbb81025c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.095+0000 7ffbbfec7700 1 --2- 172.21.15.98:0/33997680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbb8101da0 0x7ffbb8102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.096+0000 7ffbbfec7700 1 -- 172.21.15.98:0/33997680 >> 172.21.15.98:0/33997680 conn(0x7ffbb8076c60 msgr2=0x7ffbb80fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.096+0000 7ffbbfec7700 1 -- 172.21.15.98:0/33997680 shutdown_connections 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:24.096+0000 7ffbbfec7700 1 -- 172.21.15.98:0/33997680 wait complete. 2024-04-25T17:33:24.100 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T17:33:24.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:24 smithi098 ceph-mon[29192]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.8 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:24 smithi203 ceph-mon[32834]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.8 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:24 smithi173 ceph-mon[32856]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.8 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:24.826 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-04-25T17:33:22.977009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652681,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":318,"num_read_kb":273,"num_write":346,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357124,"kb_used_data":12096,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584700,"statfs":{"total":1151940427776,"available":1151574732800,"internally_reserved":0,"allocated":12386304,"data_stored":7746147,"data_compressed":106941,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35865,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":-3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":28,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002978"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190145+0000","last_change":"2024-04-25T17:32:35.075338+0000","last_active":"2024-04-25T17:33:10.190145+0000","last_peered":"2024-04-25T17:33:10.190145+0000","last_clean":"2024-04-25T17:33:10.190145+0000","last_became_active":"2024-04-25T17:32:35.075029+0000","last_became_peered":"2024-04-25T17:32:35.075029+0000","last_unstale":"2024-04-25T17:33:10.190145+0000","last_undegraded":"2024-04-25T17:33:10.190145+0000","last_fullsized":"2024-04-25T17:33:10.190145+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:23.837337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647523+0000","last_change":"2024-04-25T17:32:35.075422+0000","last_active":"2024-04-25T17:33:10.647523+0000","last_peered":"2024-04-25T17:33:10.647523+0000","last_clean":"2024-04-25T17:33:10.647523+0000","last_became_active":"2024-04-25T17:32:35.071975+0000","last_became_peered":"2024-04-25T17:32:35.071975+0000","last_unstale":"2024-04-25T17:33:10.647523+0000","last_undegraded":"2024-04-25T17:33:10.647523+0000","last_fullsized":"2024-04-25T17:33:10.647523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:20.548869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647315+0000","last_change":"2024-04-25T17:32:35.068153+0000","last_active":"2024-04-25T17:33:10.647315+0000","last_peered":"2024-04-25T17:33:10.647315+0000","last_clean":"2024-04-25T17:33:10.647315+0000","last_became_active":"2024-04-25T17:32:35.065873+0000","last_became_peered":"2024-04-25T17:32:35.065873+0000","last_unstale":"2024-04-25T17:33:10.647315+0000","last_undegraded":"2024-04-25T17:33:10.647315+0000","last_fullsized":"2024-04-25T17:33:10.647315+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:31.381875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605643+0000","last_change":"2024-04-25T17:32:35.073055+0000","last_active":"2024-04-25T17:33:10.605643+0000","last_peered":"2024-04-25T17:33:10.605643+0000","last_clean":"2024-04-25T17:33:10.605643+0000","last_became_active":"2024-04-25T17:32:35.072596+0000","last_became_peered":"2024-04-25T17:32:35.072596+0000","last_unstale":"2024-04-25T17:33:10.605643+0000","last_undegraded":"2024-04-25T17:33:10.605643+0000","last_fullsized":"2024-04-25T17:33:10.605643+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:37.529698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195326+0000","last_change":"2024-04-25T17:32:35.072721+0000","last_active":"2024-04-25T17:33:10.195326+0000","last_peered":"2024-04-25T17:33:10.195326+0000","last_clean":"2024-04-25T17:33:10.195326+0000","last_became_active":"2024-04-25T17:32:35.072370+0000","last_became_peered":"2024-04-25T17:32:35.072370+0000","last_unstale":"2024-04-25T17:33:10.195326+0000","last_undegraded":"2024-04-25T17:33:10.195326+0000","last_fullsized":"2024-04-25T17:33:10.195326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:34.429985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850042+0000","last_change":"2024-04-25T17:32:35.071538+0000","last_active":"2024-04-25T17:33:10.850042+0000","last_peered":"2024-04-25T17:33:10.850042+0000","last_clean":"2024-04-25T17:33:10.850042+0000","last_became_active":"2024-04-25T17:32:35.067427+0000","last_became_peered":"2024-04-25T17:32:35.067427+0000","last_unstale":"2024-04-25T17:33:10.850042+0000","last_undegraded":"2024-04-25T17:33:10.850042+0000","last_fullsized":"2024-04-25T17:33:10.850042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:36.052762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189890+0000","last_change":"2024-04-25T17:32:35.079476+0000","last_active":"2024-04-25T17:33:10.189890+0000","last_peered":"2024-04-25T17:33:10.189890+0000","last_clean":"2024-04-25T17:33:10.189890+0000","last_became_active":"2024-04-25T17:32:35.079170+0000","last_became_peered":"2024-04-25T17:32:35.079170+0000","last_unstale":"2024-04-25T17:33:10.189890+0000","last_undegraded":"2024-04-25T17:33:10.189890+0000","last_fullsized":"2024-04-25T17:33:10.189890+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:30.937670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189780+0000","last_change":"2024-04-25T17:32:35.077261+0000","last_active":"2024-04-25T17:33:10.189780+0000","last_peered":"2024-04-25T17:33:10.189780+0000","last_clean":"2024-04-25T17:33:10.189780+0000","last_became_active":"2024-04-25T17:32:35.076896+0000","last_became_peered":"2024-04-25T17:32:35.076896+0000","last_unstale":"2024-04-25T17:33:10.189780+0000","last_undegraded":"2024-04-25T17:33:10.189780+0000","last_fullsized":"2024-04-25T17:33:10.189780+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:24.082550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187931+0000","last_change":"2024-04-25T17:32:35.072229+0000","last_active":"2024-04-25T17:33:10.187931+0000","last_peered":"2024-04-25T17:33:10.187931+0000","last_clean":"2024-04-25T17:33:10.187931+0000","last_became_active":"2024-04-25T17:32:35.071586+0000","last_became_peered":"2024-04-25T17:32:35.071586+0000","last_unstale":"2024-04-25T17:33:10.187931+0000","last_undegraded":"2024-04-25T17:33:10.187931+0000","last_fullsized":"2024-04-25T17:33:10.187931+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:01.195638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850464+0000","last_change":"2024-04-25T17:32:35.075274+0000","last_active":"2024-04-25T17:33:10.850464+0000","last_peered":"2024-04-25T17:33:10.850464+0000","last_clean":"2024-04-25T17:33:10.850464+0000","last_became_active":"2024-04-25T17:32:35.073821+0000","last_became_peered":"2024-04-25T17:32:35.073821+0000","last_unstale":"2024-04-25T17:33:10.850464+0000","last_undegraded":"2024-04-25T17:33:10.850464+0000","last_fullsized":"2024-04-25T17:33:10.850464+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:43:04.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190303+0000","last_change":"2024-04-25T17:32:35.080747+0000","last_active":"2024-04-25T17:33:10.190303+0000","last_peered":"2024-04-25T17:33:10.190303+0000","last_clean":"2024-04-25T17:33:10.190303+0000","last_became_active":"2024-04-25T17:32:35.080383+0000","last_became_peered":"2024-04-25T17:32:35.080383+0000","last_unstale":"2024-04-25T17:33:10.190303+0000","last_undegraded":"2024-04-25T17:33:10.190303+0000","last_fullsized":"2024-04-25T17:33:10.190303+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:42.128892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605662+0000","last_change":"2024-04-25T17:32:35.061442+0000","last_active":"2024-04-25T17:33:10.605662+0000","last_peered":"2024-04-25T17:33:10.605662+0000","last_clean":"2024-04-25T17:33:10.605662+0000","last_became_active":"2024-04-25T17:32:35.061030+0000","last_became_peered":"2024-04-25T17:32:35.061030+0000","last_unstale":"2024-04-25T17:33:10.605662+0000","last_undegraded":"2024-04-25T17:33:10.605662+0000","last_fullsized":"2024-04-25T17:33:10.605662+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:03.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187927+0000","last_change":"2024-04-25T17:32:35.069698+0000","last_active":"2024-04-25T17:33:10.187927+0000","last_peered":"2024-04-25T17:33:10.187927+0000","last_clean":"2024-04-25T17:33:10.187927+0000","last_became_active":"2024-04-25T17:32:35.069374+0000","last_became_peered":"2024-04-25T17:32:35.069374+0000","last_unstale":"2024-04-25T17:33:10.187927+0000","last_undegraded":"2024-04-25T17:33:10.187927+0000","last_fullsized":"2024-04-25T17:33:10.187927+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:03.214525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647126+0000","last_change":"2024-04-25T17:32:35.069001+0000","last_active":"2024-04-25T17:33:10.647126+0000","last_peered":"2024-04-25T17:33:10.647126+0000","last_clean":"2024-04-25T17:33:10.647126+0000","last_became_active":"2024-04-25T17:32:35.064882+0000","last_became_peered":"2024-04-25T17:32:35.064882+0000","last_unstale":"2024-04-25T17:33:10.647126+0000","last_undegraded":"2024-04-25T17:33:10.647126+0000","last_fullsized":"2024-04-25T17:33:10.647126+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:32.122251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647097+0000","last_change":"2024-04-25T17:32:35.065042+0000","last_active":"2024-04-25T17:33:10.647097+0000","last_peered":"2024-04-25T17:33:10.647097+0000","last_clean":"2024-04-25T17:33:10.647097+0000","last_became_active":"2024-04-25T17:32:35.064719+0000","last_became_peered":"2024-04-25T17:32:35.064719+0000","last_unstale":"2024-04-25T17:33:10.647097+0000","last_undegraded":"2024-04-25T17:33:10.647097+0000","last_fullsized":"2024-04-25T17:33:10.647097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:54.715385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187383+0000","last_change":"2024-04-25T17:32:35.067390+0000","last_active":"2024-04-25T17:33:10.187383+0000","last_peered":"2024-04-25T17:33:10.187383+0000","last_clean":"2024-04-25T17:33:10.187383+0000","last_became_active":"2024-04-25T17:32:35.066846+0000","last_became_peered":"2024-04-25T17:32:35.066846+0000","last_unstale":"2024-04-25T17:33:10.187383+0000","last_undegraded":"2024-04-25T17:33:10.187383+0000","last_fullsized":"2024-04-25T17:33:10.187383+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:24.305767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190375+0000","last_change":"2024-04-25T17:32:35.075349+0000","last_active":"2024-04-25T17:33:10.190375+0000","last_peered":"2024-04-25T17:33:10.190375+0000","last_clean":"2024-04-25T17:33:10.190375+0000","last_became_active":"2024-04-25T17:32:35.075046+0000","last_became_peered":"2024-04-25T17:32:35.075046+0000","last_unstale":"2024-04-25T17:33:10.190375+0000","last_undegraded":"2024-04-25T17:33:10.190375+0000","last_fullsized":"2024-04-25T17:33:10.190375+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:51.703406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185881+0000","last_change":"2024-04-25T17:32:35.077597+0000","last_active":"2024-04-25T17:33:10.185881+0000","last_peered":"2024-04-25T17:33:10.185881+0000","last_clean":"2024-04-25T17:33:10.185881+0000","last_became_active":"2024-04-25T17:32:35.077232+0000","last_became_peered":"2024-04-25T17:32:35.077232+0000","last_unstale":"2024-04-25T17:33:10.185881+0000","last_undegraded":"2024-04-25T17:33:10.185881+0000","last_fullsized":"2024-04-25T17:33:10.185881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:46.842673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.188192+0000","last_change":"2024-04-25T17:32:35.078229+0000","last_active":"2024-04-25T17:33:10.188192+0000","last_peered":"2024-04-25T17:33:10.188192+0000","last_clean":"2024-04-25T17:33:10.188192+0000","last_became_active":"2024-04-25T17:32:35.077899+0000","last_became_peered":"2024-04-25T17:32:35.077899+0000","last_unstale":"2024-04-25T17:33:10.188192+0000","last_undegraded":"2024-04-25T17:33:10.188192+0000","last_fullsized":"2024-04-25T17:33:10.188192+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:22.776371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185950+0000","last_change":"2024-04-25T17:32:32.045385+0000","last_active":"2024-04-25T17:33:10.185950+0000","last_peered":"2024-04-25T17:33:10.185950+0000","last_clean":"2024-04-25T17:33:10.185950+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:33:10.185950+0000","last_undegraded":"2024-04-25T17:33:10.185950+0000","last_fullsized":"2024-04-25T17:33:10.185950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:29.732001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190537+0000","last_change":"2024-04-25T17:32:35.079966+0000","last_active":"2024-04-25T17:33:10.190537+0000","last_peered":"2024-04-25T17:33:10.190537+0000","last_clean":"2024-04-25T17:33:10.190537+0000","last_became_active":"2024-04-25T17:32:35.079649+0000","last_became_peered":"2024-04-25T17:32:35.079649+0000","last_unstale":"2024-04-25T17:33:10.190537+0000","last_undegraded":"2024-04-25T17:33:10.190537+0000","last_fullsized":"2024-04-25T17:33:10.190537+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:19:45.679273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190704+0000","last_change":"2024-04-25T17:32:32.285667+0000","last_active":"2024-04-25T17:33:10.190704+0000","last_peered":"2024-04-25T17:33:10.190704+0000","last_clean":"2024-04-25T17:33:10.190704+0000","last_became_active":"2024-04-25T17:32:32.285330+0000","last_became_peered":"2024-04-25T17:32:32.285330+0000","last_unstale":"2024-04-25T17:33:10.190704+0000","last_undegraded":"2024-04-25T17:33:10.190704+0000","last_fullsized":"2024-04-25T17:33:10.190704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.975280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.686361+0000","last_change":"2024-04-25T17:32:35.068841+0000","last_active":"2024-04-25T17:33:10.686361+0000","last_peered":"2024-04-25T17:33:10.686361+0000","last_clean":"2024-04-25T17:33:10.686361+0000","last_became_active":"2024-04-25T17:32:35.068518+0000","last_became_peered":"2024-04-25T17:32:35.068518+0000","last_unstale":"2024-04-25T17:33:10.686361+0000","last_undegraded":"2024-04-25T17:33:10.686361+0000","last_fullsized":"2024-04-25T17:33:10.686361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:23.290990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195797+0000","last_change":"2024-04-25T17:32:32.045627+0000","last_active":"2024-04-25T17:33:10.195797+0000","last_peered":"2024-04-25T17:33:10.195797+0000","last_clean":"2024-04-25T17:33:10.195797+0000","last_became_active":"2024-04-25T17:32:32.044971+0000","last_became_peered":"2024-04-25T17:32:32.044971+0000","last_unstale":"2024-04-25T17:33:10.195797+0000","last_undegraded":"2024-04-25T17:33:10.195797+0000","last_fullsized":"2024-04-25T17:33:10.195797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:50.590435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187551+0000","last_change":"2024-04-25T17:32:35.070862+0000","last_active":"2024-04-25T17:33:10.187551+0000","last_peered":"2024-04-25T17:33:10.187551+0000","last_clean":"2024-04-25T17:33:10.187551+0000","last_became_active":"2024-04-25T17:32:35.069927+0000","last_became_peered":"2024-04-25T17:32:35.069927+0000","last_unstale":"2024-04-25T17:33:10.187551+0000","last_undegraded":"2024-04-25T17:33:10.187551+0000","last_fullsized":"2024-04-25T17:33:10.187551+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:40:01.799015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195097+0000","last_change":"2024-04-25T17:32:32.046062+0000","last_active":"2024-04-25T17:33:10.195097+0000","last_peered":"2024-04-25T17:33:10.195097+0000","last_clean":"2024-04-25T17:33:10.195097+0000","last_became_active":"2024-04-25T17:32:32.044903+0000","last_became_peered":"2024-04-25T17:32:32.044903+0000","last_unstale":"2024-04-25T17:33:10.195097+0000","last_undegraded":"2024-04-25T17:33:10.195097+0000","last_fullsized":"2024-04-25T17:33:10.195097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:51.099460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193396+0000","last_change":"2024-04-25T17:32:35.075452+0000","last_active":"2024-04-25T17:33:10.193396+0000","last_peered":"2024-04-25T17:33:10.193396+0000","last_clean":"2024-04-25T17:33:10.193396+0000","last_became_active":"2024-04-25T17:32:35.075136+0000","last_became_peered":"2024-04-25T17:32:35.075136+0000","last_unstale":"2024-04-25T17:33:10.193396+0000","last_undegraded":"2024-04-25T17:33:10.193396+0000","last_fullsized":"2024-04-25T17:33:10.193396+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:02.290926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:20.185050+0000","last_change":"2024-04-25T17:32:32.042019+0000","last_active":"2024-04-25T17:33:20.185050+0000","last_peered":"2024-04-25T17:33:20.185050+0000","last_clean":"2024-04-25T17:33:20.185050+0000","last_became_active":"2024-04-25T17:32:32.041641+0000","last_became_peered":"2024-04-25T17:32:32.041641+0000","last_unstale":"2024-04-25T17:33:20.185050+0000","last_undegraded":"2024-04-25T17:33:20.185050+0000","last_fullsized":"2024-04-25T17:33:20.185050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:02.025915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.194503+0000","last_change":"2024-04-25T17:32:35.069715+0000","last_active":"2024-04-25T17:33:10.194503+0000","last_peered":"2024-04-25T17:33:10.194503+0000","last_clean":"2024-04-25T17:33:10.194503+0000","last_became_active":"2024-04-25T17:32:35.069339+0000","last_became_peered":"2024-04-25T17:32:35.069339+0000","last_unstale":"2024-04-25T17:33:10.194503+0000","last_undegraded":"2024-04-25T17:33:10.194503+0000","last_fullsized":"2024-04-25T17:33:10.194503+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:48:13.572261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.849969+0000","last_change":"2024-04-25T17:32:32.042445+0000","last_active":"2024-04-25T17:33:10.849969+0000","last_peered":"2024-04-25T17:33:10.849969+0000","last_clean":"2024-04-25T17:33:10.849969+0000","last_became_active":"2024-04-25T17:32:32.042033+0000","last_became_peered":"2024-04-25T17:32:32.042033+0000","last_unstale":"2024-04-25T17:33:10.849969+0000","last_undegraded":"2024-04-25T17:33:10.849969+0000","last_fullsized":"2024-04-25T17:33:10.849969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.968383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193657+0000","last_change":"2024-04-25T17:32:35.074492+0000","last_active":"2024-04-25T17:33:10.193657+0000","last_peered":"2024-04-25T17:33:10.193657+0000","last_clean":"2024-04-25T17:33:10.193657+0000","last_became_active":"2024-04-25T17:32:35.073652+0000","last_became_peered":"2024-04-25T17:32:35.073652+0000","last_unstale":"2024-04-25T17:33:10.193657+0000","last_undegraded":"2024-04-25T17:33:10.193657+0000","last_fullsized":"2024-04-25T17:33:10.193657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:00.300453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187848+0000","last_change":"2024-04-25T17:32:32.039568+0000","last_active":"2024-04-25T17:33:10.187848+0000","last_peered":"2024-04-25T17:33:10.187848+0000","last_clean":"2024-04-25T17:33:10.187848+0000","last_became_active":"2024-04-25T17:32:32.039228+0000","last_became_peered":"2024-04-25T17:32:32.039228+0000","last_unstale":"2024-04-25T17:33:10.187848+0000","last_undegraded":"2024-04-25T17:33:10.187848+0000","last_fullsized":"2024-04-25T17:33:10.187848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:17:29.079749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189877+0000","last_change":"2024-04-25T17:32:35.077619+0000","last_active":"2024-04-25T17:33:10.189877+0000","last_peered":"2024-04-25T17:33:10.189877+0000","last_clean":"2024-04-25T17:33:10.189877+0000","last_became_active":"2024-04-25T17:32:35.077309+0000","last_became_peered":"2024-04-25T17:32:35.077309+0000","last_unstale":"2024-04-25T17:33:10.189877+0000","last_undegraded":"2024-04-25T17:33:10.189877+0000","last_fullsized":"2024-04-25T17:33:10.189877+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:41.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193055+0000","last_change":"2024-04-25T17:32:32.044650+0000","last_active":"2024-04-25T17:33:10.193055+0000","last_peered":"2024-04-25T17:33:10.193055+0000","last_clean":"2024-04-25T17:33:10.193055+0000","last_became_active":"2024-04-25T17:32:32.043999+0000","last_became_peered":"2024-04-25T17:32:32.043999+0000","last_unstale":"2024-04-25T17:33:10.193055+0000","last_undegraded":"2024-04-25T17:33:10.193055+0000","last_fullsized":"2024-04-25T17:33:10.193055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:19.463941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185738+0000","last_change":"2024-04-25T17:32:35.080598+0000","last_active":"2024-04-25T17:33:10.185738+0000","last_peered":"2024-04-25T17:33:10.185738+0000","last_clean":"2024-04-25T17:33:10.185738+0000","last_became_active":"2024-04-25T17:32:35.080300+0000","last_became_peered":"2024-04-25T17:32:35.080300+0000","last_unstale":"2024-04-25T17:33:10.185738+0000","last_undegraded":"2024-04-25T17:33:10.185738+0000","last_fullsized":"2024-04-25T17:33:10.185738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:42.149746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:21.186011+0000","last_change":"2024-04-25T17:32:32.043889+0000","last_active":"2024-04-25T17:33:21.186011+0000","last_peered":"2024-04-25T17:33:21.186011+0000","last_clean":"2024-04-25T17:33:21.186011+0000","last_became_active":"2024-04-25T17:32:32.043520+0000","last_became_peered":"2024-04-25T17:32:32.043520+0000","last_unstale":"2024-04-25T17:33:21.186011+0000","last_undegraded":"2024-04-25T17:33:21.186011+0000","last_fullsized":"2024-04-25T17:33:21.186011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:25.134797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187839+0000","last_change":"2024-04-25T17:32:35.071733+0000","last_active":"2024-04-25T17:33:10.187839+0000","last_peered":"2024-04-25T17:33:10.187839+0000","last_clean":"2024-04-25T17:33:10.187839+0000","last_became_active":"2024-04-25T17:32:35.071383+0000","last_became_peered":"2024-04-25T17:32:35.071383+0000","last_unstale":"2024-04-25T17:33:10.187839+0000","last_undegraded":"2024-04-25T17:33:10.187839+0000","last_fullsized":"2024-04-25T17:33:10.187839+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:09:49.433675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190101+0000","last_change":"2024-04-25T17:32:32.044823+0000","last_active":"2024-04-25T17:33:10.190101+0000","last_peered":"2024-04-25T17:33:10.190101+0000","last_clean":"2024-04-25T17:33:10.190101+0000","last_became_active":"2024-04-25T17:32:32.044497+0000","last_became_peered":"2024-04-25T17:32:32.044497+0000","last_unstale":"2024-04-25T17:33:10.190101+0000","last_undegraded":"2024-04-25T17:33:10.190101+0000","last_fullsized":"2024-04-25T17:33:10.190101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:47.811172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.686285+0000","last_change":"2024-04-25T17:32:35.071109+0000","last_active":"2024-04-25T17:33:10.686285+0000","last_peered":"2024-04-25T17:33:10.686285+0000","last_clean":"2024-04-25T17:33:10.686285+0000","last_became_active":"2024-04-25T17:32:35.070782+0000","last_became_peered":"2024-04-25T17:32:35.070782+0000","last_unstale":"2024-04-25T17:33:10.686285+0000","last_undegraded":"2024-04-25T17:33:10.686285+0000","last_fullsized":"2024-04-25T17:33:10.686285+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:37.247274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193437+0000","last_change":"2024-04-25T17:32:32.288683+0000","last_active":"2024-04-25T17:33:10.193437+0000","last_peered":"2024-04-25T17:33:10.193437+0000","last_clean":"2024-04-25T17:33:10.193437+0000","last_became_active":"2024-04-25T17:32:32.288261+0000","last_became_peered":"2024-04-25T17:32:32.288261+0000","last_unstale":"2024-04-25T17:33:10.193437+0000","last_undegraded":"2024-04-25T17:33:10.193437+0000","last_fullsized":"2024-04-25T17:33:10.193437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:36.502434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187579+0000","last_change":"2024-04-25T17:32:35.060828+0000","last_active":"2024-04-25T17:33:10.187579+0000","last_peered":"2024-04-25T17:33:10.187579+0000","last_clean":"2024-04-25T17:33:10.187579+0000","last_became_active":"2024-04-25T17:32:35.060433+0000","last_became_peered":"2024-04-25T17:32:35.060433+0000","last_unstale":"2024-04-25T17:33:10.187579+0000","last_undegraded":"2024-04-25T17:33:10.187579+0000","last_fullsized":"2024-04-25T17:33:10.187579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:32.382042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'28","reported_seq":117,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:22.187513+0000","last_change":"2024-04-25T17:32:32.042979+0000","last_active":"2024-04-25T17:33:22.187513+0000","last_peered":"2024-04-25T17:33:22.187513+0000","last_clean":"2024-04-25T17:33:22.187513+0000","last_became_active":"2024-04-25T17:32:32.042257+0000","last_became_peered":"2024-04-25T17:32:32.042257+0000","last_unstale":"2024-04-25T17:33:22.187513+0000","last_undegraded":"2024-04-25T17:33:22.187513+0000","last_fullsized":"2024-04-25T17:33:22.187513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:19.316803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58078,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":88,"num_write":29,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195603+0000","last_change":"2024-04-25T17:32:35.069994+0000","last_active":"2024-04-25T17:33:10.195603+0000","last_peered":"2024-04-25T17:33:10.195603+0000","last_clean":"2024-04-25T17:33:10.195603+0000","last_became_active":"2024-04-25T17:32:35.069632+0000","last_became_peered":"2024-04-25T17:32:35.069632+0000","last_unstale":"2024-04-25T17:33:10.195603+0000","last_undegraded":"2024-04-25T17:33:10.195603+0000","last_fullsized":"2024-04-25T17:33:10.195603+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:51.169489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187904+0000","last_change":"2024-04-25T17:32:32.042652+0000","last_active":"2024-04-25T17:33:10.187904+0000","last_peered":"2024-04-25T17:33:10.187904+0000","last_clean":"2024-04-25T17:33:10.187904+0000","last_became_active":"2024-04-25T17:32:32.042214+0000","last_became_peered":"2024-04-25T17:32:32.042214+0000","last_unstale":"2024-04-25T17:33:10.187904+0000","last_undegraded":"2024-04-25T17:33:10.187904+0000","last_fullsized":"2024-04-25T17:33:10.187904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.662580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187756+0000","last_change":"2024-04-25T17:32:35.074449+0000","last_active":"2024-04-25T17:33:10.187756+0000","last_peered":"2024-04-25T17:33:10.187756+0000","last_clean":"2024-04-25T17:33:10.187756+0000","last_became_active":"2024-04-25T17:32:35.074116+0000","last_became_peered":"2024-04-25T17:32:35.074116+0000","last_unstale":"2024-04-25T17:33:10.187756+0000","last_undegraded":"2024-04-25T17:33:10.187756+0000","last_fullsized":"2024-04-25T17:33:10.187756+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:09.233206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193473+0000","last_change":"2024-04-25T17:32:32.041368+0000","last_active":"2024-04-25T17:33:10.193473+0000","last_peered":"2024-04-25T17:33:10.193473+0000","last_clean":"2024-04-25T17:33:10.193473+0000","last_became_active":"2024-04-25T17:32:32.040940+0000","last_became_peered":"2024-04-25T17:32:32.040940+0000","last_unstale":"2024-04-25T17:33:10.193473+0000","last_undegraded":"2024-04-25T17:33:10.193473+0000","last_fullsized":"2024-04-25T17:33:10.193473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:38.871786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190072+0000","last_change":"2024-04-25T17:32:35.073535+0000","last_active":"2024-04-25T17:33:10.190072+0000","last_peered":"2024-04-25T17:33:10.190072+0000","last_clean":"2024-04-25T17:33:10.190072+0000","last_became_active":"2024-04-25T17:32:35.073190+0000","last_became_peered":"2024-04-25T17:32:35.073190+0000","last_unstale":"2024-04-25T17:33:10.190072+0000","last_undegraded":"2024-04-25T17:33:10.190072+0000","last_fullsized":"2024-04-25T17:33:10.190072+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:19.841978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:21.186069+0000","last_change":"2024-04-25T17:32:32.044359+0000","last_active":"2024-04-25T17:33:21.186069+0000","last_peered":"2024-04-25T17:33:21.186069+0000","last_clean":"2024-04-25T17:33:21.186069+0000","last_became_active":"2024-04-25T17:32:32.044024+0000","last_became_peered":"2024-04-25T17:32:32.044024+0000","last_unstale":"2024-04-25T17:33:21.186069+0000","last_undegraded":"2024-04-25T17:33:21.186069+0000","last_fullsized":"2024-04-25T17:33:21.186069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:58.940554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.851665+0000","last_change":"2024-04-25T17:33:10.851665+0000","last_active":"2024-04-25T17:33:10.851665+0000","last_peered":"2024-04-25T17:33:10.851665+0000","last_clean":"2024-04-25T17:33:10.851665+0000","last_became_active":"2024-04-25T17:33:10.851217+0000","last_became_peered":"2024-04-25T17:33:10.851217+0000","last_unstale":"2024-04-25T17:33:10.851665+0000","last_undegraded":"2024-04-25T17:33:10.851665+0000","last_fullsized":"2024-04-25T17:33:10.851665+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:58.436715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.194439+0000","last_change":"2024-04-25T17:32:32.285823+0000","last_active":"2024-04-25T17:33:10.194439+0000","last_peered":"2024-04-25T17:33:10.194439+0000","last_clean":"2024-04-25T17:33:10.194439+0000","last_became_active":"2024-04-25T17:32:32.285416+0000","last_became_peered":"2024-04-25T17:32:32.285416+0000","last_unstale":"2024-04-25T17:33:10.194439+0000","last_undegraded":"2024-04-25T17:33:10.194439+0000","last_fullsized":"2024-04-25T17:33:10.194439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:24.473406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193119+0000","last_change":"2024-04-25T17:32:35.072626+0000","last_active":"2024-04-25T17:33:10.193119+0000","last_peered":"2024-04-25T17:33:10.193119+0000","last_clean":"2024-04-25T17:33:10.193119+0000","last_became_active":"2024-04-25T17:32:35.072264+0000","last_became_peered":"2024-04-25T17:32:35.072264+0000","last_unstale":"2024-04-25T17:33:10.193119+0000","last_undegraded":"2024-04-25T17:33:10.193119+0000","last_fullsized":"2024-04-25T17:33:10.193119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:54.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195985+0000","last_change":"2024-04-25T17:32:32.045589+0000","last_active":"2024-04-25T17:33:10.195985+0000","last_peered":"2024-04-25T17:33:10.195985+0000","last_clean":"2024-04-25T17:33:10.195985+0000","last_became_active":"2024-04-25T17:32:32.044838+0000","last_became_peered":"2024-04-25T17:32:32.044838+0000","last_unstale":"2024-04-25T17:33:10.195985+0000","last_undegraded":"2024-04-25T17:33:10.195985+0000","last_fullsized":"2024-04-25T17:33:10.195985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:05.748335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190137+0000","last_change":"2024-04-25T17:32:35.075077+0000","last_active":"2024-04-25T17:33:10.190137+0000","last_peered":"2024-04-25T17:33:10.190137+0000","last_clean":"2024-04-25T17:33:10.190137+0000","last_became_active":"2024-04-25T17:32:35.074622+0000","last_became_peered":"2024-04-25T17:32:35.074622+0000","last_unstale":"2024-04-25T17:33:10.190137+0000","last_undegraded":"2024-04-25T17:33:10.190137+0000","last_fullsized":"2024-04-25T17:33:10.190137+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:53.435637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185818+0000","last_change":"2024-04-25T17:32:32.288707+0000","last_active":"2024-04-25T17:33:10.185818+0000","last_peered":"2024-04-25T17:33:10.185818+0000","last_clean":"2024-04-25T17:33:10.185818+0000","last_became_active":"2024-04-25T17:32:32.288321+0000","last_became_peered":"2024-04-25T17:32:32.288321+0000","last_unstale":"2024-04-25T17:33:10.185818+0000","last_undegraded":"2024-04-25T17:33:10.185818+0000","last_fullsized":"2024-04-25T17:33:10.185818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:00.113662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'1","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850203+0000","last_change":"2024-04-25T17:32:35.072100+0000","last_active":"2024-04-25T17:33:10.850203+0000","last_peered":"2024-04-25T17:33:10.850203+0000","last_clean":"2024-04-25T17:33:10.850203+0000","last_became_active":"2024-04-25T17:32:35.070706+0000","last_became_peered":"2024-04-25T17:32:35.070706+0000","last_unstale":"2024-04-25T17:33:10.850203+0000","last_undegraded":"2024-04-25T17:33:10.850203+0000","last_fullsized":"2024-04-25T17:33:10.850203+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:18.717385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190033+0000","last_change":"2024-04-25T17:32:32.042700+0000","last_active":"2024-04-25T17:33:10.190033+0000","last_peered":"2024-04-25T17:33:10.190033+0000","last_clean":"2024-04-25T17:33:10.190033+0000","last_became_active":"2024-04-25T17:32:32.042383+0000","last_became_peered":"2024-04-25T17:32:32.042383+0000","last_unstale":"2024-04-25T17:33:10.190033+0000","last_undegraded":"2024-04-25T17:33:10.190033+0000","last_fullsized":"2024-04-25T17:33:10.190033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:11.128322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850056+0000","last_change":"2024-04-25T17:32:35.072336+0000","last_active":"2024-04-25T17:33:10.850056+0000","last_peered":"2024-04-25T17:33:10.850056+0000","last_clean":"2024-04-25T17:33:10.850056+0000","last_became_active":"2024-04-25T17:32:35.069830+0000","last_became_peered":"2024-04-25T17:32:35.069830+0000","last_unstale":"2024-04-25T17:33:10.850056+0000","last_undegraded":"2024-04-25T17:33:10.850056+0000","last_fullsized":"2024-04-25T17:33:10.850056+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:12.887428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193363+0000","last_change":"2024-04-25T17:32:35.071375+0000","last_active":"2024-04-25T17:33:10.193363+0000","last_peered":"2024-04-25T17:33:10.193363+0000","last_clean":"2024-04-25T17:33:10.193363+0000","last_became_active":"2024-04-25T17:32:35.071063+0000","last_became_peered":"2024-04-25T17:32:35.071063+0000","last_unstale":"2024-04-25T17:33:10.193363+0000","last_undegraded":"2024-04-25T17:33:10.193363+0000","last_fullsized":"2024-04-25T17:33:10.193363+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:41.917038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:16.178153+0000","last_change":"2024-04-25T17:32:32.286941+0000","last_active":"2024-04-25T17:33:16.178153+0000","last_peered":"2024-04-25T17:33:16.178153+0000","last_clean":"2024-04-25T17:33:16.178153+0000","last_became_active":"2024-04-25T17:32:32.286608+0000","last_became_peered":"2024-04-25T17:32:32.286608+0000","last_unstale":"2024-04-25T17:33:16.178153+0000","last_undegraded":"2024-04-25T17:33:16.178153+0000","last_fullsized":"2024-04-25T17:33:16.178153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.559724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190113+0000","last_change":"2024-04-25T17:32:35.068701+0000","last_active":"2024-04-25T17:33:10.190113+0000","last_peered":"2024-04-25T17:33:10.190113+0000","last_clean":"2024-04-25T17:33:10.190113+0000","last_became_active":"2024-04-25T17:32:35.068418+0000","last_became_peered":"2024-04-25T17:32:35.068418+0000","last_unstale":"2024-04-25T17:33:10.190113+0000","last_undegraded":"2024-04-25T17:33:10.190113+0000","last_fullsized":"2024-04-25T17:33:10.190113+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:42:25.431527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605621+0000","last_change":"2024-04-25T17:32:32.042167+0000","last_active":"2024-04-25T17:33:10.605621+0000","last_peered":"2024-04-25T17:33:10.605621+0000","last_clean":"2024-04-25T17:33:10.605621+0000","last_became_active":"2024-04-25T17:32:32.041765+0000","last_became_peered":"2024-04-25T17:32:32.041765+0000","last_unstale":"2024-04-25T17:33:10.605621+0000","last_undegraded":"2024-04-25T17:33:10.605621+0000","last_fullsized":"2024-04-25T17:33:10.605621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:28.901648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189925+0000","last_change":"2024-04-25T17:31:05.391870+0000","last_active":"2024-04-25T17:33:10.189925+0000","last_peered":"2024-04-25T17:33:10.189925+0000","last_clean":"2024-04-25T17:33:10.189925+0000","last_became_active":"2024-04-25T17:31:05.391426+0000","last_became_peered":"2024-04-25T17:31:05.391426+0000","last_unstale":"2024-04-25T17:33:10.189925+0000","last_undegraded":"2024-04-25T17:33:10.189925+0000","last_fullsized":"2024-04-25T17:33:10.189925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_clean_scrub_stamp":"2024-04-25T17:27:37.393149+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:18.400543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190518+0000","last_change":"2024-04-25T17:33:10.190518+0000","last_active":"2024-04-25T17:33:10.190518+0000","last_peered":"2024-04-25T17:33:10.190518+0000","last_clean":"2024-04-25T17:33:10.190518+0000","last_became_active":"2024-04-25T17:33:10.190166+0000","last_became_peered":"2024-04-25T17:33:10.190166+0000","last_unstale":"2024-04-25T17:33:10.190518+0000","last_undegraded":"2024-04-25T17:33:10.190518+0000","last_fullsized":"2024-04-25T17:33:10.190518+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:45.707078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.197412+0000","last_change":"2024-04-25T17:33:10.197412+0000","last_active":"2024-04-25T17:33:10.197412+0000","last_peered":"2024-04-25T17:33:10.197412+0000","last_clean":"2024-04-25T17:33:10.197412+0000","last_became_active":"2024-04-25T17:33:10.194388+0000","last_became_peered":"2024-04-25T17:33:10.194388+0000","last_unstale":"2024-04-25T17:33:10.197412+0000","last_undegraded":"2024-04-25T17:33:10.197412+0000","last_fullsized":"2024-04-25T17:33:10.197412+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:35.455072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195684+0000","last_change":"2024-04-25T17:32:32.285845+0000","last_active":"2024-04-25T17:33:10.195684+0000","last_peered":"2024-04-25T17:33:10.195684+0000","last_clean":"2024-04-25T17:33:10.195684+0000","last_became_active":"2024-04-25T17:32:32.285386+0000","last_became_peered":"2024-04-25T17:32:32.285386+0000","last_unstale":"2024-04-25T17:33:10.195684+0000","last_undegraded":"2024-04-25T17:33:10.195684+0000","last_fullsized":"2024-04-25T17:33:10.195684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:12.115900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187921+0000","last_change":"2024-04-25T17:32:35.065808+0000","last_active":"2024-04-25T17:33:10.187921+0000","last_peered":"2024-04-25T17:33:10.187921+0000","last_clean":"2024-04-25T17:33:10.187921+0000","last_became_active":"2024-04-25T17:32:35.065504+0000","last_became_peered":"2024-04-25T17:32:35.065504+0000","last_unstale":"2024-04-25T17:33:10.187921+0000","last_undegraded":"2024-04-25T17:33:10.187921+0000","last_fullsized":"2024-04-25T17:33:10.187921+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:33.438281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:15.082173+0000","last_change":"2024-04-25T17:32:32.045357+0000","last_active":"2024-04-25T17:33:15.082173+0000","last_peered":"2024-04-25T17:33:15.082173+0000","last_clean":"2024-04-25T17:33:15.082173+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:33:15.082173+0000","last_undegraded":"2024-04-25T17:33:15.082173+0000","last_fullsized":"2024-04-25T17:33:15.082173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:36.113130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190361+0000","last_change":"2024-04-25T17:32:35.077889+0000","last_active":"2024-04-25T17:33:10.190361+0000","last_peered":"2024-04-25T17:33:10.190361+0000","last_clean":"2024-04-25T17:33:10.190361+0000","last_became_active":"2024-04-25T17:32:35.076470+0000","last_became_peered":"2024-04-25T17:32:35.076470+0000","last_unstale":"2024-04-25T17:33:10.190361+0000","last_undegraded":"2024-04-25T17:33:10.190361+0000","last_fullsized":"2024-04-25T17:33:10.190361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:45.457146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605847+0000","last_change":"2024-04-25T17:32:32.406826+0000","last_active":"2024-04-25T17:33:10.605847+0000","last_peered":"2024-04-25T17:33:10.605847+0000","last_clean":"2024-04-25T17:33:10.605847+0000","last_became_active":"2024-04-25T17:32:32.406454+0000","last_became_peered":"2024-04-25T17:32:32.406454+0000","last_unstale":"2024-04-25T17:33:10.605847+0000","last_undegraded":"2024-04-25T17:33:10.605847+0000","last_fullsized":"2024-04-25T17:33:10.605847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:21:11.730942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647517+0000","last_change":"2024-04-25T17:32:35.075961+0000","last_active":"2024-04-25T17:33:10.647517+0000","last_peered":"2024-04-25T17:33:10.647517+0000","last_clean":"2024-04-25T17:33:10.647517+0000","last_became_active":"2024-04-25T17:32:35.075595+0000","last_became_peered":"2024-04-25T17:32:35.075595+0000","last_unstale":"2024-04-25T17:33:10.647517+0000","last_undegraded":"2024-04-25T17:33:10.647517+0000","last_fullsized":"2024-04-25T17:33:10.647517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:25.513096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187390+0000","last_change":"2024-04-25T17:32:32.289123+0000","last_active":"2024-04-25T17:33:10.187390+0000","last_peered":"2024-04-25T17:33:10.187390+0000","last_clean":"2024-04-25T17:33:10.187390+0000","last_became_active":"2024-04-25T17:32:32.288783+0000","last_became_peered":"2024-04-25T17:32:32.288783+0000","last_unstale":"2024-04-25T17:33:10.187390+0000","last_undegraded":"2024-04-25T17:33:10.187390+0000","last_fullsized":"2024-04-25T17:33:10.187390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:52.222440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.194458+0000","last_change":"2024-04-25T17:32:35.066364+0000","last_active":"2024-04-25T17:33:10.194458+0000","last_peered":"2024-04-25T17:33:10.194458+0000","last_clean":"2024-04-25T17:33:10.194458+0000","last_became_active":"2024-04-25T17:32:35.065489+0000","last_became_peered":"2024-04-25T17:32:35.065489+0000","last_unstale":"2024-04-25T17:33:10.194458+0000","last_undegraded":"2024-04-25T17:33:10.194458+0000","last_fullsized":"2024-04-25T17:33:10.194458+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:16.817796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189874+0000","last_change":"2024-04-25T17:32:32.040662+0000","last_active":"2024-04-25T17:33:10.189874+0000","last_peered":"2024-04-25T17:33:10.189874+0000","last_clean":"2024-04-25T17:33:10.189874+0000","last_became_active":"2024-04-25T17:32:32.040334+0000","last_became_peered":"2024-04-25T17:32:32.040334+0000","last_unstale":"2024-04-25T17:33:10.189874+0000","last_undegraded":"2024-04-25T17:33:10.189874+0000","last_fullsized":"2024-04-25T17:33:10.189874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:03.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189577+0000","last_change":"2024-04-25T17:32:35.079183+0000","last_active":"2024-04-25T17:33:10.189577+0000","last_peered":"2024-04-25T17:33:10.189577+0000","last_clean":"2024-04-25T17:33:10.189577+0000","last_became_active":"2024-04-25T17:32:35.078472+0000","last_became_peered":"2024-04-25T17:32:35.078472+0000","last_unstale":"2024-04-25T17:33:10.189577+0000","last_undegraded":"2024-04-25T17:33:10.189577+0000","last_fullsized":"2024-04-25T17:33:10.189577+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:39:59.120020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.201283+0000","last_change":"2024-04-25T17:33:10.201283+0000","last_active":"2024-04-25T17:33:10.201283+0000","last_peered":"2024-04-25T17:33:10.201283+0000","last_clean":"2024-04-25T17:33:10.201283+0000","last_became_active":"2024-04-25T17:33:10.196192+0000","last_became_peered":"2024-04-25T17:33:10.196192+0000","last_unstale":"2024-04-25T17:33:10.201283+0000","last_undegraded":"2024-04-25T17:33:10.201283+0000","last_fullsized":"2024-04-25T17:33:10.201283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:03.716366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190342+0000","last_change":"2024-04-25T17:32:35.081593+0000","last_active":"2024-04-25T17:33:10.190342+0000","last_peered":"2024-04-25T17:33:10.190342+0000","last_clean":"2024-04-25T17:33:10.190342+0000","last_became_active":"2024-04-25T17:32:35.081248+0000","last_became_peered":"2024-04-25T17:32:35.081248+0000","last_unstale":"2024-04-25T17:33:10.190342+0000","last_undegraded":"2024-04-25T17:33:10.190342+0000","last_fullsized":"2024-04-25T17:33:10.190342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:41:01.177899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.198901+0000","last_change":"2024-04-25T17:33:10.198901+0000","last_active":"2024-04-25T17:33:10.198901+0000","last_peered":"2024-04-25T17:33:10.198901+0000","last_clean":"2024-04-25T17:33:10.198901+0000","last_became_active":"2024-04-25T17:33:10.192191+0000","last_became_peered":"2024-04-25T17:33:10.192191+0000","last_unstale":"2024-04-25T17:33:10.198901+0000","last_undegraded":"2024-04-25T17:33:10.198901+0000","last_fullsized":"2024-04-25T17:33:10.198901+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:39.760642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190031+0000","last_change":"2024-04-25T17:32:35.068592+0000","last_active":"2024-04-25T17:33:10.190031+0000","last_peered":"2024-04-25T17:33:10.190031+0000","last_clean":"2024-04-25T17:33:10.190031+0000","last_became_active":"2024-04-25T17:32:35.068240+0000","last_became_peered":"2024-04-25T17:32:35.068240+0000","last_unstale":"2024-04-25T17:33:10.190031+0000","last_undegraded":"2024-04-25T17:33:10.190031+0000","last_fullsized":"2024-04-25T17:33:10.190031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:48.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.188180+0000","last_change":"2024-04-25T17:32:32.406702+0000","last_active":"2024-04-25T17:33:10.188180+0000","last_peered":"2024-04-25T17:33:10.188180+0000","last_clean":"2024-04-25T17:33:10.188180+0000","last_became_active":"2024-04-25T17:32:32.406263+0000","last_became_peered":"2024-04-25T17:32:32.406263+0000","last_unstale":"2024-04-25T17:33:10.188180+0000","last_undegraded":"2024-04-25T17:33:10.188180+0000","last_fullsized":"2024-04-25T17:33:10.188180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:27.627002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190508+0000","last_change":"2024-04-25T17:32:35.073585+0000","last_active":"2024-04-25T17:33:10.190508+0000","last_peered":"2024-04-25T17:33:10.190508+0000","last_clean":"2024-04-25T17:33:10.190508+0000","last_became_active":"2024-04-25T17:32:35.072902+0000","last_became_peered":"2024-04-25T17:32:35.072902+0000","last_unstale":"2024-04-25T17:33:10.190508+0000","last_undegraded":"2024-04-25T17:33:10.190508+0000","last_fullsized":"2024-04-25T17:33:10.190508+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:17.277129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193299+0000","last_change":"2024-04-25T17:32:32.042546+0000","last_active":"2024-04-25T17:33:10.193299+0000","last_peered":"2024-04-25T17:33:10.193299+0000","last_clean":"2024-04-25T17:33:10.193299+0000","last_became_active":"2024-04-25T17:32:32.042191+0000","last_became_peered":"2024-04-25T17:32:32.042191+0000","last_unstale":"2024-04-25T17:33:10.193299+0000","last_undegraded":"2024-04-25T17:33:10.193299+0000","last_fullsized":"2024-04-25T17:33:10.193299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:58.126260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'1","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190189+0000","last_change":"2024-04-25T17:32:35.077067+0000","last_active":"2024-04-25T17:33:10.190189+0000","last_peered":"2024-04-25T17:33:10.190189+0000","last_clean":"2024-04-25T17:33:10.190189+0000","last_became_active":"2024-04-25T17:32:35.076701+0000","last_became_peered":"2024-04-25T17:32:35.076701+0000","last_unstale":"2024-04-25T17:33:10.190189+0000","last_undegraded":"2024-04-25T17:33:10.190189+0000","last_fullsized":"2024-04-25T17:33:10.190189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:06.646419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.610032+0000","last_change":"2024-04-25T17:33:10.610032+0000","last_active":"2024-04-25T17:33:10.610032+0000","last_peered":"2024-04-25T17:33:10.610032+0000","last_clean":"2024-04-25T17:33:10.610032+0000","last_became_active":"2024-04-25T17:33:10.607260+0000","last_became_peered":"2024-04-25T17:33:10.607260+0000","last_unstale":"2024-04-25T17:33:10.610032+0000","last_undegraded":"2024-04-25T17:33:10.610032+0000","last_fullsized":"2024-04-25T17:33:10.610032+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:42:41.509257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190342+0000","last_change":"2024-04-25T17:32:35.071351+0000","last_active":"2024-04-25T17:33:10.190342+0000","last_peered":"2024-04-25T17:33:10.190342+0000","last_clean":"2024-04-25T17:33:10.190342+0000","last_became_active":"2024-04-25T17:32:35.070559+0000","last_became_peered":"2024-04-25T17:32:35.070559+0000","last_unstale":"2024-04-25T17:33:10.190342+0000","last_undegraded":"2024-04-25T17:33:10.190342+0000","last_fullsized":"2024-04-25T17:33:10.190342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:23.864826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189627+0000","last_change":"2024-04-25T17:32:35.073326+0000","last_active":"2024-04-25T17:33:10.189627+0000","last_peered":"2024-04-25T17:33:10.189627+0000","last_clean":"2024-04-25T17:33:10.189627+0000","last_became_active":"2024-04-25T17:32:35.072904+0000","last_became_peered":"2024-04-25T17:32:35.072904+0000","last_unstale":"2024-04-25T17:33:10.189627+0000","last_undegraded":"2024-04-25T17:33:10.189627+0000","last_fullsized":"2024-04-25T17:33:10.189627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:50.586127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195444+0000","last_change":"2024-04-25T17:33:10.195444+0000","last_active":"2024-04-25T17:33:10.195444+0000","last_peered":"2024-04-25T17:33:10.195444+0000","last_clean":"2024-04-25T17:33:10.195444+0000","last_became_active":"2024-04-25T17:33:10.195048+0000","last_became_peered":"2024-04-25T17:33:10.195048+0000","last_unstale":"2024-04-25T17:33:10.195444+0000","last_undegraded":"2024-04-25T17:33:10.195444+0000","last_fullsized":"2024-04-25T17:33:10.195444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:03.098618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187678+0000","last_change":"2024-04-25T17:32:35.072053+0000","last_active":"2024-04-25T17:33:10.187678+0000","last_peered":"2024-04-25T17:33:10.187678+0000","last_clean":"2024-04-25T17:33:10.187678+0000","last_became_active":"2024-04-25T17:32:35.071712+0000","last_became_peered":"2024-04-25T17:32:35.071712+0000","last_unstale":"2024-04-25T17:33:10.187678+0000","last_undegraded":"2024-04-25T17:33:10.187678+0000","last_fullsized":"2024-04-25T17:33:10.187678+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:42.928854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647627+0000","last_change":"2024-04-25T17:32:35.061408+0000","last_active":"2024-04-25T17:33:10.647627+0000","last_peered":"2024-04-25T17:33:10.647627+0000","last_clean":"2024-04-25T17:33:10.647627+0000","last_became_active":"2024-04-25T17:32:35.061105+0000","last_became_peered":"2024-04-25T17:32:35.061105+0000","last_unstale":"2024-04-25T17:33:10.647627+0000","last_undegraded":"2024-04-25T17:33:10.647627+0000","last_fullsized":"2024-04-25T17:33:10.647627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:48.242157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.687122+0000","last_change":"2024-04-25T17:33:10.687122+0000","last_active":"2024-04-25T17:33:10.687122+0000","last_peered":"2024-04-25T17:33:10.687122+0000","last_clean":"2024-04-25T17:33:10.687122+0000","last_became_active":"2024-04-25T17:33:10.686755+0000","last_became_peered":"2024-04-25T17:33:10.686755+0000","last_unstale":"2024-04-25T17:33:10.687122+0000","last_undegraded":"2024-04-25T17:33:10.687122+0000","last_fullsized":"2024-04-25T17:33:10.687122+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.999655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189950+0000","last_change":"2024-04-25T17:32:35.077433+0000","last_active":"2024-04-25T17:33:10.189950+0000","last_peered":"2024-04-25T17:33:10.189950+0000","last_clean":"2024-04-25T17:33:10.189950+0000","last_became_active":"2024-04-25T17:32:35.076201+0000","last_became_peered":"2024-04-25T17:32:35.076201+0000","last_unstale":"2024-04-25T17:33:10.189950+0000","last_undegraded":"2024-04-25T17:33:10.189950+0000","last_fullsized":"2024-04-25T17:33:10.189950+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:42.677022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.188374+0000","last_change":"2024-04-25T17:32:35.068017+0000","last_active":"2024-04-25T17:33:10.188374+0000","last_peered":"2024-04-25T17:33:10.188374+0000","last_clean":"2024-04-25T17:33:10.188374+0000","last_became_active":"2024-04-25T17:32:35.067647+0000","last_became_peered":"2024-04-25T17:32:35.067647+0000","last_unstale":"2024-04-25T17:33:10.188374+0000","last_undegraded":"2024-04-25T17:33:10.188374+0000","last_fullsized":"2024-04-25T17:33:10.188374+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:28:11.339402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190441+0000","last_change":"2024-04-25T17:32:35.075192+0000","last_active":"2024-04-25T17:33:10.190441+0000","last_peered":"2024-04-25T17:33:10.190441+0000","last_clean":"2024-04-25T17:33:10.190441+0000","last_became_active":"2024-04-25T17:32:35.074850+0000","last_became_peered":"2024-04-25T17:32:35.074850+0000","last_unstale":"2024-04-25T17:33:10.190441+0000","last_undegraded":"2024-04-25T17:33:10.190441+0000","last_fullsized":"2024-04-25T17:33:10.190441+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:04:41.343312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193064+0000","last_change":"2024-04-25T17:32:35.078324+0000","last_active":"2024-04-25T17:33:10.193064+0000","last_peered":"2024-04-25T17:33:10.193064+0000","last_clean":"2024-04-25T17:33:10.193064+0000","last_became_active":"2024-04-25T17:32:35.077958+0000","last_became_peered":"2024-04-25T17:32:35.077958+0000","last_unstale":"2024-04-25T17:33:10.193064+0000","last_undegraded":"2024-04-25T17:33:10.193064+0000","last_fullsized":"2024-04-25T17:33:10.193064+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:10.148060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189834+0000","last_change":"2024-04-25T17:32:35.078023+0000","last_active":"2024-04-25T17:33:10.189834+0000","last_peered":"2024-04-25T17:33:10.189834+0000","last_clean":"2024-04-25T17:33:10.189834+0000","last_became_active":"2024-04-25T17:32:35.077702+0000","last_became_peered":"2024-04-25T17:32:35.077702+0000","last_unstale":"2024-04-25T17:33:10.189834+0000","last_undegraded":"2024-04-25T17:33:10.189834+0000","last_fullsized":"2024-04-25T17:33:10.189834+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:44:37.933179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187569+0000","last_change":"2024-04-25T17:32:35.078388+0000","last_active":"2024-04-25T17:33:10.187569+0000","last_peered":"2024-04-25T17:33:10.187569+0000","last_clean":"2024-04-25T17:33:10.187569+0000","last_became_active":"2024-04-25T17:32:35.078058+0000","last_became_peered":"2024-04-25T17:32:35.078058+0000","last_unstale":"2024-04-25T17:33:10.187569+0000","last_undegraded":"2024-04-25T17:33:10.187569+0000","last_fullsized":"2024-04-25T17:33:10.187569+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:05.699542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187895+0000","last_change":"2024-04-25T17:32:35.068532+0000","last_active":"2024-04-25T17:33:10.187895+0000","last_peered":"2024-04-25T17:33:10.187895+0000","last_clean":"2024-04-25T17:33:10.187895+0000","last_became_active":"2024-04-25T17:32:35.068225+0000","last_became_peered":"2024-04-25T17:32:35.068225+0000","last_unstale":"2024-04-25T17:33:10.187895+0000","last_undegraded":"2024-04-25T17:33:10.187895+0000","last_fullsized":"2024-04-25T17:33:10.187895+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:09.911999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189767+0000","last_change":"2024-04-25T17:32:35.065526+0000","last_active":"2024-04-25T17:33:10.189767+0000","last_peered":"2024-04-25T17:33:10.189767+0000","last_clean":"2024-04-25T17:33:10.189767+0000","last_became_active":"2024-04-25T17:32:35.065146+0000","last_became_peered":"2024-04-25T17:32:35.065146+0000","last_unstale":"2024-04-25T17:33:10.189767+0000","last_undegraded":"2024-04-25T17:33:10.189767+0000","last_fullsized":"2024-04-25T17:33:10.189767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:33.662055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280256,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62079,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":182,"num_read_kb":154,"num_write":163,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":186237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350999,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":482924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612639,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":1132874,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776164,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":483160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037799,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":483080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299438,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":1075010,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462963,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":482549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":540420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":483820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":541770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":482860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888126,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":483377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724601,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1064960,"data_stored":1074303,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":59428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T17:33:24.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T17:33:25.075 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:25.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:25 smithi098 ceph-mon[29192]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:33:25.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:25 smithi203 ceph-mon[32834]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:33:25.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:25 smithi173 ceph-mon[32856]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:33:26.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:26 smithi098 ceph-mon[29192]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:26.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:26 smithi203 ceph-mon[32834]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:26.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:26 smithi173 ceph-mon[32856]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T17:33:26.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.855+0000 7fe203655700 1 -- 172.21.15.98:0/3743439433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1fc102330 msgr2=0x7fe1fc10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.855+0000 7fe203655700 1 --2- 172.21.15.98:0/3743439433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1fc102330 0x7fe1fc10e7f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400b0a0 tx=0x7fe1f400b3b0 comp rx=0 tx=0).stop 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.855+0000 7fe203655700 1 -- 172.21.15.98:0/3743439433 shutdown_connections 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.855+0000 7fe203655700 1 --2- 172.21.15.98:0/3743439433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1fc102330 0x7fe1fc10e7f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.855+0000 7fe203655700 1 --2- 172.21.15.98:0/3743439433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1fc107920 0x7fe1fc107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.855+0000 7fe203655700 1 --2- 172.21.15.98:0/3743439433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1fc101970 0x7fe1fc101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.855+0000 7fe203655700 1 -- 172.21.15.98:0/3743439433 >> 172.21.15.98:0/3743439433 conn(0x7fe1fc0788f0 msgr2=0x7fe1fc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.855+0000 7fe203655700 1 -- 172.21.15.98:0/3743439433 shutdown_connections 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.855+0000 7fe203655700 1 -- 172.21.15.98:0/3743439433 wait complete. 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe203655700 1 Processor -- start 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe203655700 1 -- start start 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe203655700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1fc101970 0x7fe1fc199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe203655700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1fc102330 0x7fe1fc1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:26.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe203655700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1fc107920 0x7fe1fc19db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe203655700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc111bc0 con 0x7fe1fc102330 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe203655700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc111a40 con 0x7fe1fc101970 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe203655700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc111d40 con 0x7fe1fc107920 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe200bf0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1fc102330 0x7fe1fc1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe200bf0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1fc102330 0x7fe1fc1997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40336/0 (socket says 172.21.15.98:40336) 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.856+0000 7fe200bf0700 1 -- 172.21.15.98:0/1765862145 learned_addr learned my addr 172.21.15.98:0/1765862145 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe2013f1700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1fc101970 0x7fe1fc199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe201bf2700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1fc107920 0x7fe1fc19db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe200bf0700 1 -- 172.21.15.98:0/1765862145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1fc107920 msgr2=0x7fe1fc19db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe200bf0700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1fc107920 0x7fe1fc19db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe200bf0700 1 -- 172.21.15.98:0/1765862145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1fc101970 msgr2=0x7fe1fc199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe200bf0700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1fc101970 0x7fe1fc199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:26.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe200bf0700 1 -- 172.21.15.98:0/1765862145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1fc19e280 con 0x7fe1fc102330 2024-04-25T17:33:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe200bf0700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1fc102330 0x7fe1fc1997b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00b530 tx=0x7fe1ec00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe1f27fc700 1 -- 172.21.15.98:0/1765862145 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec015020 con 0x7fe1fc102330 2024-04-25T17:33:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe1f27fc700 1 -- 172.21.15.98:0/1765862145 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1ec011ce0 con 0x7fe1fc102330 2024-04-25T17:33:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe203655700 1 -- 172.21.15.98:0/1765862145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1fc19e570 con 0x7fe1fc102330 2024-04-25T17:33:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe1f27fc700 1 -- 172.21.15.98:0/1765862145 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec010d20 con 0x7fe1fc102330 2024-04-25T17:33:26.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.857+0000 7fe203655700 1 -- 172.21.15.98:0/1765862145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1fc1a5df0 con 0x7fe1fc102330 2024-04-25T17:33:26.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.858+0000 7fe203655700 1 -- 172.21.15.98:0/1765862145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1fc069730 con 0x7fe1fc102330 2024-04-25T17:33:26.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.859+0000 7fe1f27fc700 1 -- 172.21.15.98:0/1765862145 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe1ec018020 con 0x7fe1fc102330 2024-04-25T17:33:26.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.860+0000 7fe1f27fc700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe1e80a3d50 0x7fe1e80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:26.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.860+0000 7fe2013f1700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe1e80a3d50 0x7fe1e80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:26.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.860+0000 7fe1f27fc700 1 -- 172.21.15.98:0/1765862145 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fe1ec0d0210 con 0x7fe1fc102330 2024-04-25T17:33:26.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.860+0000 7fe2013f1700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe1e80a3d50 0x7fe1e80a6210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8006fd0 tx=0x7fe1f8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:26.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:26.863+0000 7fe1f27fc700 1 -- 172.21.15.98:0/1765862145 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe1ec01f020 con 0x7fe1fc102330 2024-04-25T17:33:27.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.058+0000 7fe203655700 1 -- 172.21.15.98:0/1765862145 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1fc062e80 con 0x7fe1e80a3d50 2024-04-25T17:33:27.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.065+0000 7fe1f27fc700 1 -- 172.21.15.98:0/1765862145 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258888 (secure 0 0 0) 0x7fe1fc062e80 con 0x7fe1e80a3d50 2024-04-25T17:33:27.070 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 -- 172.21.15.98:0/1765862145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe1e80a3d50 msgr2=0x7fe1e80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe1e80a3d50 0x7fe1e80a6210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8006fd0 tx=0x7fe1f8008040 comp rx=0 tx=0).stop 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 -- 172.21.15.98:0/1765862145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1fc102330 msgr2=0x7fe1fc1997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1fc102330 0x7fe1fc1997b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec00b530 tx=0x7fe1ec00b8f0 comp rx=0 tx=0).stop 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 -- 172.21.15.98:0/1765862145 shutdown_connections 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe1e80a3d50 0x7fe1e80a6210 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1fc107920 0x7fe1fc19db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe1fc101970 0x7fe1fc199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 --2- 172.21.15.98:0/1765862145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1fc102330 0x7fe1fc1997b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 -- 172.21.15.98:0/1765862145 >> 172.21.15.98:0/1765862145 conn(0x7fe1fc0788f0 msgr2=0x7fe1fc06b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 -- 172.21.15.98:0/1765862145 shutdown_connections 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:27.068+0000 7fe203655700 1 -- 172.21.15.98:0/1765862145 wait complete. 2024-04-25T17:33:27.072 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T17:33:27.809 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-04-25T17:33:26.978038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652681,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":323,"num_read_kb":278,"num_write":348,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357128,"kb_used_data":12100,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584696,"statfs":{"total":1151940427776,"available":1151574728704,"internally_reserved":0,"allocated":12390400,"data_stored":7750404,"data_compressed":106941,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15877,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":62,"num_read_kb":47,"num_write":17,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003007"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190145+0000","last_change":"2024-04-25T17:32:35.075338+0000","last_active":"2024-04-25T17:33:10.190145+0000","last_peered":"2024-04-25T17:33:10.190145+0000","last_clean":"2024-04-25T17:33:10.190145+0000","last_became_active":"2024-04-25T17:32:35.075029+0000","last_became_peered":"2024-04-25T17:32:35.075029+0000","last_unstale":"2024-04-25T17:33:10.190145+0000","last_undegraded":"2024-04-25T17:33:10.190145+0000","last_fullsized":"2024-04-25T17:33:10.190145+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:23.837337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647523+0000","last_change":"2024-04-25T17:32:35.075422+0000","last_active":"2024-04-25T17:33:10.647523+0000","last_peered":"2024-04-25T17:33:10.647523+0000","last_clean":"2024-04-25T17:33:10.647523+0000","last_became_active":"2024-04-25T17:32:35.071975+0000","last_became_peered":"2024-04-25T17:32:35.071975+0000","last_unstale":"2024-04-25T17:33:10.647523+0000","last_undegraded":"2024-04-25T17:33:10.647523+0000","last_fullsized":"2024-04-25T17:33:10.647523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:20.548869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647315+0000","last_change":"2024-04-25T17:32:35.068153+0000","last_active":"2024-04-25T17:33:10.647315+0000","last_peered":"2024-04-25T17:33:10.647315+0000","last_clean":"2024-04-25T17:33:10.647315+0000","last_became_active":"2024-04-25T17:32:35.065873+0000","last_became_peered":"2024-04-25T17:32:35.065873+0000","last_unstale":"2024-04-25T17:33:10.647315+0000","last_undegraded":"2024-04-25T17:33:10.647315+0000","last_fullsized":"2024-04-25T17:33:10.647315+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:31.381875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605643+0000","last_change":"2024-04-25T17:32:35.073055+0000","last_active":"2024-04-25T17:33:10.605643+0000","last_peered":"2024-04-25T17:33:10.605643+0000","last_clean":"2024-04-25T17:33:10.605643+0000","last_became_active":"2024-04-25T17:32:35.072596+0000","last_became_peered":"2024-04-25T17:32:35.072596+0000","last_unstale":"2024-04-25T17:33:10.605643+0000","last_undegraded":"2024-04-25T17:33:10.605643+0000","last_fullsized":"2024-04-25T17:33:10.605643+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:37.529698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195326+0000","last_change":"2024-04-25T17:32:35.072721+0000","last_active":"2024-04-25T17:33:10.195326+0000","last_peered":"2024-04-25T17:33:10.195326+0000","last_clean":"2024-04-25T17:33:10.195326+0000","last_became_active":"2024-04-25T17:32:35.072370+0000","last_became_peered":"2024-04-25T17:32:35.072370+0000","last_unstale":"2024-04-25T17:33:10.195326+0000","last_undegraded":"2024-04-25T17:33:10.195326+0000","last_fullsized":"2024-04-25T17:33:10.195326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:34.429985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850042+0000","last_change":"2024-04-25T17:32:35.071538+0000","last_active":"2024-04-25T17:33:10.850042+0000","last_peered":"2024-04-25T17:33:10.850042+0000","last_clean":"2024-04-25T17:33:10.850042+0000","last_became_active":"2024-04-25T17:32:35.067427+0000","last_became_peered":"2024-04-25T17:32:35.067427+0000","last_unstale":"2024-04-25T17:33:10.850042+0000","last_undegraded":"2024-04-25T17:33:10.850042+0000","last_fullsized":"2024-04-25T17:33:10.850042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:36.052762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189890+0000","last_change":"2024-04-25T17:32:35.079476+0000","last_active":"2024-04-25T17:33:10.189890+0000","last_peered":"2024-04-25T17:33:10.189890+0000","last_clean":"2024-04-25T17:33:10.189890+0000","last_became_active":"2024-04-25T17:32:35.079170+0000","last_became_peered":"2024-04-25T17:32:35.079170+0000","last_unstale":"2024-04-25T17:33:10.189890+0000","last_undegraded":"2024-04-25T17:33:10.189890+0000","last_fullsized":"2024-04-25T17:33:10.189890+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:30.937670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189780+0000","last_change":"2024-04-25T17:32:35.077261+0000","last_active":"2024-04-25T17:33:10.189780+0000","last_peered":"2024-04-25T17:33:10.189780+0000","last_clean":"2024-04-25T17:33:10.189780+0000","last_became_active":"2024-04-25T17:32:35.076896+0000","last_became_peered":"2024-04-25T17:32:35.076896+0000","last_unstale":"2024-04-25T17:33:10.189780+0000","last_undegraded":"2024-04-25T17:33:10.189780+0000","last_fullsized":"2024-04-25T17:33:10.189780+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:24.082550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187931+0000","last_change":"2024-04-25T17:32:35.072229+0000","last_active":"2024-04-25T17:33:10.187931+0000","last_peered":"2024-04-25T17:33:10.187931+0000","last_clean":"2024-04-25T17:33:10.187931+0000","last_became_active":"2024-04-25T17:32:35.071586+0000","last_became_peered":"2024-04-25T17:32:35.071586+0000","last_unstale":"2024-04-25T17:33:10.187931+0000","last_undegraded":"2024-04-25T17:33:10.187931+0000","last_fullsized":"2024-04-25T17:33:10.187931+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:01.195638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850464+0000","last_change":"2024-04-25T17:32:35.075274+0000","last_active":"2024-04-25T17:33:10.850464+0000","last_peered":"2024-04-25T17:33:10.850464+0000","last_clean":"2024-04-25T17:33:10.850464+0000","last_became_active":"2024-04-25T17:32:35.073821+0000","last_became_peered":"2024-04-25T17:32:35.073821+0000","last_unstale":"2024-04-25T17:33:10.850464+0000","last_undegraded":"2024-04-25T17:33:10.850464+0000","last_fullsized":"2024-04-25T17:33:10.850464+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:43:04.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190303+0000","last_change":"2024-04-25T17:32:35.080747+0000","last_active":"2024-04-25T17:33:10.190303+0000","last_peered":"2024-04-25T17:33:10.190303+0000","last_clean":"2024-04-25T17:33:10.190303+0000","last_became_active":"2024-04-25T17:32:35.080383+0000","last_became_peered":"2024-04-25T17:32:35.080383+0000","last_unstale":"2024-04-25T17:33:10.190303+0000","last_undegraded":"2024-04-25T17:33:10.190303+0000","last_fullsized":"2024-04-25T17:33:10.190303+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:42.128892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605662+0000","last_change":"2024-04-25T17:32:35.061442+0000","last_active":"2024-04-25T17:33:10.605662+0000","last_peered":"2024-04-25T17:33:10.605662+0000","last_clean":"2024-04-25T17:33:10.605662+0000","last_became_active":"2024-04-25T17:32:35.061030+0000","last_became_peered":"2024-04-25T17:32:35.061030+0000","last_unstale":"2024-04-25T17:33:10.605662+0000","last_undegraded":"2024-04-25T17:33:10.605662+0000","last_fullsized":"2024-04-25T17:33:10.605662+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:03.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187927+0000","last_change":"2024-04-25T17:32:35.069698+0000","last_active":"2024-04-25T17:33:10.187927+0000","last_peered":"2024-04-25T17:33:10.187927+0000","last_clean":"2024-04-25T17:33:10.187927+0000","last_became_active":"2024-04-25T17:32:35.069374+0000","last_became_peered":"2024-04-25T17:32:35.069374+0000","last_unstale":"2024-04-25T17:33:10.187927+0000","last_undegraded":"2024-04-25T17:33:10.187927+0000","last_fullsized":"2024-04-25T17:33:10.187927+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:03.214525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647126+0000","last_change":"2024-04-25T17:32:35.069001+0000","last_active":"2024-04-25T17:33:10.647126+0000","last_peered":"2024-04-25T17:33:10.647126+0000","last_clean":"2024-04-25T17:33:10.647126+0000","last_became_active":"2024-04-25T17:32:35.064882+0000","last_became_peered":"2024-04-25T17:32:35.064882+0000","last_unstale":"2024-04-25T17:33:10.647126+0000","last_undegraded":"2024-04-25T17:33:10.647126+0000","last_fullsized":"2024-04-25T17:33:10.647126+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:32.122251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647097+0000","last_change":"2024-04-25T17:32:35.065042+0000","last_active":"2024-04-25T17:33:10.647097+0000","last_peered":"2024-04-25T17:33:10.647097+0000","last_clean":"2024-04-25T17:33:10.647097+0000","last_became_active":"2024-04-25T17:32:35.064719+0000","last_became_peered":"2024-04-25T17:32:35.064719+0000","last_unstale":"2024-04-25T17:33:10.647097+0000","last_undegraded":"2024-04-25T17:33:10.647097+0000","last_fullsized":"2024-04-25T17:33:10.647097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:54.715385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187383+0000","last_change":"2024-04-25T17:32:35.067390+0000","last_active":"2024-04-25T17:33:10.187383+0000","last_peered":"2024-04-25T17:33:10.187383+0000","last_clean":"2024-04-25T17:33:10.187383+0000","last_became_active":"2024-04-25T17:32:35.066846+0000","last_became_peered":"2024-04-25T17:32:35.066846+0000","last_unstale":"2024-04-25T17:33:10.187383+0000","last_undegraded":"2024-04-25T17:33:10.187383+0000","last_fullsized":"2024-04-25T17:33:10.187383+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:24.305767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190375+0000","last_change":"2024-04-25T17:32:35.075349+0000","last_active":"2024-04-25T17:33:10.190375+0000","last_peered":"2024-04-25T17:33:10.190375+0000","last_clean":"2024-04-25T17:33:10.190375+0000","last_became_active":"2024-04-25T17:32:35.075046+0000","last_became_peered":"2024-04-25T17:32:35.075046+0000","last_unstale":"2024-04-25T17:33:10.190375+0000","last_undegraded":"2024-04-25T17:33:10.190375+0000","last_fullsized":"2024-04-25T17:33:10.190375+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:51.703406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185881+0000","last_change":"2024-04-25T17:32:35.077597+0000","last_active":"2024-04-25T17:33:10.185881+0000","last_peered":"2024-04-25T17:33:10.185881+0000","last_clean":"2024-04-25T17:33:10.185881+0000","last_became_active":"2024-04-25T17:32:35.077232+0000","last_became_peered":"2024-04-25T17:32:35.077232+0000","last_unstale":"2024-04-25T17:33:10.185881+0000","last_undegraded":"2024-04-25T17:33:10.185881+0000","last_fullsized":"2024-04-25T17:33:10.185881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:46.842673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.188192+0000","last_change":"2024-04-25T17:32:35.078229+0000","last_active":"2024-04-25T17:33:10.188192+0000","last_peered":"2024-04-25T17:33:10.188192+0000","last_clean":"2024-04-25T17:33:10.188192+0000","last_became_active":"2024-04-25T17:32:35.077899+0000","last_became_peered":"2024-04-25T17:32:35.077899+0000","last_unstale":"2024-04-25T17:33:10.188192+0000","last_undegraded":"2024-04-25T17:33:10.188192+0000","last_fullsized":"2024-04-25T17:33:10.188192+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:22.776371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185950+0000","last_change":"2024-04-25T17:32:32.045385+0000","last_active":"2024-04-25T17:33:10.185950+0000","last_peered":"2024-04-25T17:33:10.185950+0000","last_clean":"2024-04-25T17:33:10.185950+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:33:10.185950+0000","last_undegraded":"2024-04-25T17:33:10.185950+0000","last_fullsized":"2024-04-25T17:33:10.185950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:29.732001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190537+0000","last_change":"2024-04-25T17:32:35.079966+0000","last_active":"2024-04-25T17:33:10.190537+0000","last_peered":"2024-04-25T17:33:10.190537+0000","last_clean":"2024-04-25T17:33:10.190537+0000","last_became_active":"2024-04-25T17:32:35.079649+0000","last_became_peered":"2024-04-25T17:32:35.079649+0000","last_unstale":"2024-04-25T17:33:10.190537+0000","last_undegraded":"2024-04-25T17:33:10.190537+0000","last_fullsized":"2024-04-25T17:33:10.190537+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:19:45.679273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190704+0000","last_change":"2024-04-25T17:32:32.285667+0000","last_active":"2024-04-25T17:33:10.190704+0000","last_peered":"2024-04-25T17:33:10.190704+0000","last_clean":"2024-04-25T17:33:10.190704+0000","last_became_active":"2024-04-25T17:32:32.285330+0000","last_became_peered":"2024-04-25T17:32:32.285330+0000","last_unstale":"2024-04-25T17:33:10.190704+0000","last_undegraded":"2024-04-25T17:33:10.190704+0000","last_fullsized":"2024-04-25T17:33:10.190704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.975280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.686361+0000","last_change":"2024-04-25T17:32:35.068841+0000","last_active":"2024-04-25T17:33:10.686361+0000","last_peered":"2024-04-25T17:33:10.686361+0000","last_clean":"2024-04-25T17:33:10.686361+0000","last_became_active":"2024-04-25T17:32:35.068518+0000","last_became_peered":"2024-04-25T17:32:35.068518+0000","last_unstale":"2024-04-25T17:33:10.686361+0000","last_undegraded":"2024-04-25T17:33:10.686361+0000","last_fullsized":"2024-04-25T17:33:10.686361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:23.290990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195797+0000","last_change":"2024-04-25T17:32:32.045627+0000","last_active":"2024-04-25T17:33:10.195797+0000","last_peered":"2024-04-25T17:33:10.195797+0000","last_clean":"2024-04-25T17:33:10.195797+0000","last_became_active":"2024-04-25T17:32:32.044971+0000","last_became_peered":"2024-04-25T17:32:32.044971+0000","last_unstale":"2024-04-25T17:33:10.195797+0000","last_undegraded":"2024-04-25T17:33:10.195797+0000","last_fullsized":"2024-04-25T17:33:10.195797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:50.590435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187551+0000","last_change":"2024-04-25T17:32:35.070862+0000","last_active":"2024-04-25T17:33:10.187551+0000","last_peered":"2024-04-25T17:33:10.187551+0000","last_clean":"2024-04-25T17:33:10.187551+0000","last_became_active":"2024-04-25T17:32:35.069927+0000","last_became_peered":"2024-04-25T17:32:35.069927+0000","last_unstale":"2024-04-25T17:33:10.187551+0000","last_undegraded":"2024-04-25T17:33:10.187551+0000","last_fullsized":"2024-04-25T17:33:10.187551+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:40:01.799015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195097+0000","last_change":"2024-04-25T17:32:32.046062+0000","last_active":"2024-04-25T17:33:10.195097+0000","last_peered":"2024-04-25T17:33:10.195097+0000","last_clean":"2024-04-25T17:33:10.195097+0000","last_became_active":"2024-04-25T17:32:32.044903+0000","last_became_peered":"2024-04-25T17:32:32.044903+0000","last_unstale":"2024-04-25T17:33:10.195097+0000","last_undegraded":"2024-04-25T17:33:10.195097+0000","last_fullsized":"2024-04-25T17:33:10.195097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:51.099460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193396+0000","last_change":"2024-04-25T17:32:35.075452+0000","last_active":"2024-04-25T17:33:10.193396+0000","last_peered":"2024-04-25T17:33:10.193396+0000","last_clean":"2024-04-25T17:33:10.193396+0000","last_became_active":"2024-04-25T17:32:35.075136+0000","last_became_peered":"2024-04-25T17:32:35.075136+0000","last_unstale":"2024-04-25T17:33:10.193396+0000","last_undegraded":"2024-04-25T17:33:10.193396+0000","last_fullsized":"2024-04-25T17:33:10.193396+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:02.290926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:20.185050+0000","last_change":"2024-04-25T17:32:32.042019+0000","last_active":"2024-04-25T17:33:20.185050+0000","last_peered":"2024-04-25T17:33:20.185050+0000","last_clean":"2024-04-25T17:33:20.185050+0000","last_became_active":"2024-04-25T17:32:32.041641+0000","last_became_peered":"2024-04-25T17:32:32.041641+0000","last_unstale":"2024-04-25T17:33:20.185050+0000","last_undegraded":"2024-04-25T17:33:20.185050+0000","last_fullsized":"2024-04-25T17:33:20.185050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:02.025915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.194503+0000","last_change":"2024-04-25T17:32:35.069715+0000","last_active":"2024-04-25T17:33:10.194503+0000","last_peered":"2024-04-25T17:33:10.194503+0000","last_clean":"2024-04-25T17:33:10.194503+0000","last_became_active":"2024-04-25T17:32:35.069339+0000","last_became_peered":"2024-04-25T17:32:35.069339+0000","last_unstale":"2024-04-25T17:33:10.194503+0000","last_undegraded":"2024-04-25T17:33:10.194503+0000","last_fullsized":"2024-04-25T17:33:10.194503+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:48:13.572261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.849969+0000","last_change":"2024-04-25T17:32:32.042445+0000","last_active":"2024-04-25T17:33:10.849969+0000","last_peered":"2024-04-25T17:33:10.849969+0000","last_clean":"2024-04-25T17:33:10.849969+0000","last_became_active":"2024-04-25T17:32:32.042033+0000","last_became_peered":"2024-04-25T17:32:32.042033+0000","last_unstale":"2024-04-25T17:33:10.849969+0000","last_undegraded":"2024-04-25T17:33:10.849969+0000","last_fullsized":"2024-04-25T17:33:10.849969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.968383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193657+0000","last_change":"2024-04-25T17:32:35.074492+0000","last_active":"2024-04-25T17:33:10.193657+0000","last_peered":"2024-04-25T17:33:10.193657+0000","last_clean":"2024-04-25T17:33:10.193657+0000","last_became_active":"2024-04-25T17:32:35.073652+0000","last_became_peered":"2024-04-25T17:32:35.073652+0000","last_unstale":"2024-04-25T17:33:10.193657+0000","last_undegraded":"2024-04-25T17:33:10.193657+0000","last_fullsized":"2024-04-25T17:33:10.193657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:00.300453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187848+0000","last_change":"2024-04-25T17:32:32.039568+0000","last_active":"2024-04-25T17:33:10.187848+0000","last_peered":"2024-04-25T17:33:10.187848+0000","last_clean":"2024-04-25T17:33:10.187848+0000","last_became_active":"2024-04-25T17:32:32.039228+0000","last_became_peered":"2024-04-25T17:32:32.039228+0000","last_unstale":"2024-04-25T17:33:10.187848+0000","last_undegraded":"2024-04-25T17:33:10.187848+0000","last_fullsized":"2024-04-25T17:33:10.187848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:17:29.079749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189877+0000","last_change":"2024-04-25T17:32:35.077619+0000","last_active":"2024-04-25T17:33:10.189877+0000","last_peered":"2024-04-25T17:33:10.189877+0000","last_clean":"2024-04-25T17:33:10.189877+0000","last_became_active":"2024-04-25T17:32:35.077309+0000","last_became_peered":"2024-04-25T17:32:35.077309+0000","last_unstale":"2024-04-25T17:33:10.189877+0000","last_undegraded":"2024-04-25T17:33:10.189877+0000","last_fullsized":"2024-04-25T17:33:10.189877+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:41.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193055+0000","last_change":"2024-04-25T17:32:32.044650+0000","last_active":"2024-04-25T17:33:10.193055+0000","last_peered":"2024-04-25T17:33:10.193055+0000","last_clean":"2024-04-25T17:33:10.193055+0000","last_became_active":"2024-04-25T17:32:32.043999+0000","last_became_peered":"2024-04-25T17:32:32.043999+0000","last_unstale":"2024-04-25T17:33:10.193055+0000","last_undegraded":"2024-04-25T17:33:10.193055+0000","last_fullsized":"2024-04-25T17:33:10.193055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:19.463941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185738+0000","last_change":"2024-04-25T17:32:35.080598+0000","last_active":"2024-04-25T17:33:10.185738+0000","last_peered":"2024-04-25T17:33:10.185738+0000","last_clean":"2024-04-25T17:33:10.185738+0000","last_became_active":"2024-04-25T17:32:35.080300+0000","last_became_peered":"2024-04-25T17:32:35.080300+0000","last_unstale":"2024-04-25T17:33:10.185738+0000","last_undegraded":"2024-04-25T17:33:10.185738+0000","last_fullsized":"2024-04-25T17:33:10.185738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:42.149746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:21.186011+0000","last_change":"2024-04-25T17:32:32.043889+0000","last_active":"2024-04-25T17:33:21.186011+0000","last_peered":"2024-04-25T17:33:21.186011+0000","last_clean":"2024-04-25T17:33:21.186011+0000","last_became_active":"2024-04-25T17:32:32.043520+0000","last_became_peered":"2024-04-25T17:32:32.043520+0000","last_unstale":"2024-04-25T17:33:21.186011+0000","last_undegraded":"2024-04-25T17:33:21.186011+0000","last_fullsized":"2024-04-25T17:33:21.186011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:25.134797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187839+0000","last_change":"2024-04-25T17:32:35.071733+0000","last_active":"2024-04-25T17:33:10.187839+0000","last_peered":"2024-04-25T17:33:10.187839+0000","last_clean":"2024-04-25T17:33:10.187839+0000","last_became_active":"2024-04-25T17:32:35.071383+0000","last_became_peered":"2024-04-25T17:32:35.071383+0000","last_unstale":"2024-04-25T17:33:10.187839+0000","last_undegraded":"2024-04-25T17:33:10.187839+0000","last_fullsized":"2024-04-25T17:33:10.187839+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:09:49.433675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190101+0000","last_change":"2024-04-25T17:32:32.044823+0000","last_active":"2024-04-25T17:33:10.190101+0000","last_peered":"2024-04-25T17:33:10.190101+0000","last_clean":"2024-04-25T17:33:10.190101+0000","last_became_active":"2024-04-25T17:32:32.044497+0000","last_became_peered":"2024-04-25T17:32:32.044497+0000","last_unstale":"2024-04-25T17:33:10.190101+0000","last_undegraded":"2024-04-25T17:33:10.190101+0000","last_fullsized":"2024-04-25T17:33:10.190101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:47.811172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.686285+0000","last_change":"2024-04-25T17:32:35.071109+0000","last_active":"2024-04-25T17:33:10.686285+0000","last_peered":"2024-04-25T17:33:10.686285+0000","last_clean":"2024-04-25T17:33:10.686285+0000","last_became_active":"2024-04-25T17:32:35.070782+0000","last_became_peered":"2024-04-25T17:32:35.070782+0000","last_unstale":"2024-04-25T17:33:10.686285+0000","last_undegraded":"2024-04-25T17:33:10.686285+0000","last_fullsized":"2024-04-25T17:33:10.686285+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:37.247274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193437+0000","last_change":"2024-04-25T17:32:32.288683+0000","last_active":"2024-04-25T17:33:10.193437+0000","last_peered":"2024-04-25T17:33:10.193437+0000","last_clean":"2024-04-25T17:33:10.193437+0000","last_became_active":"2024-04-25T17:32:32.288261+0000","last_became_peered":"2024-04-25T17:32:32.288261+0000","last_unstale":"2024-04-25T17:33:10.193437+0000","last_undegraded":"2024-04-25T17:33:10.193437+0000","last_fullsized":"2024-04-25T17:33:10.193437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:36.502434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187579+0000","last_change":"2024-04-25T17:32:35.060828+0000","last_active":"2024-04-25T17:33:10.187579+0000","last_peered":"2024-04-25T17:33:10.187579+0000","last_clean":"2024-04-25T17:33:10.187579+0000","last_became_active":"2024-04-25T17:32:35.060433+0000","last_became_peered":"2024-04-25T17:32:35.060433+0000","last_unstale":"2024-04-25T17:33:10.187579+0000","last_undegraded":"2024-04-25T17:33:10.187579+0000","last_fullsized":"2024-04-25T17:33:10.187579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:32.382042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'28","reported_seq":117,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:22.187513+0000","last_change":"2024-04-25T17:32:32.042979+0000","last_active":"2024-04-25T17:33:22.187513+0000","last_peered":"2024-04-25T17:33:22.187513+0000","last_clean":"2024-04-25T17:33:22.187513+0000","last_became_active":"2024-04-25T17:32:32.042257+0000","last_became_peered":"2024-04-25T17:32:32.042257+0000","last_unstale":"2024-04-25T17:33:22.187513+0000","last_undegraded":"2024-04-25T17:33:22.187513+0000","last_fullsized":"2024-04-25T17:33:22.187513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:19.316803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58078,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":88,"num_write":29,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195603+0000","last_change":"2024-04-25T17:32:35.069994+0000","last_active":"2024-04-25T17:33:10.195603+0000","last_peered":"2024-04-25T17:33:10.195603+0000","last_clean":"2024-04-25T17:33:10.195603+0000","last_became_active":"2024-04-25T17:32:35.069632+0000","last_became_peered":"2024-04-25T17:32:35.069632+0000","last_unstale":"2024-04-25T17:33:10.195603+0000","last_undegraded":"2024-04-25T17:33:10.195603+0000","last_fullsized":"2024-04-25T17:33:10.195603+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:51.169489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187904+0000","last_change":"2024-04-25T17:32:32.042652+0000","last_active":"2024-04-25T17:33:10.187904+0000","last_peered":"2024-04-25T17:33:10.187904+0000","last_clean":"2024-04-25T17:33:10.187904+0000","last_became_active":"2024-04-25T17:32:32.042214+0000","last_became_peered":"2024-04-25T17:32:32.042214+0000","last_unstale":"2024-04-25T17:33:10.187904+0000","last_undegraded":"2024-04-25T17:33:10.187904+0000","last_fullsized":"2024-04-25T17:33:10.187904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.662580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187756+0000","last_change":"2024-04-25T17:32:35.074449+0000","last_active":"2024-04-25T17:33:10.187756+0000","last_peered":"2024-04-25T17:33:10.187756+0000","last_clean":"2024-04-25T17:33:10.187756+0000","last_became_active":"2024-04-25T17:32:35.074116+0000","last_became_peered":"2024-04-25T17:32:35.074116+0000","last_unstale":"2024-04-25T17:33:10.187756+0000","last_undegraded":"2024-04-25T17:33:10.187756+0000","last_fullsized":"2024-04-25T17:33:10.187756+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:09.233206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193473+0000","last_change":"2024-04-25T17:32:32.041368+0000","last_active":"2024-04-25T17:33:10.193473+0000","last_peered":"2024-04-25T17:33:10.193473+0000","last_clean":"2024-04-25T17:33:10.193473+0000","last_became_active":"2024-04-25T17:32:32.040940+0000","last_became_peered":"2024-04-25T17:32:32.040940+0000","last_unstale":"2024-04-25T17:33:10.193473+0000","last_undegraded":"2024-04-25T17:33:10.193473+0000","last_fullsized":"2024-04-25T17:33:10.193473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:38.871786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190072+0000","last_change":"2024-04-25T17:32:35.073535+0000","last_active":"2024-04-25T17:33:10.190072+0000","last_peered":"2024-04-25T17:33:10.190072+0000","last_clean":"2024-04-25T17:33:10.190072+0000","last_became_active":"2024-04-25T17:32:35.073190+0000","last_became_peered":"2024-04-25T17:32:35.073190+0000","last_unstale":"2024-04-25T17:33:10.190072+0000","last_undegraded":"2024-04-25T17:33:10.190072+0000","last_fullsized":"2024-04-25T17:33:10.190072+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:19.841978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:21.186069+0000","last_change":"2024-04-25T17:32:32.044359+0000","last_active":"2024-04-25T17:33:21.186069+0000","last_peered":"2024-04-25T17:33:21.186069+0000","last_clean":"2024-04-25T17:33:21.186069+0000","last_became_active":"2024-04-25T17:32:32.044024+0000","last_became_peered":"2024-04-25T17:32:32.044024+0000","last_unstale":"2024-04-25T17:33:21.186069+0000","last_undegraded":"2024-04-25T17:33:21.186069+0000","last_fullsized":"2024-04-25T17:33:21.186069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:58.940554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.851665+0000","last_change":"2024-04-25T17:33:10.851665+0000","last_active":"2024-04-25T17:33:10.851665+0000","last_peered":"2024-04-25T17:33:10.851665+0000","last_clean":"2024-04-25T17:33:10.851665+0000","last_became_active":"2024-04-25T17:33:10.851217+0000","last_became_peered":"2024-04-25T17:33:10.851217+0000","last_unstale":"2024-04-25T17:33:10.851665+0000","last_undegraded":"2024-04-25T17:33:10.851665+0000","last_fullsized":"2024-04-25T17:33:10.851665+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:58.436715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.194439+0000","last_change":"2024-04-25T17:32:32.285823+0000","last_active":"2024-04-25T17:33:10.194439+0000","last_peered":"2024-04-25T17:33:10.194439+0000","last_clean":"2024-04-25T17:33:10.194439+0000","last_became_active":"2024-04-25T17:32:32.285416+0000","last_became_peered":"2024-04-25T17:32:32.285416+0000","last_unstale":"2024-04-25T17:33:10.194439+0000","last_undegraded":"2024-04-25T17:33:10.194439+0000","last_fullsized":"2024-04-25T17:33:10.194439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:24.473406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193119+0000","last_change":"2024-04-25T17:32:35.072626+0000","last_active":"2024-04-25T17:33:10.193119+0000","last_peered":"2024-04-25T17:33:10.193119+0000","last_clean":"2024-04-25T17:33:10.193119+0000","last_became_active":"2024-04-25T17:32:35.072264+0000","last_became_peered":"2024-04-25T17:32:35.072264+0000","last_unstale":"2024-04-25T17:33:10.193119+0000","last_undegraded":"2024-04-25T17:33:10.193119+0000","last_fullsized":"2024-04-25T17:33:10.193119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:54.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195985+0000","last_change":"2024-04-25T17:32:32.045589+0000","last_active":"2024-04-25T17:33:10.195985+0000","last_peered":"2024-04-25T17:33:10.195985+0000","last_clean":"2024-04-25T17:33:10.195985+0000","last_became_active":"2024-04-25T17:32:32.044838+0000","last_became_peered":"2024-04-25T17:32:32.044838+0000","last_unstale":"2024-04-25T17:33:10.195985+0000","last_undegraded":"2024-04-25T17:33:10.195985+0000","last_fullsized":"2024-04-25T17:33:10.195985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:05.748335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190137+0000","last_change":"2024-04-25T17:32:35.075077+0000","last_active":"2024-04-25T17:33:10.190137+0000","last_peered":"2024-04-25T17:33:10.190137+0000","last_clean":"2024-04-25T17:33:10.190137+0000","last_became_active":"2024-04-25T17:32:35.074622+0000","last_became_peered":"2024-04-25T17:32:35.074622+0000","last_unstale":"2024-04-25T17:33:10.190137+0000","last_undegraded":"2024-04-25T17:33:10.190137+0000","last_fullsized":"2024-04-25T17:33:10.190137+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:53.435637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185818+0000","last_change":"2024-04-25T17:32:32.288707+0000","last_active":"2024-04-25T17:33:10.185818+0000","last_peered":"2024-04-25T17:33:10.185818+0000","last_clean":"2024-04-25T17:33:10.185818+0000","last_became_active":"2024-04-25T17:32:32.288321+0000","last_became_peered":"2024-04-25T17:32:32.288321+0000","last_unstale":"2024-04-25T17:33:10.185818+0000","last_undegraded":"2024-04-25T17:33:10.185818+0000","last_fullsized":"2024-04-25T17:33:10.185818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:00.113662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'1","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850203+0000","last_change":"2024-04-25T17:32:35.072100+0000","last_active":"2024-04-25T17:33:10.850203+0000","last_peered":"2024-04-25T17:33:10.850203+0000","last_clean":"2024-04-25T17:33:10.850203+0000","last_became_active":"2024-04-25T17:32:35.070706+0000","last_became_peered":"2024-04-25T17:32:35.070706+0000","last_unstale":"2024-04-25T17:33:10.850203+0000","last_undegraded":"2024-04-25T17:33:10.850203+0000","last_fullsized":"2024-04-25T17:33:10.850203+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:18.717385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190033+0000","last_change":"2024-04-25T17:32:32.042700+0000","last_active":"2024-04-25T17:33:10.190033+0000","last_peered":"2024-04-25T17:33:10.190033+0000","last_clean":"2024-04-25T17:33:10.190033+0000","last_became_active":"2024-04-25T17:32:32.042383+0000","last_became_peered":"2024-04-25T17:32:32.042383+0000","last_unstale":"2024-04-25T17:33:10.190033+0000","last_undegraded":"2024-04-25T17:33:10.190033+0000","last_fullsized":"2024-04-25T17:33:10.190033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:11.128322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850056+0000","last_change":"2024-04-25T17:32:35.072336+0000","last_active":"2024-04-25T17:33:10.850056+0000","last_peered":"2024-04-25T17:33:10.850056+0000","last_clean":"2024-04-25T17:33:10.850056+0000","last_became_active":"2024-04-25T17:32:35.069830+0000","last_became_peered":"2024-04-25T17:32:35.069830+0000","last_unstale":"2024-04-25T17:33:10.850056+0000","last_undegraded":"2024-04-25T17:33:10.850056+0000","last_fullsized":"2024-04-25T17:33:10.850056+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:12.887428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193363+0000","last_change":"2024-04-25T17:32:35.071375+0000","last_active":"2024-04-25T17:33:10.193363+0000","last_peered":"2024-04-25T17:33:10.193363+0000","last_clean":"2024-04-25T17:33:10.193363+0000","last_became_active":"2024-04-25T17:32:35.071063+0000","last_became_peered":"2024-04-25T17:32:35.071063+0000","last_unstale":"2024-04-25T17:33:10.193363+0000","last_undegraded":"2024-04-25T17:33:10.193363+0000","last_fullsized":"2024-04-25T17:33:10.193363+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:41.917038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:21.185316+0000","last_change":"2024-04-25T17:32:32.286941+0000","last_active":"2024-04-25T17:33:21.185316+0000","last_peered":"2024-04-25T17:33:21.185316+0000","last_clean":"2024-04-25T17:33:21.185316+0000","last_became_active":"2024-04-25T17:32:32.286608+0000","last_became_peered":"2024-04-25T17:32:32.286608+0000","last_unstale":"2024-04-25T17:33:21.185316+0000","last_undegraded":"2024-04-25T17:33:21.185316+0000","last_fullsized":"2024-04-25T17:33:21.185316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.559724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190113+0000","last_change":"2024-04-25T17:32:35.068701+0000","last_active":"2024-04-25T17:33:10.190113+0000","last_peered":"2024-04-25T17:33:10.190113+0000","last_clean":"2024-04-25T17:33:10.190113+0000","last_became_active":"2024-04-25T17:32:35.068418+0000","last_became_peered":"2024-04-25T17:32:35.068418+0000","last_unstale":"2024-04-25T17:33:10.190113+0000","last_undegraded":"2024-04-25T17:33:10.190113+0000","last_fullsized":"2024-04-25T17:33:10.190113+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:42:25.431527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605621+0000","last_change":"2024-04-25T17:32:32.042167+0000","last_active":"2024-04-25T17:33:10.605621+0000","last_peered":"2024-04-25T17:33:10.605621+0000","last_clean":"2024-04-25T17:33:10.605621+0000","last_became_active":"2024-04-25T17:32:32.041765+0000","last_became_peered":"2024-04-25T17:32:32.041765+0000","last_unstale":"2024-04-25T17:33:10.605621+0000","last_undegraded":"2024-04-25T17:33:10.605621+0000","last_fullsized":"2024-04-25T17:33:10.605621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:28.901648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189925+0000","last_change":"2024-04-25T17:31:05.391870+0000","last_active":"2024-04-25T17:33:10.189925+0000","last_peered":"2024-04-25T17:33:10.189925+0000","last_clean":"2024-04-25T17:33:10.189925+0000","last_became_active":"2024-04-25T17:31:05.391426+0000","last_became_peered":"2024-04-25T17:31:05.391426+0000","last_unstale":"2024-04-25T17:33:10.189925+0000","last_undegraded":"2024-04-25T17:33:10.189925+0000","last_fullsized":"2024-04-25T17:33:10.189925+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_clean_scrub_stamp":"2024-04-25T17:27:37.393149+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:18.400543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190518+0000","last_change":"2024-04-25T17:33:10.190518+0000","last_active":"2024-04-25T17:33:10.190518+0000","last_peered":"2024-04-25T17:33:10.190518+0000","last_clean":"2024-04-25T17:33:10.190518+0000","last_became_active":"2024-04-25T17:33:10.190166+0000","last_became_peered":"2024-04-25T17:33:10.190166+0000","last_unstale":"2024-04-25T17:33:10.190518+0000","last_undegraded":"2024-04-25T17:33:10.190518+0000","last_fullsized":"2024-04-25T17:33:10.190518+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:45.707078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.197412+0000","last_change":"2024-04-25T17:33:10.197412+0000","last_active":"2024-04-25T17:33:10.197412+0000","last_peered":"2024-04-25T17:33:10.197412+0000","last_clean":"2024-04-25T17:33:10.197412+0000","last_became_active":"2024-04-25T17:33:10.194388+0000","last_became_peered":"2024-04-25T17:33:10.194388+0000","last_unstale":"2024-04-25T17:33:10.197412+0000","last_undegraded":"2024-04-25T17:33:10.197412+0000","last_fullsized":"2024-04-25T17:33:10.197412+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:35.455072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195684+0000","last_change":"2024-04-25T17:32:32.285845+0000","last_active":"2024-04-25T17:33:10.195684+0000","last_peered":"2024-04-25T17:33:10.195684+0000","last_clean":"2024-04-25T17:33:10.195684+0000","last_became_active":"2024-04-25T17:32:32.285386+0000","last_became_peered":"2024-04-25T17:32:32.285386+0000","last_unstale":"2024-04-25T17:33:10.195684+0000","last_undegraded":"2024-04-25T17:33:10.195684+0000","last_fullsized":"2024-04-25T17:33:10.195684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:12.115900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187921+0000","last_change":"2024-04-25T17:32:35.065808+0000","last_active":"2024-04-25T17:33:10.187921+0000","last_peered":"2024-04-25T17:33:10.187921+0000","last_clean":"2024-04-25T17:33:10.187921+0000","last_became_active":"2024-04-25T17:32:35.065504+0000","last_became_peered":"2024-04-25T17:32:35.065504+0000","last_unstale":"2024-04-25T17:33:10.187921+0000","last_undegraded":"2024-04-25T17:33:10.187921+0000","last_fullsized":"2024-04-25T17:33:10.187921+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:33.438281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:18.083369+0000","last_change":"2024-04-25T17:32:32.045357+0000","last_active":"2024-04-25T17:33:18.083369+0000","last_peered":"2024-04-25T17:33:18.083369+0000","last_clean":"2024-04-25T17:33:18.083369+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:33:18.083369+0000","last_undegraded":"2024-04-25T17:33:18.083369+0000","last_fullsized":"2024-04-25T17:33:18.083369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:36.113130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190361+0000","last_change":"2024-04-25T17:32:35.077889+0000","last_active":"2024-04-25T17:33:10.190361+0000","last_peered":"2024-04-25T17:33:10.190361+0000","last_clean":"2024-04-25T17:33:10.190361+0000","last_became_active":"2024-04-25T17:32:35.076470+0000","last_became_peered":"2024-04-25T17:32:35.076470+0000","last_unstale":"2024-04-25T17:33:10.190361+0000","last_undegraded":"2024-04-25T17:33:10.190361+0000","last_fullsized":"2024-04-25T17:33:10.190361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:45.457146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605847+0000","last_change":"2024-04-25T17:32:32.406826+0000","last_active":"2024-04-25T17:33:10.605847+0000","last_peered":"2024-04-25T17:33:10.605847+0000","last_clean":"2024-04-25T17:33:10.605847+0000","last_became_active":"2024-04-25T17:32:32.406454+0000","last_became_peered":"2024-04-25T17:32:32.406454+0000","last_unstale":"2024-04-25T17:33:10.605847+0000","last_undegraded":"2024-04-25T17:33:10.605847+0000","last_fullsized":"2024-04-25T17:33:10.605847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:21:11.730942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647517+0000","last_change":"2024-04-25T17:32:35.075961+0000","last_active":"2024-04-25T17:33:10.647517+0000","last_peered":"2024-04-25T17:33:10.647517+0000","last_clean":"2024-04-25T17:33:10.647517+0000","last_became_active":"2024-04-25T17:32:35.075595+0000","last_became_peered":"2024-04-25T17:32:35.075595+0000","last_unstale":"2024-04-25T17:33:10.647517+0000","last_undegraded":"2024-04-25T17:33:10.647517+0000","last_fullsized":"2024-04-25T17:33:10.647517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:25.513096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187390+0000","last_change":"2024-04-25T17:32:32.289123+0000","last_active":"2024-04-25T17:33:10.187390+0000","last_peered":"2024-04-25T17:33:10.187390+0000","last_clean":"2024-04-25T17:33:10.187390+0000","last_became_active":"2024-04-25T17:32:32.288783+0000","last_became_peered":"2024-04-25T17:32:32.288783+0000","last_unstale":"2024-04-25T17:33:10.187390+0000","last_undegraded":"2024-04-25T17:33:10.187390+0000","last_fullsized":"2024-04-25T17:33:10.187390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:52.222440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.194458+0000","last_change":"2024-04-25T17:32:35.066364+0000","last_active":"2024-04-25T17:33:10.194458+0000","last_peered":"2024-04-25T17:33:10.194458+0000","last_clean":"2024-04-25T17:33:10.194458+0000","last_became_active":"2024-04-25T17:32:35.065489+0000","last_became_peered":"2024-04-25T17:32:35.065489+0000","last_unstale":"2024-04-25T17:33:10.194458+0000","last_undegraded":"2024-04-25T17:33:10.194458+0000","last_fullsized":"2024-04-25T17:33:10.194458+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:16.817796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189874+0000","last_change":"2024-04-25T17:32:32.040662+0000","last_active":"2024-04-25T17:33:10.189874+0000","last_peered":"2024-04-25T17:33:10.189874+0000","last_clean":"2024-04-25T17:33:10.189874+0000","last_became_active":"2024-04-25T17:32:32.040334+0000","last_became_peered":"2024-04-25T17:32:32.040334+0000","last_unstale":"2024-04-25T17:33:10.189874+0000","last_undegraded":"2024-04-25T17:33:10.189874+0000","last_fullsized":"2024-04-25T17:33:10.189874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:03.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189577+0000","last_change":"2024-04-25T17:32:35.079183+0000","last_active":"2024-04-25T17:33:10.189577+0000","last_peered":"2024-04-25T17:33:10.189577+0000","last_clean":"2024-04-25T17:33:10.189577+0000","last_became_active":"2024-04-25T17:32:35.078472+0000","last_became_peered":"2024-04-25T17:32:35.078472+0000","last_unstale":"2024-04-25T17:33:10.189577+0000","last_undegraded":"2024-04-25T17:33:10.189577+0000","last_fullsized":"2024-04-25T17:33:10.189577+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:39:59.120020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.201283+0000","last_change":"2024-04-25T17:33:10.201283+0000","last_active":"2024-04-25T17:33:10.201283+0000","last_peered":"2024-04-25T17:33:10.201283+0000","last_clean":"2024-04-25T17:33:10.201283+0000","last_became_active":"2024-04-25T17:33:10.196192+0000","last_became_peered":"2024-04-25T17:33:10.196192+0000","last_unstale":"2024-04-25T17:33:10.201283+0000","last_undegraded":"2024-04-25T17:33:10.201283+0000","last_fullsized":"2024-04-25T17:33:10.201283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:03.716366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190342+0000","last_change":"2024-04-25T17:32:35.081593+0000","last_active":"2024-04-25T17:33:10.190342+0000","last_peered":"2024-04-25T17:33:10.190342+0000","last_clean":"2024-04-25T17:33:10.190342+0000","last_became_active":"2024-04-25T17:32:35.081248+0000","last_became_peered":"2024-04-25T17:32:35.081248+0000","last_unstale":"2024-04-25T17:33:10.190342+0000","last_undegraded":"2024-04-25T17:33:10.190342+0000","last_fullsized":"2024-04-25T17:33:10.190342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:41:01.177899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.198901+0000","last_change":"2024-04-25T17:33:10.198901+0000","last_active":"2024-04-25T17:33:10.198901+0000","last_peered":"2024-04-25T17:33:10.198901+0000","last_clean":"2024-04-25T17:33:10.198901+0000","last_became_active":"2024-04-25T17:33:10.192191+0000","last_became_peered":"2024-04-25T17:33:10.192191+0000","last_unstale":"2024-04-25T17:33:10.198901+0000","last_undegraded":"2024-04-25T17:33:10.198901+0000","last_fullsized":"2024-04-25T17:33:10.198901+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:39.760642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190031+0000","last_change":"2024-04-25T17:32:35.068592+0000","last_active":"2024-04-25T17:33:10.190031+0000","last_peered":"2024-04-25T17:33:10.190031+0000","last_clean":"2024-04-25T17:33:10.190031+0000","last_became_active":"2024-04-25T17:32:35.068240+0000","last_became_peered":"2024-04-25T17:32:35.068240+0000","last_unstale":"2024-04-25T17:33:10.190031+0000","last_undegraded":"2024-04-25T17:33:10.190031+0000","last_fullsized":"2024-04-25T17:33:10.190031+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:48.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.188180+0000","last_change":"2024-04-25T17:32:32.406702+0000","last_active":"2024-04-25T17:33:10.188180+0000","last_peered":"2024-04-25T17:33:10.188180+0000","last_clean":"2024-04-25T17:33:10.188180+0000","last_became_active":"2024-04-25T17:32:32.406263+0000","last_became_peered":"2024-04-25T17:32:32.406263+0000","last_unstale":"2024-04-25T17:33:10.188180+0000","last_undegraded":"2024-04-25T17:33:10.188180+0000","last_fullsized":"2024-04-25T17:33:10.188180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:27.627002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190508+0000","last_change":"2024-04-25T17:32:35.073585+0000","last_active":"2024-04-25T17:33:10.190508+0000","last_peered":"2024-04-25T17:33:10.190508+0000","last_clean":"2024-04-25T17:33:10.190508+0000","last_became_active":"2024-04-25T17:32:35.072902+0000","last_became_peered":"2024-04-25T17:32:35.072902+0000","last_unstale":"2024-04-25T17:33:10.190508+0000","last_undegraded":"2024-04-25T17:33:10.190508+0000","last_fullsized":"2024-04-25T17:33:10.190508+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:17.277129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193299+0000","last_change":"2024-04-25T17:32:32.042546+0000","last_active":"2024-04-25T17:33:10.193299+0000","last_peered":"2024-04-25T17:33:10.193299+0000","last_clean":"2024-04-25T17:33:10.193299+0000","last_became_active":"2024-04-25T17:32:32.042191+0000","last_became_peered":"2024-04-25T17:32:32.042191+0000","last_unstale":"2024-04-25T17:33:10.193299+0000","last_undegraded":"2024-04-25T17:33:10.193299+0000","last_fullsized":"2024-04-25T17:33:10.193299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:58.126260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'1","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190189+0000","last_change":"2024-04-25T17:32:35.077067+0000","last_active":"2024-04-25T17:33:10.190189+0000","last_peered":"2024-04-25T17:33:10.190189+0000","last_clean":"2024-04-25T17:33:10.190189+0000","last_became_active":"2024-04-25T17:32:35.076701+0000","last_became_peered":"2024-04-25T17:32:35.076701+0000","last_unstale":"2024-04-25T17:33:10.190189+0000","last_undegraded":"2024-04-25T17:33:10.190189+0000","last_fullsized":"2024-04-25T17:33:10.190189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:06.646419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.610032+0000","last_change":"2024-04-25T17:33:10.610032+0000","last_active":"2024-04-25T17:33:10.610032+0000","last_peered":"2024-04-25T17:33:10.610032+0000","last_clean":"2024-04-25T17:33:10.610032+0000","last_became_active":"2024-04-25T17:33:10.607260+0000","last_became_peered":"2024-04-25T17:33:10.607260+0000","last_unstale":"2024-04-25T17:33:10.610032+0000","last_undegraded":"2024-04-25T17:33:10.610032+0000","last_fullsized":"2024-04-25T17:33:10.610032+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:42:41.509257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190342+0000","last_change":"2024-04-25T17:32:35.071351+0000","last_active":"2024-04-25T17:33:10.190342+0000","last_peered":"2024-04-25T17:33:10.190342+0000","last_clean":"2024-04-25T17:33:10.190342+0000","last_became_active":"2024-04-25T17:32:35.070559+0000","last_became_peered":"2024-04-25T17:32:35.070559+0000","last_unstale":"2024-04-25T17:33:10.190342+0000","last_undegraded":"2024-04-25T17:33:10.190342+0000","last_fullsized":"2024-04-25T17:33:10.190342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:23.864826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189627+0000","last_change":"2024-04-25T17:32:35.073326+0000","last_active":"2024-04-25T17:33:10.189627+0000","last_peered":"2024-04-25T17:33:10.189627+0000","last_clean":"2024-04-25T17:33:10.189627+0000","last_became_active":"2024-04-25T17:32:35.072904+0000","last_became_peered":"2024-04-25T17:32:35.072904+0000","last_unstale":"2024-04-25T17:33:10.189627+0000","last_undegraded":"2024-04-25T17:33:10.189627+0000","last_fullsized":"2024-04-25T17:33:10.189627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:50.586127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195444+0000","last_change":"2024-04-25T17:33:10.195444+0000","last_active":"2024-04-25T17:33:10.195444+0000","last_peered":"2024-04-25T17:33:10.195444+0000","last_clean":"2024-04-25T17:33:10.195444+0000","last_became_active":"2024-04-25T17:33:10.195048+0000","last_became_peered":"2024-04-25T17:33:10.195048+0000","last_unstale":"2024-04-25T17:33:10.195444+0000","last_undegraded":"2024-04-25T17:33:10.195444+0000","last_fullsized":"2024-04-25T17:33:10.195444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:03.098618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187678+0000","last_change":"2024-04-25T17:32:35.072053+0000","last_active":"2024-04-25T17:33:10.187678+0000","last_peered":"2024-04-25T17:33:10.187678+0000","last_clean":"2024-04-25T17:33:10.187678+0000","last_became_active":"2024-04-25T17:32:35.071712+0000","last_became_peered":"2024-04-25T17:32:35.071712+0000","last_unstale":"2024-04-25T17:33:10.187678+0000","last_undegraded":"2024-04-25T17:33:10.187678+0000","last_fullsized":"2024-04-25T17:33:10.187678+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:42.928854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647627+0000","last_change":"2024-04-25T17:32:35.061408+0000","last_active":"2024-04-25T17:33:10.647627+0000","last_peered":"2024-04-25T17:33:10.647627+0000","last_clean":"2024-04-25T17:33:10.647627+0000","last_became_active":"2024-04-25T17:32:35.061105+0000","last_became_peered":"2024-04-25T17:32:35.061105+0000","last_unstale":"2024-04-25T17:33:10.647627+0000","last_undegraded":"2024-04-25T17:33:10.647627+0000","last_fullsized":"2024-04-25T17:33:10.647627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:48.242157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.687122+0000","last_change":"2024-04-25T17:33:10.687122+0000","last_active":"2024-04-25T17:33:10.687122+0000","last_peered":"2024-04-25T17:33:10.687122+0000","last_clean":"2024-04-25T17:33:10.687122+0000","last_became_active":"2024-04-25T17:33:10.686755+0000","last_became_peered":"2024-04-25T17:33:10.686755+0000","last_unstale":"2024-04-25T17:33:10.687122+0000","last_undegraded":"2024-04-25T17:33:10.687122+0000","last_fullsized":"2024-04-25T17:33:10.687122+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.999655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189950+0000","last_change":"2024-04-25T17:32:35.077433+0000","last_active":"2024-04-25T17:33:10.189950+0000","last_peered":"2024-04-25T17:33:10.189950+0000","last_clean":"2024-04-25T17:33:10.189950+0000","last_became_active":"2024-04-25T17:32:35.076201+0000","last_became_peered":"2024-04-25T17:32:35.076201+0000","last_unstale":"2024-04-25T17:33:10.189950+0000","last_undegraded":"2024-04-25T17:33:10.189950+0000","last_fullsized":"2024-04-25T17:33:10.189950+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:42.677022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.188374+0000","last_change":"2024-04-25T17:32:35.068017+0000","last_active":"2024-04-25T17:33:10.188374+0000","last_peered":"2024-04-25T17:33:10.188374+0000","last_clean":"2024-04-25T17:33:10.188374+0000","last_became_active":"2024-04-25T17:32:35.067647+0000","last_became_peered":"2024-04-25T17:32:35.067647+0000","last_unstale":"2024-04-25T17:33:10.188374+0000","last_undegraded":"2024-04-25T17:33:10.188374+0000","last_fullsized":"2024-04-25T17:33:10.188374+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:28:11.339402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190441+0000","last_change":"2024-04-25T17:32:35.075192+0000","last_active":"2024-04-25T17:33:10.190441+0000","last_peered":"2024-04-25T17:33:10.190441+0000","last_clean":"2024-04-25T17:33:10.190441+0000","last_became_active":"2024-04-25T17:32:35.074850+0000","last_became_peered":"2024-04-25T17:32:35.074850+0000","last_unstale":"2024-04-25T17:33:10.190441+0000","last_undegraded":"2024-04-25T17:33:10.190441+0000","last_fullsized":"2024-04-25T17:33:10.190441+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:04:41.343312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193064+0000","last_change":"2024-04-25T17:32:35.078324+0000","last_active":"2024-04-25T17:33:10.193064+0000","last_peered":"2024-04-25T17:33:10.193064+0000","last_clean":"2024-04-25T17:33:10.193064+0000","last_became_active":"2024-04-25T17:32:35.077958+0000","last_became_peered":"2024-04-25T17:32:35.077958+0000","last_unstale":"2024-04-25T17:33:10.193064+0000","last_undegraded":"2024-04-25T17:33:10.193064+0000","last_fullsized":"2024-04-25T17:33:10.193064+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:10.148060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189834+0000","last_change":"2024-04-25T17:32:35.078023+0000","last_active":"2024-04-25T17:33:10.189834+0000","last_peered":"2024-04-25T17:33:10.189834+0000","last_clean":"2024-04-25T17:33:10.189834+0000","last_became_active":"2024-04-25T17:32:35.077702+0000","last_became_peered":"2024-04-25T17:32:35.077702+0000","last_unstale":"2024-04-25T17:33:10.189834+0000","last_undegraded":"2024-04-25T17:33:10.189834+0000","last_fullsized":"2024-04-25T17:33:10.189834+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:44:37.933179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187569+0000","last_change":"2024-04-25T17:32:35.078388+0000","last_active":"2024-04-25T17:33:10.187569+0000","last_peered":"2024-04-25T17:33:10.187569+0000","last_clean":"2024-04-25T17:33:10.187569+0000","last_became_active":"2024-04-25T17:32:35.078058+0000","last_became_peered":"2024-04-25T17:32:35.078058+0000","last_unstale":"2024-04-25T17:33:10.187569+0000","last_undegraded":"2024-04-25T17:33:10.187569+0000","last_fullsized":"2024-04-25T17:33:10.187569+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:05.699542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187895+0000","last_change":"2024-04-25T17:32:35.068532+0000","last_active":"2024-04-25T17:33:10.187895+0000","last_peered":"2024-04-25T17:33:10.187895+0000","last_clean":"2024-04-25T17:33:10.187895+0000","last_became_active":"2024-04-25T17:32:35.068225+0000","last_became_peered":"2024-04-25T17:32:35.068225+0000","last_unstale":"2024-04-25T17:33:10.187895+0000","last_undegraded":"2024-04-25T17:33:10.187895+0000","last_fullsized":"2024-04-25T17:33:10.187895+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:09.911999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189767+0000","last_change":"2024-04-25T17:32:35.065526+0000","last_active":"2024-04-25T17:33:10.189767+0000","last_peered":"2024-04-25T17:33:10.189767+0000","last_clean":"2024-04-25T17:33:10.189767+0000","last_became_active":"2024-04-25T17:32:35.065146+0000","last_became_peered":"2024-04-25T17:32:35.065146+0000","last_unstale":"2024-04-25T17:33:10.189767+0000","last_undegraded":"2024-04-25T17:33:10.189767+0000","last_fullsized":"2024-04-25T17:33:10.189767+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:33.662055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280256,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62079,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":187,"num_read_kb":159,"num_write":165,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":190494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351000,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":482924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612639,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":1132874,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776165,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":483160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037800,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":483080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299439,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":1075010,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462964,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":482549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":540420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":483820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":546027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":482860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888126,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":483377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724601,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1064960,"data_stored":1074303,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T17:33:27.813 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T17:33:27.814 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T17:33:27.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T17:33:28.078 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:28.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:28 smithi098 ceph-mon[29192]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-25T17:33:28.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:28 smithi098 ceph-mon[29192]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:33:28.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:28 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:33:28.313 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:28 smithi203 ceph-mon[32834]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-25T17:33:28.313 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:28 smithi203 ceph-mon[32834]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:33:28.313 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:28 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:33:28.383 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:28 smithi173 ceph-mon[32856]: pgmap v330: 97 pgs: 97 active+clean; 637 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-25T17:33:28.383 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:28 smithi173 ceph-mon[32856]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:33:28.383 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:28 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:33:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:30 smithi098 ceph-mon[29192]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-25T17:33:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:30 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:30 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:30 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:30 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:30 smithi203 ceph-mon[32834]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-25T17:33:30.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:30 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:30 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:30 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:30 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.472+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1110944430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d78103570 msgr2=0x7f5d78109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.472+0000 7f5d7ee61700 1 --2- 172.21.15.98:0/1110944430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d78103570 0x7f5d78109e20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000b0d0 tx=0x7f5d7000b3e0 comp rx=0 tx=0).stop 2024-04-25T17:33:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.472+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1110944430 shutdown_connections 2024-04-25T17:33:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.472+0000 7f5d7ee61700 1 --2- 172.21.15.98:0/1110944430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d781019d0 0x7f5d78101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.472+0000 7f5d7ee61700 1 --2- 172.21.15.98:0/1110944430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d78103570 0x7f5d78109e20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.472+0000 7f5d7ee61700 1 --2- 172.21.15.98:0/1110944430 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d78102bd0 0x7f5d78103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.472+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1110944430 >> 172.21.15.98:0/1110944430 conn(0x7f5d780fcf90 msgr2=0x7f5d780ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.472+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1110944430 shutdown_connections 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.472+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1110944430 wait complete. 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d7ee61700 1 Processor -- start 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d7ee61700 1 -- start start 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d7ee61700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d781019d0 0x7f5d78199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d7ee61700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d78102bd0 0x7f5d7819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d7ee61700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d78103570 0x7f5d7819dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d7ee61700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7810d560 con 0x7f5d78102bd0 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d7ee61700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7810d3e0 con 0x7f5d78103570 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d7ee61700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7810d6e0 con 0x7f5d781019d0 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d77fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d78102bd0 0x7f5d7819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:30.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d77fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d78102bd0 0x7f5d7819b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40350/0 (socket says 172.21.15.98:40350) 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.473+0000 7f5d77fff700 1 -- 172.21.15.98:0/1206595128 learned_addr learned my addr 172.21.15.98:0/1206595128 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d7cbfd700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d781019d0 0x7f5d78199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d7d3fe700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d78103570 0x7f5d7819dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d77fff700 1 -- 172.21.15.98:0/1206595128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d781019d0 msgr2=0x7f5d78199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d77fff700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d781019d0 0x7f5d78199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d77fff700 1 -- 172.21.15.98:0/1206595128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d78103570 msgr2=0x7f5d7819dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d77fff700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d78103570 0x7f5d7819dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d77fff700 1 -- 172.21.15.98:0/1206595128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7819e700 con 0x7f5d78102bd0 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d7d3fe700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d78103570 0x7f5d7819dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d77fff700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d78102bd0 0x7f5d7819b7c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400b530 tx=0x7f5d6400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d75ffb700 1 -- 172.21.15.98:0/1206595128 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d64015020 con 0x7f5d78102bd0 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1206595128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7819e990 con 0x7f5d78102bd0 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d75ffb700 1 -- 172.21.15.98:0/1206595128 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d64011ce0 con 0x7f5d78102bd0 2024-04-25T17:33:30.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.474+0000 7f5d75ffb700 1 -- 172.21.15.98:0/1206595128 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d64010e40 con 0x7f5d78102bd0 2024-04-25T17:33:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.475+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1206595128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d7819efa0 con 0x7f5d78102bd0 2024-04-25T17:33:30.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.476+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1206595128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d78050f80 con 0x7f5d78102bd0 2024-04-25T17:33:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.476+0000 7f5d75ffb700 1 -- 172.21.15.98:0/1206595128 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5d64018020 con 0x7f5d78102bd0 2024-04-25T17:33:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.477+0000 7f5d75ffb700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5d680a3d80 0x7f5d680a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.477+0000 7f5d7cbfd700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5d680a3d80 0x7f5d680a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.477+0000 7f5d75ffb700 1 -- 172.21.15.98:0/1206595128 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f5d640cfb30 con 0x7f5d78102bd0 2024-04-25T17:33:30.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.477+0000 7f5d7cbfd700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5d680a3d80 0x7f5d680a6240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5d78102a30 tx=0x7f5d6c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:30.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.481+0000 7f5d75ffb700 1 -- 172.21.15.98:0/1206595128 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d6401f030 con 0x7f5d78102bd0 2024-04-25T17:33:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:30 smithi173 ceph-mon[32856]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-04-25T17:33:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:30 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:30 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:30 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:30 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.808+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1206595128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f5d7819f480 con 0x7f5d78102bd0 2024-04-25T17:33:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.810+0000 7f5d75ffb700 1 -- 172.21.15.98:0/1206595128 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f5d7819f480 con 0x7f5d78102bd0 2024-04-25T17:33:30.811 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-04-25T17:33:30.811 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDqkypmSHNDMBAAp9s+ibw2Z+JTz48bVaj8Cw== 2024-04-25T17:33:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1206595128 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5d680a3d80 msgr2=0x7f5d680a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5d680a3d80 0x7f5d680a6240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5d78102a30 tx=0x7f5d6c009190 comp rx=0 tx=0).stop 2024-04-25T17:33:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1206595128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d78102bd0 msgr2=0x7f5d7819b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d78102bd0 0x7f5d7819b7c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400b530 tx=0x7f5d6400b840 comp rx=0 tx=0).stop 2024-04-25T17:33:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1206595128 shutdown_connections 2024-04-25T17:33:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f5d680a3d80 0x7f5d680a6240 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d781019d0 0x7f5d78199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d78103570 0x7f5d7819dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 --2- 172.21.15.98:0/1206595128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d78102bd0 0x7f5d7819b7c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1206595128 >> 172.21.15.98:0/1206595128 conn(0x7f5d780fcf90 msgr2=0x7f5d780fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1206595128 shutdown_connections 2024-04-25T17:33:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:30.813+0000 7f5d7ee61700 1 -- 172.21.15.98:0/1206595128 wait complete. 2024-04-25T17:33:31.538 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:33:31.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T17:33:31.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T17:33:31.573 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T17:33:31.574 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T17:33:31.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T17:33:31.859 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:32.015 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:32.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:32.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:31 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1206595128' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T17:33:32.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:31 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1206595128' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T17:33:32.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:31 smithi098 ceph-mon[29192]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-25T17:33:32.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:33:32.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:33:32.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:31 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:32.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:32.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:32.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:31 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1206595128' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T17:33:32.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:31 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1206595128' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T17:33:32.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:31 smithi173 ceph-mon[32856]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-25T17:33:32.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:33:32.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:33:32.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:31 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:32.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:32.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:32.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:31 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1206595128' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T17:33:32.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:31 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1206595128' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T17:33:32.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:31 smithi203 ceph-mon[32834]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-25T17:33:32.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:33:32.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:33:32.214 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:31 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 -- 172.21.15.98:0/2528452872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb668103450 msgr2=0x7fb66810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 --2- 172.21.15.98:0/2528452872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb668103450 0x7fb66810f8b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb66000b0a0 tx=0x7fb66000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 -- 172.21.15.98:0/2528452872 shutdown_connections 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 --2- 172.21.15.98:0/2528452872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb668103450 0x7fb66810f8b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 --2- 172.21.15.98:0/2528452872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb668108a50 0x7fb668108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 --2- 172.21.15.98:0/2528452872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb668102a90 0x7fb668102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 -- 172.21.15.98:0/2528452872 >> 172.21.15.98:0/2528452872 conn(0x7fb668079ba0 msgr2=0x7fb668079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 -- 172.21.15.98:0/2528452872 shutdown_connections 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 -- 172.21.15.98:0/2528452872 wait complete. 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 Processor -- start 2024-04-25T17:33:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 -- start start 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.586+0000 7fb66ed40700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb668102a90 0x7fb668199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb66ed40700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb668103450 0x7fb6681996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb66ed40700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb668108a50 0x7fb66819da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb66ed40700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb668112c80 con 0x7fb668103450 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb66ed40700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb668112b00 con 0x7fb668108a50 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb66ed40700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb668112e00 con 0x7fb668102a90 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb667fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb668103450 0x7fb6681996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb667fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb668103450 0x7fb6681996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33050/0 (socket says 172.21.15.98:33050) 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb667fff700 1 -- 172.21.15.98:0/3106257544 learned_addr learned my addr 172.21.15.98:0/3106257544 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb66d2dd700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb668108a50 0x7fb66819da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb66cadc700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb668102a90 0x7fb668199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb667fff700 1 -- 172.21.15.98:0/3106257544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb668102a90 msgr2=0x7fb668199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb667fff700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb668102a90 0x7fb668199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb667fff700 1 -- 172.21.15.98:0/3106257544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb668108a50 msgr2=0x7fb66819da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb667fff700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb668108a50 0x7fb66819da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb667fff700 1 -- 172.21.15.98:0/3106257544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66819e150 con 0x7fb668103450 2024-04-25T17:33:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb66cadc700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb668102a90 0x7fb668199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:33:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb667fff700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb668103450 0x7fb6681996d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb65400b530 tx=0x7fb65400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb665ffb700 1 -- 172.21.15.98:0/3106257544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654015020 con 0x7fb668103450 2024-04-25T17:33:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb665ffb700 1 -- 172.21.15.98:0/3106257544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb654011ce0 con 0x7fb668103450 2024-04-25T17:33:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb66ed40700 1 -- 172.21.15.98:0/3106257544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66819e440 con 0x7fb668103450 2024-04-25T17:33:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.587+0000 7fb665ffb700 1 -- 172.21.15.98:0/3106257544 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654010e40 con 0x7fb668103450 2024-04-25T17:33:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.588+0000 7fb66ed40700 1 -- 172.21.15.98:0/3106257544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6681a5cc0 con 0x7fb668103450 2024-04-25T17:33:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.588+0000 7fb66ed40700 1 -- 172.21.15.98:0/3106257544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb668069730 con 0x7fb668103450 2024-04-25T17:33:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.589+0000 7fb665ffb700 1 -- 172.21.15.98:0/3106257544 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb654018020 con 0x7fb668103450 2024-04-25T17:33:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.590+0000 7fb665ffb700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb6580a3cd0 0x7fb6580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.590+0000 7fb66cadc700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb6580a3cd0 0x7fb6580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.590+0000 7fb665ffb700 1 -- 172.21.15.98:0/3106257544 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fb6540cfd00 con 0x7fb668103450 2024-04-25T17:33:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.590+0000 7fb66cadc700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb6580a3cd0 0x7fb6580a6190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb65c005950 tx=0x7fb65c005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.595+0000 7fb665ffb700 1 -- 172.21.15.98:0/3106257544 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb65401f020 con 0x7fb668103450 2024-04-25T17:33:33.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.850+0000 7fb66ed40700 1 -- 172.21.15.98:0/3106257544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb66802df80 con 0x7fb668103450 2024-04-25T17:33:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.855+0000 7fb665ffb700 1 -- 172.21.15.98:0/3106257544 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264995 (secure 0 0 0) 0x7fb65409c6a0 con 0x7fb668103450 2024-04-25T17:33:33.860 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:33:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 -- 172.21.15.98:0/3106257544 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb6580a3cd0 msgr2=0x7fb6580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb6580a3cd0 0x7fb6580a6190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb65c005950 tx=0x7fb65c005f50 comp rx=0 tx=0).stop 2024-04-25T17:33:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 -- 172.21.15.98:0/3106257544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb668103450 msgr2=0x7fb6681996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb668103450 0x7fb6681996d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb65400b530 tx=0x7fb65400b840 comp rx=0 tx=0).stop 2024-04-25T17:33:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 -- 172.21.15.98:0/3106257544 shutdown_connections 2024-04-25T17:33:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb6580a3cd0 0x7fb6580a6190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb668102a90 0x7fb668199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:33.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb668108a50 0x7fb66819da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 --2- 172.21.15.98:0/3106257544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb668103450 0x7fb6681996d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 -- 172.21.15.98:0/3106257544 >> 172.21.15.98:0/3106257544 conn(0x7fb668079ba0 msgr2=0x7fb668101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 -- 172.21.15.98:0/3106257544 shutdown_connections 2024-04-25T17:33:33.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:33.858+0000 7fb66ed40700 1 -- 172.21.15.98:0/3106257544 wait complete. 2024-04-25T17:33:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:34 smithi098 ceph-mon[29192]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-25T17:33:34.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:34 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3106257544' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T17:33:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:34 smithi203 ceph-mon[32834]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-25T17:33:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:34 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3106257544' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T17:33:34.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:34 smithi173 ceph-mon[32856]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-25T17:33:34.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:34 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3106257544' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T17:33:34.603 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":4145487482},{"type":"v1","addr":"172.21.15.98:6801","nonce":4145487482}]},"active_addr":"172.21.15.98:6801/4145487482","active_change":"2024-04-25T17:25:08.883231+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3206373024}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1530074164}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":674729727}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":2167465649}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":36749710}]}]} 2024-04-25T17:33:34.610 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T17:33:34.610 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T17:33:34.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T17:33:34.851 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:36.351 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:36 smithi098 ceph-mon[29192]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:33:36.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:36 smithi203 ceph-mon[32834]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:33:36.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:36 smithi173 ceph-mon[32856]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:33:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.585+0000 7f557a148700 1 -- 172.21.15.98:0/425592349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574107b00 msgr2=0x7f5574107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.585+0000 7f557a148700 1 --2- 172.21.15.98:0/425592349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574107b00 0x7f5574107ee0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5564009990 tx=0x7f5564009ca0 comp rx=0 tx=0).stop 2024-04-25T17:33:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.585+0000 7f557a148700 1 -- 172.21.15.98:0/425592349 shutdown_connections 2024-04-25T17:33:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.585+0000 7f557a148700 1 --2- 172.21.15.98:0/425592349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5574101da0 0x7f557410f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.585+0000 7f557a148700 1 --2- 172.21.15.98:0/425592349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574107b00 0x7f5574107ee0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.585+0000 7f557a148700 1 --2- 172.21.15.98:0/425592349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55741013e0 0x7f5574101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.585+0000 7f557a148700 1 -- 172.21.15.98:0/425592349 >> 172.21.15.98:0/425592349 conn(0x7f5574076c60 msgr2=0x7f5574077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.586+0000 7f557a148700 1 -- 172.21.15.98:0/425592349 shutdown_connections 2024-04-25T17:33:36.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.586+0000 7f557a148700 1 -- 172.21.15.98:0/425592349 wait complete. 2024-04-25T17:33:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.586+0000 7f557a148700 1 Processor -- start 2024-04-25T17:33:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.586+0000 7f557a148700 1 -- start start 2024-04-25T17:33:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f557a148700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55741013e0 0x7f55741992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f557a148700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5574101da0 0x7f55741997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f557a148700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574107b00 0x7f557419db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f557a148700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5574112cf0 con 0x7f5574107b00 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f557a148700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5574112b70 con 0x7f55741013e0 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f557a148700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5574112e70 con 0x7f5574101da0 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f5573fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574107b00 0x7f557419db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f5573fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574107b00 0x7f557419db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33070/0 (socket says 172.21.15.98:33070) 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f5573fff700 1 -- 172.21.15.98:0/3175431214 learned_addr learned my addr 172.21.15.98:0/3175431214 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f556bfff700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5574101da0 0x7f55741997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f55737fe700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55741013e0 0x7f55741992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f5573fff700 1 -- 172.21.15.98:0/3175431214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5574101da0 msgr2=0x7f55741997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f5573fff700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5574101da0 0x7f55741997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f5573fff700 1 -- 172.21.15.98:0/3175431214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55741013e0 msgr2=0x7f55741992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f5573fff700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55741013e0 0x7f55741992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:36.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f5573fff700 1 -- 172.21.15.98:0/3175431214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557419e2b0 con 0x7f5574107b00 2024-04-25T17:33:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f5573fff700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574107b00 0x7f557419db70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f556c00b970 tx=0x7f556c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f55717fa700 1 -- 172.21.15.98:0/3175431214 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c00d860 con 0x7f5574107b00 2024-04-25T17:33:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f55717fa700 1 -- 172.21.15.98:0/3175431214 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f556c00e430 con 0x7f5574107b00 2024-04-25T17:33:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f557a148700 1 -- 172.21.15.98:0/3175431214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557419e5a0 con 0x7f5574107b00 2024-04-25T17:33:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f55717fa700 1 -- 172.21.15.98:0/3175431214 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c017660 con 0x7f5574107b00 2024-04-25T17:33:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.587+0000 7f557a148700 1 -- 172.21.15.98:0/3175431214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55741a5e20 con 0x7f5574107b00 2024-04-25T17:33:36.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.588+0000 7f557a148700 1 -- 172.21.15.98:0/3175431214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5574069730 con 0x7f5574107b00 2024-04-25T17:33:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.590+0000 7f55717fa700 1 -- 172.21.15.98:0/3175431214 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f556c00da00 con 0x7f5574107b00 2024-04-25T17:33:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.590+0000 7f55717fa700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f55600a3cd0 0x7f55600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.590+0000 7f55737fe700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f55600a3cd0 0x7f55600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.590+0000 7f55717fa700 1 -- 172.21.15.98:0/3175431214 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f556c0d0570 con 0x7f5574107b00 2024-04-25T17:33:36.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.591+0000 7f55737fe700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f55600a3cd0 0x7f55600a6190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5564006010 tx=0x7f5564005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.594+0000 7f55717fa700 1 -- 172.21.15.98:0/3175431214 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f556c013030 con 0x7f5574107b00 2024-04-25T17:33:36.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.793+0000 7f557a148700 1 -- 172.21.15.98:0/3175431214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f557402df80 con 0x7f5574107b00 2024-04-25T17:33:36.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.794+0000 7f55717fa700 1 -- 172.21.15.98:0/3175431214 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22686 (secure 0 0 0) 0x7f556c09ce00 con 0x7f5574107b00 2024-04-25T17:33:36.796 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:33:36.796 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:33:10.182926+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T17:32:30.267079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T17:32:33.956902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:33:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 -- 172.21.15.98:0/3175431214 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f55600a3cd0 msgr2=0x7f55600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f55600a3cd0 0x7f55600a6190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5564006010 tx=0x7f5564005960 comp rx=0 tx=0).stop 2024-04-25T17:33:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 -- 172.21.15.98:0/3175431214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574107b00 msgr2=0x7f557419db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574107b00 0x7f557419db70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f556c00b970 tx=0x7f556c00bc80 comp rx=0 tx=0).stop 2024-04-25T17:33:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 -- 172.21.15.98:0/3175431214 shutdown_connections 2024-04-25T17:33:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f55600a3cd0 0x7f55600a6190 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5574107b00 0x7f557419db70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55741013e0 0x7f55741992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 --2- 172.21.15.98:0/3175431214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5574101da0 0x7f55741997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:36.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 -- 172.21.15.98:0/3175431214 >> 172.21.15.98:0/3175431214 conn(0x7f5574076c60 msgr2=0x7f5574100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 -- 172.21.15.98:0/3175431214 shutdown_connections 2024-04-25T17:33:36.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:36.797+0000 7f557a148700 1 -- 172.21.15.98:0/3175431214 wait complete. 2024-04-25T17:33:37.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:37 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3175431214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:33:37.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:37 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3175431214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:33:37.502 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T17:33:37.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T17:33:37.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:37 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3175431214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:33:37.747 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:38 smithi098 ceph-mon[29192]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:33:38.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:38 smithi203 ceph-mon[32834]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:33:38.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:38 smithi173 ceph-mon[32856]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:33:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.454+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/2829997232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce0101030 msgr2=0x7f7ce0101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.454+0000 7f7ce7a2c700 1 --2- 172.21.15.98:0/2829997232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce0101030 0x7f7ce0101490 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009990 tx=0x7f7cd0009ca0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.454+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/2829997232 shutdown_connections 2024-04-25T17:33:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.454+0000 7f7ce7a2c700 1 --2- 172.21.15.98:0/2829997232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce0108ad0 0x7f7ce0108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.454+0000 7f7ce7a2c700 1 --2- 172.21.15.98:0/2829997232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ce01019d0 0x7f7ce010f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.454+0000 7f7ce7a2c700 1 --2- 172.21.15.98:0/2829997232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce0101030 0x7f7ce0101490 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.454+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/2829997232 >> 172.21.15.98:0/2829997232 conn(0x7f7ce0077fb0 msgr2=0x7f7ce00ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.455+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/2829997232 shutdown_connections 2024-04-25T17:33:39.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.455+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/2829997232 wait complete. 2024-04-25T17:33:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.455+0000 7f7ce7a2c700 1 Processor -- start 2024-04-25T17:33:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.455+0000 7f7ce7a2c700 1 -- start start 2024-04-25T17:33:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce7a2c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce0101030 0x7f7ce0102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce7a2c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ce01019d0 0x7f7ce0102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce7a2c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce0108ad0 0x7f7ce0102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:39.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce7a2c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce0112c40 con 0x7f7ce0101030 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce7a2c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce0112ac0 con 0x7f7ce01019d0 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce7a2c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce0112dc0 con 0x7f7ce0108ad0 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce57c8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce0101030 0x7f7ce0102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce57c8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce0101030 0x7f7ce0102210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33096/0 (socket says 172.21.15.98:33096) 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce57c8700 1 -- 172.21.15.98:0/1589908799 learned_addr learned my addr 172.21.15.98:0/1589908799 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce5fc9700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce0108ad0 0x7f7ce0102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce4fc7700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ce01019d0 0x7f7ce0102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce57c8700 1 -- 172.21.15.98:0/1589908799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce0108ad0 msgr2=0x7f7ce0102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce57c8700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce0108ad0 0x7f7ce0102c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce57c8700 1 -- 172.21.15.98:0/1589908799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ce01019d0 msgr2=0x7f7ce0102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce57c8700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ce01019d0 0x7f7ce0102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce57c8700 1 -- 172.21.15.98:0/1589908799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce01a74b0 con 0x7f7ce0101030 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce5fc9700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce0108ad0 0x7f7ce0102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:33:39.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce57c8700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce0101030 0x7f7ce0102210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00b530 tx=0x7f7cdc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:39.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7cd67fc700 1 -- 172.21.15.98:0/1589908799 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc015020 con 0x7f7ce0101030 2024-04-25T17:33:39.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/1589908799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce01a77a0 con 0x7f7ce0101030 2024-04-25T17:33:39.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7cd67fc700 1 -- 172.21.15.98:0/1589908799 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cdc011ce0 con 0x7f7ce0101030 2024-04-25T17:33:39.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.456+0000 7f7cd67fc700 1 -- 172.21.15.98:0/1589908799 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc010e40 con 0x7f7ce0101030 2024-04-25T17:33:39.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.457+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/1589908799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce01a7c90 con 0x7f7ce0101030 2024-04-25T17:33:39.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.458+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/1589908799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce0069730 con 0x7f7ce0101030 2024-04-25T17:33:39.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.459+0000 7f7cd67fc700 1 -- 172.21.15.98:0/1589908799 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7cdc018020 con 0x7f7ce0101030 2024-04-25T17:33:39.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.459+0000 7f7cd67fc700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f7ccc0a3cd0 0x7f7ccc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:39.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.459+0000 7f7ce4fc7700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f7ccc0a3cd0 0x7f7ccc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:39.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.459+0000 7f7cd67fc700 1 -- 172.21.15.98:0/1589908799 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f7cdc0cfd20 con 0x7f7ce0101030 2024-04-25T17:33:39.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.460+0000 7f7ce4fc7700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f7ccc0a3cd0 0x7f7ccc0a6190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0108110 tx=0x7f7cd00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:39.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.464+0000 7f7cd67fc700 1 -- 172.21.15.98:0/1589908799 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7cdc01f020 con 0x7f7ce0101030 2024-04-25T17:33:39.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.659+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/1589908799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ce0104330 con 0x7f7ce0101030 2024-04-25T17:33:39.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.661+0000 7f7cd67fc700 1 -- 172.21.15.98:0/1589908799 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22686 (secure 0 0 0) 0x7f7cdc09c6c0 con 0x7f7ce0101030 2024-04-25T17:33:39.663 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:33:39.663 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:33:10.182926+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T17:32:30.267079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T17:32:33.956902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:33:39.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/1589908799 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f7ccc0a3cd0 msgr2=0x7f7ccc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f7ccc0a3cd0 0x7f7ccc0a6190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0108110 tx=0x7f7cd00058e0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/1589908799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce0101030 msgr2=0x7f7ce0102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce0101030 0x7f7ce0102210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00b530 tx=0x7f7cdc00b840 comp rx=0 tx=0).stop 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/1589908799 shutdown_connections 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f7ccc0a3cd0 0x7f7ccc0a6190 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce0108ad0 0x7f7ce0102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ce0101030 0x7f7ce0102210 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 --2- 172.21.15.98:0/1589908799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ce01019d0 0x7f7ce0102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/1589908799 >> 172.21.15.98:0/1589908799 conn(0x7f7ce0077fb0 msgr2=0x7f7ce00ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/1589908799 shutdown_connections 2024-04-25T17:33:39.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:39.663+0000 7f7ce7a2c700 1 -- 172.21.15.98:0/1589908799 wait complete. 2024-04-25T17:33:40.354 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:40 smithi098 ceph-mon[29192]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T17:33:40.355 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:40 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1589908799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:33:40.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T17:33:40.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T17:33:40.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T17:33:40.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T17:33:40.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T17:33:40.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T17:33:40.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T17:33:40.387 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T17:33:40.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T17:33:40.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T17:33:40.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T17:33:40.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T17:33:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:40 smithi203 ceph-mon[32834]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T17:33:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:40 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1589908799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:33:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:40 smithi173 ceph-mon[32856]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T17:33:40.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:40 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1589908799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:33:41.055 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:41.056 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:41.063 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:41.118 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:41.144 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:42.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:42 smithi098 ceph-mon[29192]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:42 smithi203 ceph-mon[32834]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:42.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:42 smithi173 ceph-mon[32856]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:43.220 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:43.220 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:43.221 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:43.222 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:43.226 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:43.227 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:33:44.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:44 smithi098 ceph-mon[29192]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:33:44.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:44 smithi203 ceph-mon[32834]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:33:44.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:44 smithi173 ceph-mon[32856]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:33:46.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:46 smithi098 ceph-mon[29192]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:33:46.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:46 smithi203 ceph-mon[32834]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:33:46.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:46 smithi173 ceph-mon[32856]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:33:48.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:48 smithi098 ceph-mon[29192]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:48.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:48 smithi203 ceph-mon[32834]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:48.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:48 smithi173 ceph-mon[32856]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:50.324 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:50 smithi173 ceph-mon[32856]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:33:50.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:50 smithi098 ceph-mon[29192]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:33:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:50 smithi203 ceph-mon[32834]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:33:52.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:52 smithi098 ceph-mon[29192]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:52 smithi203 ceph-mon[32834]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:52 smithi173 ceph-mon[32856]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:33:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.961+0000 7fce612d7700 1 -- 172.21.15.98:0/2355384820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5c107980 msgr2=0x7fce5c107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.961+0000 7fce612d7700 1 --2- 172.21.15.98:0/2355384820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5c107980 0x7fce5c107d60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009990 tx=0x7fce4c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:33:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.961+0000 7fce612d7700 1 -- 172.21.15.98:0/2355384820 shutdown_connections 2024-04-25T17:33:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.961+0000 7fce612d7700 1 --2- 172.21.15.98:0/2355384820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1022f0 0x7fce5c10e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.961+0000 7fce612d7700 1 --2- 172.21.15.98:0/2355384820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5c107980 0x7fce5c107d60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.961+0000 7fce612d7700 1 --2- 172.21.15.98:0/2355384820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c101930 0x7fce5c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.961+0000 7fce612d7700 1 -- 172.21.15.98:0/2355384820 >> 172.21.15.98:0/2355384820 conn(0x7fce5c0788b0 msgr2=0x7fce5c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.962+0000 7fce612d7700 1 -- 172.21.15.98:0/2355384820 shutdown_connections 2024-04-25T17:33:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.962+0000 7fce612d7700 1 -- 172.21.15.98:0/2355384820 wait complete. 2024-04-25T17:33:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.962+0000 7fce612d7700 1 Processor -- start 2024-04-25T17:33:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.962+0000 7fce612d7700 1 -- start start 2024-04-25T17:33:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.962+0000 7fce612d7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c101930 0x7fce5c1034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.962+0000 7fce612d7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1022f0 0x7fce5c103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.962+0000 7fce612d7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5c107980 0x7fce5c103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce612d7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c111b50 con 0x7fce5c107980 2024-04-25T17:33:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce612d7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c1119d0 con 0x7fce5c101930 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce612d7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c111cd0 con 0x7fce5c1022f0 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5c107980 0x7fce5c103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5c107980 0x7fce5c103f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55434/0 (socket says 172.21.15.98:55434) 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5b7fe700 1 -- 172.21.15.98:0/2787819680 learned_addr learned my addr 172.21.15.98:0/2787819680 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5a7fc700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1022f0 0x7fce5c103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5affd700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c101930 0x7fce5c1034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5b7fe700 1 -- 172.21.15.98:0/2787819680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1022f0 msgr2=0x7fce5c103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5b7fe700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1022f0 0x7fce5c103a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5b7fe700 1 -- 172.21.15.98:0/2787819680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c101930 msgr2=0x7fce5c1034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5b7fe700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c101930 0x7fce5c1034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5b7fe700 1 -- 172.21.15.98:0/2787819680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c1a76e0 con 0x7fce5c107980 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5affd700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce5c101930 0x7fce5c1034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:33:53.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce5b7fe700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5c107980 0x7fce5c103f60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fce5400ada0 tx=0x7fce5400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce43fff700 1 -- 172.21.15.98:0/2787819680 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54017020 con 0x7fce5c107980 2024-04-25T17:33:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5c1a79d0 con 0x7fce5c107980 2024-04-25T17:33:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce43fff700 1 -- 172.21.15.98:0/2787819680 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce5400ce30 con 0x7fce5c107980 2024-04-25T17:33:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce43fff700 1 -- 172.21.15.98:0/2787819680 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54005350 con 0x7fce5c107980 2024-04-25T17:33:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.963+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5c1a7f10 con 0x7fce5c107980 2024-04-25T17:33:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.964+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fce5c1068c0 con 0x7fce5c107980 2024-04-25T17:33:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.965+0000 7fce43fff700 1 -- 172.21.15.98:0/2787819680 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fce5400c980 con 0x7fce5c107980 2024-04-25T17:33:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.966+0000 7fce43fff700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fce440a3cd0 0x7fce440a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.966+0000 7fce5affd700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fce440a3cd0 0x7fce440a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.966+0000 7fce43fff700 1 -- 172.21.15.98:0/2787819680 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fce540d06a0 con 0x7fce5c107980 2024-04-25T17:33:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.966+0000 7fce43fff700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] conn(0x7fce440ad810 0x7fce440afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:53.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.966+0000 7fce43fff700 1 -- 172.21.15.98:0/2787819680 --> [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce5400fe50 con 0x7fce440ad810 2024-04-25T17:33:53.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.966+0000 7fce43fff700 1 -- 172.21.15.98:0/2787819680 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fce540d0af0 con 0x7fce5c107980 2024-04-25T17:33:53.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.966+0000 7fce5a7fc700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] conn(0x7fce440ad810 0x7fce440afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:53.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.966+0000 7fce5affd700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fce440a3cd0 0x7fce440a6190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fce4c006010 tx=0x7fce4c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:53.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.967+0000 7fce5a7fc700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] conn(0x7fce440ad810 0x7fce440afcb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:53.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.968+0000 7fce43fff700 1 -- 172.21.15.98:0/2787819680 <== osd.6 v2:172.21.15.98:6818/3355912265 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce5400fe50 con 0x7fce440ad810 2024-04-25T17:33:53.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.984+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 --> [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fce5c103050 con 0x7fce440ad810 2024-04-25T17:33:53.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.984+0000 7fce43fff700 1 -- 172.21.15.98:0/2787819680 <== osd.6 v2:172.21.15.98:6818/3355912265 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fce5c103050 con 0x7fce440ad810 2024-04-25T17:33:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] conn(0x7fce440ad810 msgr2=0x7fce440afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce612d7700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] conn(0x7fce440ad810 0x7fce440afcb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fce440a3cd0 msgr2=0x7fce440a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce612d7700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fce440a3cd0 0x7fce440a6190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fce4c006010 tx=0x7fce4c00b540 comp rx=0 tx=0).stop 2024-04-25T17:33:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5c107980 msgr2=0x7fce5c103f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce612d7700 1 --2- 172.21.15.98:0/2787819680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce5c107980 0x7fce5c103f60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fce5400ada0 tx=0x7fce5400c330 comp rx=0 tx=0).stop 2024-04-25T17:33:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce5b7fe700 1 -- 172.21.15.98:0/2787819680 reap_dead start 2024-04-25T17:33:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 shutdown_connections 2024-04-25T17:33:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 >> 172.21.15.98:0/2787819680 conn(0x7fce5c0788b0 msgr2=0x7fce5c06b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 shutdown_connections 2024-04-25T17:33:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:53.985+0000 7fce612d7700 1 -- 172.21.15.98:0/2787819680 wait complete. 2024-04-25T17:33:54.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:54 smithi098 ceph-mon[29192]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:33:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:54 smithi203 ceph-mon[32834]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:33:54.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.448+0000 7fa09eda1700 1 -- 172.21.15.98:0/493056465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa09806b980 msgr2=0x7fa09810f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:54.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.448+0000 7fa09eda1700 1 --2- 172.21.15.98:0/493056465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa09806b980 0x7fa09810f8f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa09000b0a0 tx=0x7fa09000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:33:54.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.448+0000 7fa09eda1700 1 -- 172.21.15.98:0/493056465 shutdown_connections 2024-04-25T17:33:54.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.448+0000 7fa09eda1700 1 --2- 172.21.15.98:0/493056465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa09806b980 0x7fa09810f8f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.448+0000 7fa09eda1700 1 --2- 172.21.15.98:0/493056465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa098108a20 0x7fa098108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.448+0000 7fa09eda1700 1 --2- 172.21.15.98:0/493056465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa09806afe0 0x7fa09806b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.448+0000 7fa09eda1700 1 -- 172.21.15.98:0/493056465 >> 172.21.15.98:0/493056465 conn(0x7fa0980788b0 msgr2=0x7fa098078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.448+0000 7fa09eda1700 1 -- 172.21.15.98:0/493056465 shutdown_connections 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.448+0000 7fa09eda1700 1 -- 172.21.15.98:0/493056465 wait complete. 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09eda1700 1 Processor -- start 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09eda1700 1 -- start start 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09eda1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa09806afe0 0x7fa0981990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09eda1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa09806b980 0x7fa098199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09eda1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa098108a20 0x7fa09819d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09eda1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa098112c60 con 0x7fa09806afe0 2024-04-25T17:33:54.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09eda1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa098112ae0 con 0x7fa09806b980 2024-04-25T17:33:54.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09eda1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa098112de0 con 0x7fa098108a20 2024-04-25T17:33:54.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09cb3d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa09806afe0 0x7fa0981990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:54.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09cb3d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa09806afe0 0x7fa0981990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55454/0 (socket says 172.21.15.98:55454) 2024-04-25T17:33:54.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09cb3d700 1 -- 172.21.15.98:0/83441958 learned_addr learned my addr 172.21.15.98:0/83441958 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:54.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa097fff700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa09806b980 0x7fa098199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:54.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09d33e700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa098108a20 0x7fa09819d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09cb3d700 1 -- 172.21.15.98:0/83441958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa098108a20 msgr2=0x7fa09819d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09cb3d700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa098108a20 0x7fa09819d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09cb3d700 1 -- 172.21.15.98:0/83441958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa09806b980 msgr2=0x7fa098199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09cb3d700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa09806b980 0x7fa098199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09cb3d700 1 -- 172.21.15.98:0/83441958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa09819e100 con 0x7fa09806afe0 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa097fff700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa09806b980 0x7fa098199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09d33e700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa098108a20 0x7fa09819d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.449+0000 7fa09cb3d700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa09806afe0 0x7fa0981990f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00d950 tx=0x7fa08c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.450+0000 7fa095ffb700 1 -- 172.21.15.98:0/83441958 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08c016070 con 0x7fa09806afe0 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.450+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa09819e3f0 con 0x7fa09806afe0 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.450+0000 7fa095ffb700 1 -- 172.21.15.98:0/83441958 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa08c010460 con 0x7fa09806afe0 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.450+0000 7fa095ffb700 1 -- 172.21.15.98:0/83441958 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08c00f620 con 0x7fa09806afe0 2024-04-25T17:33:54.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.450+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0981a5c70 con 0x7fa09806afe0 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.451+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa0980631d0 con 0x7fa09806afe0 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.452+0000 7fa095ffb700 1 -- 172.21.15.98:0/83441958 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa08c019070 con 0x7fa09806afe0 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.452+0000 7fa095ffb700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa0880a3cd0 0x7fa0880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.452+0000 7fa097fff700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa0880a3cd0 0x7fa0880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.452+0000 7fa095ffb700 1 -- 172.21.15.98:0/83441958 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fa08c0d0700 con 0x7fa09806afe0 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.452+0000 7fa095ffb700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] conn(0x7fa0880ad810 0x7fa0880afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.452+0000 7fa095ffb700 1 -- 172.21.15.98:0/83441958 --> [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa08c013ec0 con 0x7fa0880ad810 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.452+0000 7fa095ffb700 1 -- 172.21.15.98:0/83441958 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa08c0d0b50 con 0x7fa09806afe0 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.453+0000 7fa097fff700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa0880a3cd0 0x7fa0880a6190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa098078e90 tx=0x7fa084008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.453+0000 7fa09d33e700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] conn(0x7fa0880ad810 0x7fa0880afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.453+0000 7fa09d33e700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] conn(0x7fa0880ad810 0x7fa0880afcb0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.454+0000 7fa095ffb700 1 -- 172.21.15.98:0/83441958 <== osd.11 v2:172.21.15.203:6824/2267615274 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa08c013ec0 con 0x7fa0880ad810 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.470+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 --> [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa0980635d0 con 0x7fa0880ad810 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa095ffb700 1 -- 172.21.15.98:0/83441958 <== osd.11 v2:172.21.15.203:6824/2267615274 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa0980635d0 con 0x7fa0880ad810 2024-04-25T17:33:54.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 >> [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] conn(0x7fa0880ad810 msgr2=0x7fa0880afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09eda1700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] conn(0x7fa0880ad810 0x7fa0880afcb0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa0880a3cd0 msgr2=0x7fa0880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09eda1700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa0880a3cd0 0x7fa0880a6190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa098078e90 tx=0x7fa084008040 comp rx=0 tx=0).stop 2024-04-25T17:33:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa09806afe0 msgr2=0x7fa0981990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09eda1700 1 --2- 172.21.15.98:0/83441958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa09806afe0 0x7fa0981990f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00d950 tx=0x7fa08c00dd10 comp rx=0 tx=0).stop 2024-04-25T17:33:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09d33e700 1 -- 172.21.15.98:0/83441958 reap_dead start 2024-04-25T17:33:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 shutdown_connections 2024-04-25T17:33:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 >> 172.21.15.98:0/83441958 conn(0x7fa0980788b0 msgr2=0x7fa0980ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 shutdown_connections 2024-04-25T17:33:54.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:54.471+0000 7fa09eda1700 1 -- 172.21.15.98:0/83441958 wait complete. 2024-04-25T17:33:54.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:54 smithi173 ceph-mon[32856]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:33:56.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:56 smithi098 ceph-mon[29192]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T17:33:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:56 smithi203 ceph-mon[32834]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T17:33:56.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.516+0000 7fda10381700 1 -- 172.21.15.98:0/379862398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda08103490 msgr2=0x7fda0810f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.516+0000 7fda10381700 1 --2- 172.21.15.98:0/379862398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda08103490 0x7fda0810f970 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fda0000b0a0 tx=0x7fda0000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.516+0000 7fda10381700 1 -- 172.21.15.98:0/379862398 shutdown_connections 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.516+0000 7fda10381700 1 --2- 172.21.15.98:0/379862398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda08103490 0x7fda0810f970 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.516+0000 7fda10381700 1 --2- 172.21.15.98:0/379862398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08108aa0 0x7fda08108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.516+0000 7fda10381700 1 --2- 172.21.15.98:0/379862398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda08102ad0 0x7fda08102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.516+0000 7fda10381700 1 -- 172.21.15.98:0/379862398 >> 172.21.15.98:0/379862398 conn(0x7fda08079c30 msgr2=0x7fda0807a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.516+0000 7fda10381700 1 -- 172.21.15.98:0/379862398 shutdown_connections 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.516+0000 7fda10381700 1 -- 172.21.15.98:0/379862398 wait complete. 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.516+0000 7fda10381700 1 Processor -- start 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda10381700 1 -- start start 2024-04-25T17:33:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda10381700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda08102ad0 0x7fda081991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda10381700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda08103490 0x7fda081996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda10381700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08108aa0 0x7fda0819da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda10381700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fda08112ce0 con 0x7fda08102ad0 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda10381700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fda08112b60 con 0x7fda08108aa0 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda10381700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fda08112e60 con 0x7fda08103490 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e11d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda08102ad0 0x7fda081991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e11d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda08102ad0 0x7fda081991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55490/0 (socket says 172.21.15.98:55490) 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e11d700 1 -- 172.21.15.98:0/392170230 learned_addr learned my addr 172.21.15.98:0/392170230 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e91e700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08108aa0 0x7fda0819da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0d91c700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda08103490 0x7fda081996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e11d700 1 -- 172.21.15.98:0/392170230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda08103490 msgr2=0x7fda081996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e11d700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda08103490 0x7fda081996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e11d700 1 -- 172.21.15.98:0/392170230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08108aa0 msgr2=0x7fda0819da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e11d700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08108aa0 0x7fda0819da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e11d700 1 -- 172.21.15.98:0/392170230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0819e1c0 con 0x7fda08102ad0 2024-04-25T17:33:56.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e91e700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fda08108aa0 0x7fda0819da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:33:56.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.517+0000 7fda0e11d700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda08102ad0 0x7fda081991b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fda0400c920 tx=0x7fda0400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:56.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.518+0000 7fd9ff7fe700 1 -- 172.21.15.98:0/392170230 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda04007b00 con 0x7fda08102ad0 2024-04-25T17:33:56.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.518+0000 7fd9ff7fe700 1 -- 172.21.15.98:0/392170230 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda04004ce0 con 0x7fda08102ad0 2024-04-25T17:33:56.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.518+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda0819e4b0 con 0x7fda08102ad0 2024-04-25T17:33:56.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.518+0000 7fd9ff7fe700 1 -- 172.21.15.98:0/392170230 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda04005650 con 0x7fda08102ad0 2024-04-25T17:33:56.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.518+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda081a5d30 con 0x7fda08102ad0 2024-04-25T17:33:56.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.519+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fda08063150 con 0x7fda08102ad0 2024-04-25T17:33:56.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.520+0000 7fd9ff7fe700 1 -- 172.21.15.98:0/392170230 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fda04004750 con 0x7fda08102ad0 2024-04-25T17:33:56.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.520+0000 7fd9ff7fe700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd9f40a3cd0 0x7fd9f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:56.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.521+0000 7fda0d91c700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd9f40a3cd0 0x7fd9f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:56.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.521+0000 7fd9ff7fe700 1 -- 172.21.15.98:0/392170230 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fda040d05d0 con 0x7fda08102ad0 2024-04-25T17:33:56.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.521+0000 7fd9ff7fe700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] conn(0x7fd9f40ad810 0x7fd9f40afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.521+0000 7fd9ff7fe700 1 -- 172.21.15.98:0/392170230 --> [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9f40b0380 con 0x7fd9f40ad810 2024-04-25T17:33:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.521+0000 7fd9ff7fe700 1 -- 172.21.15.98:0/392170230 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fda040d0a20 con 0x7fda08102ad0 2024-04-25T17:33:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.521+0000 7fda0d91c700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd9f40a3cd0 0x7fd9f40a6190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8006fd0 tx=0x7fd9f8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.521+0000 7fda0e91e700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] conn(0x7fd9f40ad810 0x7fd9f40afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.521+0000 7fda0e91e700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] conn(0x7fd9f40ad810 0x7fd9f40afcb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:56.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:56 smithi173 ceph-mon[32856]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T17:33:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.522+0000 7fd9ff7fe700 1 -- 172.21.15.98:0/392170230 <== osd.0 v2:172.21.15.98:6802/4172359021 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9f40b0380 con 0x7fd9f40ad810 2024-04-25T17:33:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.538+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 --> [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fda080635d0 con 0x7fd9f40ad810 2024-04-25T17:33:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fd9ff7fe700 1 -- 172.21.15.98:0/392170230 <== osd.0 v2:172.21.15.98:6802/4172359021 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fda080635d0 con 0x7fd9f40ad810 2024-04-25T17:33:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] conn(0x7fd9f40ad810 msgr2=0x7fd9f40afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda10381700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] conn(0x7fd9f40ad810 0x7fd9f40afcb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd9f40a3cd0 msgr2=0x7fd9f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda10381700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd9f40a3cd0 0x7fd9f40a6190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8006fd0 tx=0x7fd9f8008040 comp rx=0 tx=0).stop 2024-04-25T17:33:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda08102ad0 msgr2=0x7fda081991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda10381700 1 --2- 172.21.15.98:0/392170230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fda08102ad0 0x7fda081991b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fda0400c920 tx=0x7fda0400cce0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda0e91e700 1 -- 172.21.15.98:0/392170230 reap_dead start 2024-04-25T17:33:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 shutdown_connections 2024-04-25T17:33:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 >> 172.21.15.98:0/392170230 conn(0x7fda08079c30 msgr2=0x7fda08100a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 shutdown_connections 2024-04-25T17:33:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.539+0000 7fda10381700 1 -- 172.21.15.98:0/392170230 wait complete. 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.498+0000 7fc328d40700 1 -- 172.21.15.98:0/1960417853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32406a750 msgr2=0x7fc32406ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.498+0000 7fc328d40700 1 --2- 172.21.15.98:0/1960417853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32406a750 0x7fc32406ab30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00b0a0 tx=0x7fc31c00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.498+0000 7fc328d40700 1 -- 172.21.15.98:0/1960417853 shutdown_connections 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.498+0000 7fc328d40700 1 --2- 172.21.15.98:0/1960417853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc32410f4f0 0x7fc3241118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.498+0000 7fc328d40700 1 --2- 172.21.15.98:0/1960417853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32406a750 0x7fc32406ab30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.498+0000 7fc328d40700 1 --2- 172.21.15.98:0/1960417853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc32406b070 0x7fc32410efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.498+0000 7fc328d40700 1 -- 172.21.15.98:0/1960417853 >> 172.21.15.98:0/1960417853 conn(0x7fc3240788f0 msgr2=0x7fc324078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.498+0000 7fc328d40700 1 -- 172.21.15.98:0/1960417853 shutdown_connections 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.498+0000 7fc328d40700 1 -- 172.21.15.98:0/1960417853 wait complete. 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc328d40700 1 Processor -- start 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc328d40700 1 -- start start 2024-04-25T17:33:56.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc328d40700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc32406a750 0x7fc32419b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc328d40700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32406b070 0x7fc32419b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc328d40700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc32410f4f0 0x7fc32419fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc328d40700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc324114e80 con 0x7fc32406b070 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc328d40700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc324114d00 con 0x7fc32406a750 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc328d40700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc324115000 con 0x7fc32410f4f0 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc321d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32406b070 0x7fc32419b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc321d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32406b070 0x7fc32419b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55474/0 (socket says 172.21.15.98:55474) 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc321d9b700 1 -- 172.21.15.98:0/2700599720 learned_addr learned my addr 172.21.15.98:0/2700599720 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.499+0000 7fc322d9d700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc32410f4f0 0x7fc32419fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc321d9b700 1 -- 172.21.15.98:0/2700599720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc32410f4f0 msgr2=0x7fc32419fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc321d9b700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc32410f4f0 0x7fc32419fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc321d9b700 1 -- 172.21.15.98:0/2700599720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc32406a750 msgr2=0x7fc32419b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc321d9b700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc32406a750 0x7fc32419b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc321d9b700 1 -- 172.21.15.98:0/2700599720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3241a03a0 con 0x7fc32406b070 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc321d9b700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32406b070 0x7fc32419b8d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc31800b530 tx=0x7fc31800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc3137fe700 1 -- 172.21.15.98:0/2700599720 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc318015020 con 0x7fc32406b070 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3241a0690 con 0x7fc32406b070 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc3137fe700 1 -- 172.21.15.98:0/2700599720 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc318011ce0 con 0x7fc32406b070 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc3137fe700 1 -- 172.21.15.98:0/2700599720 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc318010d20 con 0x7fc32406b070 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.500+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3241a7f10 con 0x7fc32406b070 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.501+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc324063190 con 0x7fc32406b070 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.502+0000 7fc3137fe700 1 -- 172.21.15.98:0/2700599720 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc318018020 con 0x7fc32406b070 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.502+0000 7fc3137fe700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fc30c0a3cd0 0x7fc30c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.502+0000 7fc32259c700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fc30c0a3cd0 0x7fc30c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.503+0000 7fc3137fe700 1 -- 172.21.15.98:0/2700599720 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fc3180d0400 con 0x7fc32406b070 2024-04-25T17:33:56.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.503+0000 7fc3137fe700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] conn(0x7fc30c0ad810 0x7fc30c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.503+0000 7fc3137fe700 1 -- 172.21.15.98:0/2700599720 --> [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc318010ec0 con 0x7fc30c0ad810 2024-04-25T17:33:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.503+0000 7fc3137fe700 1 -- 172.21.15.98:0/2700599720 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fc3180d0850 con 0x7fc32406b070 2024-04-25T17:33:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.503+0000 7fc32259c700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fc30c0a3cd0 0x7fc30c0a6190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00b070 tx=0x7fc31c0060c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.503+0000 7fc322d9d700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] conn(0x7fc30c0ad810 0x7fc30c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.503+0000 7fc322d9d700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] conn(0x7fc30c0ad810 0x7fc30c0afcb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.504+0000 7fc3137fe700 1 -- 172.21.15.98:0/2700599720 <== osd.2 v2:172.21.15.203:6800/2010273295 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc318010ec0 con 0x7fc30c0ad810 2024-04-25T17:33:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.521+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 --> [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc324063610 con 0x7fc30c0ad810 2024-04-25T17:33:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc3137fe700 1 -- 172.21.15.98:0/2700599720 <== osd.2 v2:172.21.15.203:6800/2010273295 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc324063610 con 0x7fc30c0ad810 2024-04-25T17:33:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 >> [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] conn(0x7fc30c0ad810 msgr2=0x7fc30c0afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc328d40700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] conn(0x7fc30c0ad810 0x7fc30c0afcb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fc30c0a3cd0 msgr2=0x7fc30c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc328d40700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fc30c0a3cd0 0x7fc30c0a6190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00b070 tx=0x7fc31c0060c0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32406b070 msgr2=0x7fc32419b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc328d40700 1 --2- 172.21.15.98:0/2700599720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32406b070 0x7fc32419b8d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc31800b530 tx=0x7fc31800b8f0 comp rx=0 tx=0).stop 2024-04-25T17:33:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc322d9d700 1 -- 172.21.15.98:0/2700599720 reap_dead start 2024-04-25T17:33:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 shutdown_connections 2024-04-25T17:33:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 >> 172.21.15.98:0/2700599720 conn(0x7fc3240788f0 msgr2=0x7fc3241109c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 shutdown_connections 2024-04-25T17:33:56.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:55.522+0000 7fc328d40700 1 -- 172.21.15.98:0/2700599720 wait complete. 2024-04-25T17:33:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.733+0000 7f711d13d700 1 -- 172.21.15.98:0/723426781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7118102300 msgr2=0x7f711810e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.733+0000 7f711d13d700 1 --2- 172.21.15.98:0/723426781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7118102300 0x7f711810e7f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f711000b0a0 tx=0x7f711000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:33:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.733+0000 7f711d13d700 1 -- 172.21.15.98:0/723426781 shutdown_connections 2024-04-25T17:33:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.733+0000 7f711d13d700 1 --2- 172.21.15.98:0/723426781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7118107990 0x7f7118107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.733+0000 7f711d13d700 1 --2- 172.21.15.98:0/723426781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7118102300 0x7f711810e7f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.733+0000 7f711d13d700 1 --2- 172.21.15.98:0/723426781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7118101940 0x7f7118101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.733+0000 7f711d13d700 1 -- 172.21.15.98:0/723426781 >> 172.21.15.98:0/723426781 conn(0x7f71180788f0 msgr2=0x7f7118078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.733+0000 7f711d13d700 1 -- 172.21.15.98:0/723426781 shutdown_connections 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.734+0000 7f711d13d700 1 -- 172.21.15.98:0/723426781 wait complete. 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.734+0000 7f711d13d700 1 Processor -- start 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.734+0000 7f711d13d700 1 -- start start 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711d13d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7118101940 0x7f7118199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711d13d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7118102300 0x7f71181997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711d13d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7118107990 0x7f711819db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711d13d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7118111bd0 con 0x7f7118107990 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711d13d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7118111a50 con 0x7f7118102300 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711d13d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7118111d50 con 0x7f7118101940 2024-04-25T17:33:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7118107990 0x7f711819db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7118107990 0x7f711819db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55502/0 (socket says 172.21.15.98:55502) 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711759e700 1 -- 172.21.15.98:0/1090473305 learned_addr learned my addr 172.21.15.98:0/1090473305 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711659c700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7118102300 0x7f71181997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f7116d9d700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7118101940 0x7f7118199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711759e700 1 -- 172.21.15.98:0/1090473305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7118101940 msgr2=0x7f7118199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711759e700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7118101940 0x7f7118199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711759e700 1 -- 172.21.15.98:0/1090473305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7118102300 msgr2=0x7f71181997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711759e700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7118102300 0x7f71181997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711759e700 1 -- 172.21.15.98:0/1090473305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711819e280 con 0x7f7118107990 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f7116d9d700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7118101940 0x7f7118199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711659c700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7118102300 0x7f71181997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711759e700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7118107990 0x7f711819db40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f71100062a0 tx=0x7f7110004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f70fffff700 1 -- 172.21.15.98:0/1090473305 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110021070 con 0x7f7118107990 2024-04-25T17:33:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711819e510 con 0x7f7118107990 2024-04-25T17:33:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f70fffff700 1 -- 172.21.15.98:0/1090473305 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7110007d00 con 0x7f7118107990 2024-04-25T17:33:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f70fffff700 1 -- 172.21.15.98:0/1090473305 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711001d450 con 0x7f7118107990 2024-04-25T17:33:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.735+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71181a5df0 con 0x7f7118107990 2024-04-25T17:33:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.736+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7118063150 con 0x7f7118107990 2024-04-25T17:33:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.737+0000 7f70fffff700 1 -- 172.21.15.98:0/1090473305 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7110004000 con 0x7f7118107990 2024-04-25T17:33:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.738+0000 7f70fffff700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f71000a3cd0 0x7f71000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.738+0000 7f7116d9d700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f71000a3cd0 0x7f71000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.738+0000 7f70fffff700 1 -- 172.21.15.98:0/1090473305 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f711000f020 con 0x7f7118107990 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.738+0000 7f70fffff700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] conn(0x7f71000ad810 0x7f71000afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.738+0000 7f70fffff700 1 -- 172.21.15.98:0/1090473305 --> [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71100020b0 con 0x7f71000ad810 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.738+0000 7f70fffff700 1 -- 172.21.15.98:0/1090473305 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f71100d2310 con 0x7f7118107990 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.738+0000 7f711659c700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] conn(0x7f71000ad810 0x7f71000afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.738+0000 7f7116d9d700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f71000a3cd0 0x7f71000a6190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f71080099d0 tx=0x7f7108008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.739+0000 7f711659c700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] conn(0x7f71000ad810 0x7f71000afcb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.739+0000 7f70fffff700 1 -- 172.21.15.98:0/1090473305 <== osd.9 v2:172.21.15.98:6826/3104319246 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f71100020b0 con 0x7f71000ad810 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.755+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 --> [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7118063580 con 0x7f71000ad810 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f70fffff700 1 -- 172.21.15.98:0/1090473305 <== osd.9 v2:172.21.15.98:6826/3104319246 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7118063580 con 0x7f71000ad810 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] conn(0x7f71000ad810 msgr2=0x7f71000afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711d13d700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] conn(0x7f71000ad810 0x7f71000afcb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f71000a3cd0 msgr2=0x7f71000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711d13d700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f71000a3cd0 0x7f71000a6190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f71080099d0 tx=0x7f7108008040 comp rx=0 tx=0).stop 2024-04-25T17:33:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7118107990 msgr2=0x7f711819db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711d13d700 1 --2- 172.21.15.98:0/1090473305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7118107990 0x7f711819db40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f71100062a0 tx=0x7f7110004ed0 comp rx=0 tx=0).stop 2024-04-25T17:33:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711759e700 1 -- 172.21.15.98:0/1090473305 reap_dead start 2024-04-25T17:33:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 shutdown_connections 2024-04-25T17:33:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 >> 172.21.15.98:0/1090473305 conn(0x7f71180788f0 msgr2=0x7f711806ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 shutdown_connections 2024-04-25T17:33:57.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:56.756+0000 7f711d13d700 1 -- 172.21.15.98:0/1090473305 wait complete. 2024-04-25T17:33:58.196 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:33:58 smithi098 ceph-mon[29192]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T17:33:58.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:33:58 smithi203 ceph-mon[32834]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T17:33:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:33:58 smithi173 ceph-mon[32856]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T17:33:58.635 INFO:teuthology.orchestra.run.smithi098.stdout:77309411407 2024-04-25T17:33:58.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T17:33:59.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.056+0000 7fefa4a60700 1 -- 172.21.15.98:0/3365985200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa0103490 msgr2=0x7fefa010f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:59.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.056+0000 7fefa4a60700 1 --2- 172.21.15.98:0/3365985200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa0103490 0x7fefa010f970 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fef9800b0a0 tx=0x7fef9800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:33:59.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.056+0000 7fefa4a60700 1 -- 172.21.15.98:0/3365985200 shutdown_connections 2024-04-25T17:33:59.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.056+0000 7fefa4a60700 1 --2- 172.21.15.98:0/3365985200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa0103490 0x7fefa010f970 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.056+0000 7fefa4a60700 1 --2- 172.21.15.98:0/3365985200 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa0108aa0 0x7fefa0108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.056+0000 7fefa4a60700 1 --2- 172.21.15.98:0/3365985200 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefa0102ad0 0x7fefa0102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.056+0000 7fefa4a60700 1 -- 172.21.15.98:0/3365985200 >> 172.21.15.98:0/3365985200 conn(0x7fefa0079c30 msgr2=0x7fefa007a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.056+0000 7fefa4a60700 1 -- 172.21.15.98:0/3365985200 shutdown_connections 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.056+0000 7fefa4a60700 1 -- 172.21.15.98:0/3365985200 wait complete. 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fefa4a60700 1 Processor -- start 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fefa4a60700 1 -- start start 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fefa4a60700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa0102ad0 0x7fefa01990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fefa4a60700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa0103490 0x7fefa0199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fefa4a60700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefa0108aa0 0x7fefa019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fefa4a60700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fefa0112ce0 con 0x7fefa0102ad0 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fefa4a60700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fefa0112b60 con 0x7fefa0103490 2024-04-25T17:33:59.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fefa4a60700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefa0112e60 con 0x7fefa0108aa0 2024-04-25T17:33:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fef9e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa0102ad0 0x7fefa01990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fef9e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa0102ad0 0x7fefa01990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55528/0 (socket says 172.21.15.98:55528) 2024-04-25T17:33:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fef9e59c700 1 -- 172.21.15.98:0/3006798267 learned_addr learned my addr 172.21.15.98:0/3006798267 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:33:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fef9dd9b700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa0103490 0x7fefa0199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fef9ed9d700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefa0108aa0 0x7fefa019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fef9e59c700 1 -- 172.21.15.98:0/3006798267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefa0108aa0 msgr2=0x7fefa019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:59.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fef9e59c700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefa0108aa0 0x7fefa019d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fef9e59c700 1 -- 172.21.15.98:0/3006798267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa0103490 msgr2=0x7fefa0199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fef9e59c700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefa0103490 0x7fefa0199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.057+0000 7fef9e59c700 1 -- 172.21.15.98:0/3006798267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa019e0e0 con 0x7fefa0102ad0 2024-04-25T17:33:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.058+0000 7fef9e59c700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa0102ad0 0x7fefa01990d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fef9000c9b0 tx=0x7fef9000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.058+0000 7fef8f7fe700 1 -- 172.21.15.98:0/3006798267 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef90007b00 con 0x7fefa0102ad0 2024-04-25T17:33:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.058+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa019e3d0 con 0x7fefa0102ad0 2024-04-25T17:33:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.058+0000 7fef8f7fe700 1 -- 172.21.15.98:0/3006798267 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef90007ca0 con 0x7fefa0102ad0 2024-04-25T17:33:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.058+0000 7fef8f7fe700 1 -- 172.21.15.98:0/3006798267 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef90018660 con 0x7fefa0102ad0 2024-04-25T17:33:59.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.058+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa01a5c50 con 0x7fefa0102ad0 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.059+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fefa01013e0 con 0x7fefa0102ad0 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.060+0000 7fef8f7fe700 1 -- 172.21.15.98:0/3006798267 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fef9000f450 con 0x7fefa0102ad0 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.061+0000 7fef8f7fe700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fef880a3cd0 0x7fef880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.061+0000 7fef9dd9b700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fef880a3cd0 0x7fef880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.061+0000 7fef8f7fe700 1 -- 172.21.15.98:0/3006798267 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fef900d09b0 con 0x7fefa0102ad0 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.061+0000 7fef8f7fe700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] conn(0x7fef880ad810 0x7fef880afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.061+0000 7fef8f7fe700 1 -- 172.21.15.98:0/3006798267 --> [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef880b0380 con 0x7fef880ad810 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.061+0000 7fef8f7fe700 1 -- 172.21.15.98:0/3006798267 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fef900d0e00 con 0x7fefa0102ad0 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.061+0000 7fef9dd9b700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fef880a3cd0 0x7fef880a6190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fefa0102810 tx=0x7fef94009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.061+0000 7fef9ed9d700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] conn(0x7fef880ad810 0x7fef880afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.062+0000 7fef9ed9d700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] conn(0x7fef880ad810 0x7fef880afcb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:33:59.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.063+0000 7fef8f7fe700 1 -- 172.21.15.98:0/3006798267 <== osd.10 v2:172.21.15.173:6824/3555470519 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fef880b0380 con 0x7fef880ad810 2024-04-25T17:33:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.079+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 --> [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fefa01041e0 con 0x7fef880ad810 2024-04-25T17:33:59.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.079+0000 7fef8f7fe700 1 -- 172.21.15.98:0/3006798267 <== osd.10 v2:172.21.15.173:6824/3555470519 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fefa01041e0 con 0x7fef880ad810 2024-04-25T17:33:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 >> [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] conn(0x7fef880ad810 msgr2=0x7fef880afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fefa4a60700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] conn(0x7fef880ad810 0x7fef880afcb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:33:59.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fef880a3cd0 msgr2=0x7fef880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fefa4a60700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fef880a3cd0 0x7fef880a6190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fefa0102810 tx=0x7fef94009190 comp rx=0 tx=0).stop 2024-04-25T17:33:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa0102ad0 msgr2=0x7fefa01990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:33:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fefa4a60700 1 --2- 172.21.15.98:0/3006798267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefa0102ad0 0x7fefa01990d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fef9000c9b0 tx=0x7fef9000ccc0 comp rx=0 tx=0).stop 2024-04-25T17:33:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fef9ed9d700 1 -- 172.21.15.98:0/3006798267 reap_dead start 2024-04-25T17:33:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 shutdown_connections 2024-04-25T17:33:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 >> 172.21.15.98:0/3006798267 conn(0x7fefa0079c30 msgr2=0x7fefa0100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:33:59.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 shutdown_connections 2024-04-25T17:33:59.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:33:59.080+0000 7fefa4a60700 1 -- 172.21.15.98:0/3006798267 wait complete. 2024-04-25T17:33:59.543 INFO:teuthology.orchestra.run.smithi098.stdout:292057776172 2024-04-25T17:33:59.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T17:34:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:00 smithi098 ceph-mon[29192]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:00 smithi203 ceph-mon[32834]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:00 smithi173 ceph-mon[32856]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:01.006 INFO:teuthology.orchestra.run.smithi098.stdout:201863462971 2024-04-25T17:34:01.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T17:34:01.397 INFO:teuthology.orchestra.run.smithi098.stdout:347892351007 2024-04-25T17:34:01.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T17:34:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.709+0000 7f7828e09700 1 -- 172.21.15.98:0/2770937238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824108a30 msgr2=0x7f7824108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.710+0000 7f7828e09700 1 --2- 172.21.15.98:0/2770937238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824108a30 0x7f7824108e10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7814009990 tx=0x7f7814009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.710+0000 7f7828e09700 1 -- 172.21.15.98:0/2770937238 shutdown_connections 2024-04-25T17:34:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.710+0000 7f7828e09700 1 --2- 172.21.15.98:0/2770937238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f782406b0f0 0x7f782410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.710+0000 7f7828e09700 1 --2- 172.21.15.98:0/2770937238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7824108a30 0x7f7824108e10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.710+0000 7f7828e09700 1 --2- 172.21.15.98:0/2770937238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f782406a750 0x7f782406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.710+0000 7f7828e09700 1 -- 172.21.15.98:0/2770937238 >> 172.21.15.98:0/2770937238 conn(0x7f78240788f0 msgr2=0x7f7824078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:01.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.710+0000 7f7828e09700 1 -- 172.21.15.98:0/2770937238 shutdown_connections 2024-04-25T17:34:01.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.710+0000 7f7828e09700 1 -- 172.21.15.98:0/2770937238 wait complete. 2024-04-25T17:34:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.710+0000 7f7828e09700 1 Processor -- start 2024-04-25T17:34:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.710+0000 7f7828e09700 1 -- start start 2024-04-25T17:34:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f7828e09700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f782406a750 0x7f7824101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f7828e09700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f782406b0f0 0x7f7824101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f7828e09700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7824108a30 0x7f7824102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f7828e09700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7824112bb0 con 0x7f782406a750 2024-04-25T17:34:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f7828e09700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7824112a30 con 0x7f782406b0f0 2024-04-25T17:34:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f7828e09700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7824112d30 con 0x7f7824108a30 2024-04-25T17:34:01.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f782259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f782406a750 0x7f7824101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f782259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f782406a750 0x7f7824101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55548/0 (socket says 172.21.15.98:55548) 2024-04-25T17:34:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f782259c700 1 -- 172.21.15.98:0/2602681920 learned_addr learned my addr 172.21.15.98:0/2602681920 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f7821d9b700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f782406b0f0 0x7f7824101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f7822d9d700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7824108a30 0x7f7824102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f782259c700 1 -- 172.21.15.98:0/2602681920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7824108a30 msgr2=0x7f7824102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f782259c700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7824108a30 0x7f7824102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f782259c700 1 -- 172.21.15.98:0/2602681920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f782406b0f0 msgr2=0x7f7824101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f782259c700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f782406b0f0 0x7f7824101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f782259c700 1 -- 172.21.15.98:0/2602681920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78241a76b0 con 0x7f782406a750 2024-04-25T17:34:01.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f7822d9d700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7824108a30 0x7f7824102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:34:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f782259c700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f782406a750 0x7f7824101a10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7814000c00 tx=0x7f78140047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f78137fe700 1 -- 172.21.15.98:0/2602681920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814027070 con 0x7f782406a750 2024-04-25T17:34:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f78137fe700 1 -- 172.21.15.98:0/2602681920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7814003f20 con 0x7f782406a750 2024-04-25T17:34:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78241a7940 con 0x7f782406a750 2024-04-25T17:34:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.711+0000 7f78137fe700 1 -- 172.21.15.98:0/2602681920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814022cf0 con 0x7f782406a750 2024-04-25T17:34:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.712+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78241a7d90 con 0x7f782406a750 2024-04-25T17:34:01.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.712+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f78240630d0 con 0x7f782406a750 2024-04-25T17:34:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.714+0000 7f78137fe700 1 -- 172.21.15.98:0/2602681920 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f781402a070 con 0x7f782406a750 2024-04-25T17:34:01.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.714+0000 7f78137fe700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f780c0a3cd0 0x7f780c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.714+0000 7f7821d9b700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f780c0a3cd0 0x7f780c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.714+0000 7f78137fe700 1 -- 172.21.15.98:0/2602681920 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f78140d1990 con 0x7f782406a750 2024-04-25T17:34:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.714+0000 7f78137fe700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] conn(0x7f780c0ad810 0x7f780c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.714+0000 7f78137fe700 1 -- 172.21.15.98:0/2602681920 --> [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78140095d0 con 0x7f780c0ad810 2024-04-25T17:34:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.714+0000 7f78137fe700 1 -- 172.21.15.98:0/2602681920 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f78140d1de0 con 0x7f782406a750 2024-04-25T17:34:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.715+0000 7f7821d9b700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f780c0a3cd0 0x7f780c0a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7824079030 tx=0x7f781800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.715+0000 7f7822d9d700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] conn(0x7f780c0ad810 0x7f780c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.715+0000 7f7822d9d700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] conn(0x7f780c0ad810 0x7f780c0afcb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:01.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.716+0000 7f78137fe700 1 -- 172.21.15.98:0/2602681920 <== osd.5 v2:172.21.15.203:6808/2904529843 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78140095d0 con 0x7f780c0ad810 2024-04-25T17:34:01.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.732+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 --> [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7824063550 con 0x7f780c0ad810 2024-04-25T17:34:01.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.733+0000 7f78137fe700 1 -- 172.21.15.98:0/2602681920 <== osd.5 v2:172.21.15.203:6808/2904529843 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7824063550 con 0x7f780c0ad810 2024-04-25T17:34:01.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.733+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 >> [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] conn(0x7f780c0ad810 msgr2=0x7f780c0afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:01.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.733+0000 7f7828e09700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] conn(0x7f780c0ad810 0x7f780c0afcb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:01.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.733+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f780c0a3cd0 msgr2=0x7f780c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:01.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.733+0000 7f7828e09700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f780c0a3cd0 0x7f780c0a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7824079030 tx=0x7f781800b410 comp rx=0 tx=0).stop 2024-04-25T17:34:01.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.733+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f782406a750 msgr2=0x7f7824101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:01.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.733+0000 7f7828e09700 1 --2- 172.21.15.98:0/2602681920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f782406a750 0x7f7824101a10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7814000c00 tx=0x7f78140047b0 comp rx=0 tx=0).stop 2024-04-25T17:34:01.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.733+0000 7f7822d9d700 1 -- 172.21.15.98:0/2602681920 reap_dead start 2024-04-25T17:34:01.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.734+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 shutdown_connections 2024-04-25T17:34:01.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.734+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 >> 172.21.15.98:0/2602681920 conn(0x7f78240788f0 msgr2=0x7f78240ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:01.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.734+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 shutdown_connections 2024-04-25T17:34:01.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:01.734+0000 7f7828e09700 1 -- 172.21.15.98:0/2602681920 wait complete. 2024-04-25T17:34:02.181 INFO:teuthology.orchestra.run.smithi098.stdout:313532612647 2024-04-25T17:34:02.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T17:34:02.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:02 smithi098 ceph-mon[29192]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:34:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:02 smithi203 ceph-mon[32834]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:34:02.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:02 smithi173 ceph-mon[32856]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:34:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.615+0000 7f4934ff6700 1 -- 172.21.15.98:0/3634017805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4930107980 msgr2=0x7f4930107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.615+0000 7f4934ff6700 1 --2- 172.21.15.98:0/3634017805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4930107980 0x7f4930107d60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4920009990 tx=0x7f4920009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.615+0000 7f4934ff6700 1 -- 172.21.15.98:0/3634017805 shutdown_connections 2024-04-25T17:34:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.615+0000 7f4934ff6700 1 --2- 172.21.15.98:0/3634017805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49301022f0 0x7f493010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.615+0000 7f4934ff6700 1 --2- 172.21.15.98:0/3634017805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4930107980 0x7f4930107d60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.615+0000 7f4934ff6700 1 --2- 172.21.15.98:0/3634017805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930101930 0x7f4930101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.615+0000 7f4934ff6700 1 -- 172.21.15.98:0/3634017805 >> 172.21.15.98:0/3634017805 conn(0x7f49300788b0 msgr2=0x7f4930078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.615+0000 7f4934ff6700 1 -- 172.21.15.98:0/3634017805 shutdown_connections 2024-04-25T17:34:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.615+0000 7f4934ff6700 1 -- 172.21.15.98:0/3634017805 wait complete. 2024-04-25T17:34:02.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f4934ff6700 1 Processor -- start 2024-04-25T17:34:02.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f4934ff6700 1 -- start start 2024-04-25T17:34:02.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f4934ff6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4930101930 0x7f493019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:02.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f4934ff6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49301022f0 0x7f493019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:02.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f4934ff6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930107980 0x7f49301a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:02.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f4934ff6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4930111ad0 con 0x7f49301022f0 2024-04-25T17:34:02.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f4934ff6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4930111950 con 0x7f4930101930 2024-04-25T17:34:02.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f4934ff6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4930111c50 con 0x7f4930107980 2024-04-25T17:34:02.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f492dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49301022f0 0x7f493019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f492dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49301022f0 0x7f493019a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55562/0 (socket says 172.21.15.98:55562) 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f492dd9b700 1 -- 172.21.15.98:0/2357623278 learned_addr learned my addr 172.21.15.98:0/2357623278 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f492e59c700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4930101930 0x7f493019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.616+0000 7f492ed9d700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930107980 0x7f49301a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f492dd9b700 1 -- 172.21.15.98:0/2357623278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930107980 msgr2=0x7f49301a5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f492dd9b700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4930107980 0x7f49301a5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f492dd9b700 1 -- 172.21.15.98:0/2357623278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4930101930 msgr2=0x7f493019a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f492dd9b700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4930101930 0x7f493019a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f492dd9b700 1 -- 172.21.15.98:0/2357623278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49301a5660 con 0x7f49301022f0 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f492dd9b700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49301022f0 0x7f493019a5e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f491800c9b0 tx=0x7f491800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f49277fe700 1 -- 172.21.15.98:0/2357623278 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4918007b00 con 0x7f49301022f0 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49301a5890 con 0x7f49301022f0 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f49277fe700 1 -- 172.21.15.98:0/2357623278 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f491800ce50 con 0x7f49301022f0 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f49277fe700 1 -- 172.21.15.98:0/2357623278 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4918018660 con 0x7f49301022f0 2024-04-25T17:34:02.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.617+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49301a5dd0 con 0x7f49301022f0 2024-04-25T17:34:02.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.618+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f49300630d0 con 0x7f49301022f0 2024-04-25T17:34:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.619+0000 7f49277fe700 1 -- 172.21.15.98:0/2357623278 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f491800f450 con 0x7f49301022f0 2024-04-25T17:34:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.620+0000 7f49277fe700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f491c0a3cd0 0x7f491c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.620+0000 7f492e59c700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f491c0a3cd0 0x7f491c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.620+0000 7f49277fe700 1 -- 172.21.15.98:0/2357623278 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f49180d08c0 con 0x7f49301022f0 2024-04-25T17:34:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.620+0000 7f49277fe700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] conn(0x7f491c0ad810 0x7f491c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.620+0000 7f49277fe700 1 -- 172.21.15.98:0/2357623278 --> [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f491c0b0380 con 0x7f491c0ad810 2024-04-25T17:34:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.620+0000 7f49277fe700 1 -- 172.21.15.98:0/2357623278 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f49180d0d10 con 0x7f49301022f0 2024-04-25T17:34:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.620+0000 7f492e59c700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f491c0a3cd0 0x7f491c0a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f49200098e0 tx=0x7f4920005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.620+0000 7f492ed9d700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] conn(0x7f491c0ad810 0x7f491c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:02.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.620+0000 7f492ed9d700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] conn(0x7f491c0ad810 0x7f491c0afcb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:02.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.621+0000 7f49277fe700 1 -- 172.21.15.98:0/2357623278 <== osd.3 v2:172.21.15.98:6810/2541442286 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f491c0b0380 con 0x7f491c0ad810 2024-04-25T17:34:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.638+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 --> [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f49300634d0 con 0x7f491c0ad810 2024-04-25T17:34:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.638+0000 7f49277fe700 1 -- 172.21.15.98:0/2357623278 <== osd.3 v2:172.21.15.98:6810/2541442286 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f49300634d0 con 0x7f491c0ad810 2024-04-25T17:34:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] conn(0x7f491c0ad810 msgr2=0x7f491c0afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f4934ff6700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] conn(0x7f491c0ad810 0x7f491c0afcb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f491c0a3cd0 msgr2=0x7f491c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f4934ff6700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f491c0a3cd0 0x7f491c0a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f49200098e0 tx=0x7f4920005960 comp rx=0 tx=0).stop 2024-04-25T17:34:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49301022f0 msgr2=0x7f493019a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f4934ff6700 1 --2- 172.21.15.98:0/2357623278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49301022f0 0x7f493019a5e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f491800c9b0 tx=0x7f491800ccc0 comp rx=0 tx=0).stop 2024-04-25T17:34:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f492ed9d700 1 -- 172.21.15.98:0/2357623278 reap_dead start 2024-04-25T17:34:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 shutdown_connections 2024-04-25T17:34:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 >> 172.21.15.98:0/2357623278 conn(0x7f49300788b0 msgr2=0x7f4930100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 shutdown_connections 2024-04-25T17:34:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:02.639+0000 7f4934ff6700 1 -- 172.21.15.98:0/2357623278 wait complete. 2024-04-25T17:34:03.329 INFO:teuthology.orchestra.run.smithi098.stdout:167503724609 2024-04-25T17:34:03.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T17:34:04.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:04 smithi098 ceph-mon[29192]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:34:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:04 smithi203 ceph-mon[32834]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:34:04.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:04 smithi173 ceph-mon[32856]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:34:04.657 INFO:teuthology.orchestra.run.smithi098.stdout:111669149772 2024-04-25T17:34:04.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T17:34:05.516 INFO:teuthology.orchestra.run.smithi098.stdout:34359738457 2024-04-25T17:34:05.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T17:34:06.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:05 smithi098 ceph-mon[29192]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:06.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 -- 172.21.15.98:0/2731832779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc06b6b0 msgr2=0x7f1ffc06bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:06.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 --2- 172.21.15.98:0/2731832779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc06b6b0 0x7f1ffc06bb30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009990 tx=0x7f1ff0009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:06.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 -- 172.21.15.98:0/2731832779 shutdown_connections 2024-04-25T17:34:06.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 --2- 172.21.15.98:0/2731832779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ffc107960 0x7f1ffc06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 --2- 172.21.15.98:0/2731832779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc105470 0x7f1ffc105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 --2- 172.21.15.98:0/2731832779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc06b6b0 0x7f1ffc06bb30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 -- 172.21.15.98:0/2731832779 >> 172.21.15.98:0/2731832779 conn(0x7f1ffc0788f0 msgr2=0x7f1ffc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 -- 172.21.15.98:0/2731832779 shutdown_connections 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 -- 172.21.15.98:0/2731832779 wait complete. 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 Processor -- start 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.012+0000 7f2002269700 1 -- start start 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f2002269700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc06b6b0 0x7f1ffc19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f2002269700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc105470 0x7f1ffc19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f2002269700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ffc107960 0x7f1ffc1a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f2002269700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc1191c0 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f2002269700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc119040 con 0x7f1ffc105470 2024-04-25T17:34:06.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f2002269700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc119340 con 0x7f1ffc107960 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f1ffbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc06b6b0 0x7f1ffc19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f1ffbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc06b6b0 0x7f1ffc19f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50812/0 (socket says 172.21.15.98:50812) 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f1ffbfff700 1 -- 172.21.15.98:0/3442834581 learned_addr learned my addr 172.21.15.98:0/3442834581 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f1ffb7fe700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc105470 0x7f1ffc19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f2000806700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ffc107960 0x7f1ffc1a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f1ffbfff700 1 -- 172.21.15.98:0/3442834581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ffc107960 msgr2=0x7f1ffc1a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f1ffbfff700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ffc107960 0x7f1ffc1a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f1ffbfff700 1 -- 172.21.15.98:0/3442834581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc105470 msgr2=0x7f1ffc19fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f1ffbfff700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ffc105470 0x7f1ffc19fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f1ffbfff700 1 -- 172.21.15.98:0/3442834581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc1a46a0 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.013+0000 7f1ffbfff700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc06b6b0 0x7f1ffc19f690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c920 tx=0x7f1fec00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.014+0000 7f1ff97fa700 1 -- 172.21.15.98:0/3442834581 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec004a30 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.014+0000 7f1ff97fa700 1 -- 172.21.15.98:0/3442834581 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fec0054d0 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.014+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc1a4990 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.014+0000 7f1ff97fa700 1 -- 172.21.15.98:0/3442834581 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec007c70 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.014+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ffc1ac210 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.014+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1ffc104e70 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.016+0000 7f1ff97fa700 1 -- 172.21.15.98:0/3442834581 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1fec004c60 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.016+0000 7f1ff97fa700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1fe40a3cd0 0x7f1fe40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:06.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.016+0000 7f1ffb7fe700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1fe40a3cd0 0x7f1fe40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:06.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.016+0000 7f1ff97fa700 1 -- 172.21.15.98:0/3442834581 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f1fec0d0540 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.016+0000 7f1ff97fa700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] conn(0x7f1fe40ad810 0x7f1fe40afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.016+0000 7f1ff97fa700 1 -- 172.21.15.98:0/3442834581 --> [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fec012ec0 con 0x7f1fe40ad810 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.016+0000 7f1ff97fa700 1 -- 172.21.15.98:0/3442834581 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f1fec0d0990 con 0x7f1ffc06b6b0 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.017+0000 7f1ffb7fe700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1fe40a3cd0 0x7f1fe40a6190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000b5c0 tx=0x7f1ff0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.017+0000 7f2000806700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] conn(0x7f1fe40ad810 0x7f1fe40afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.017+0000 7f2000806700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] conn(0x7f1fe40ad810 0x7f1fe40afcb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.018+0000 7f1ff97fa700 1 -- 172.21.15.98:0/3442834581 <== osd.7 v2:172.21.15.173:6816/3192706308 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1fec012ec0 con 0x7f1fe40ad810 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.034+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 --> [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1ffc101f50 con 0x7f1fe40ad810 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f1ff97fa700 1 -- 172.21.15.98:0/3442834581 <== osd.7 v2:172.21.15.173:6816/3192706308 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1ffc101f50 con 0x7f1fe40ad810 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 >> [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] conn(0x7f1fe40ad810 msgr2=0x7f1fe40afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2002269700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] conn(0x7f1fe40ad810 0x7f1fe40afcb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1fe40a3cd0 msgr2=0x7f1fe40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2002269700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f1fe40a3cd0 0x7f1fe40a6190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000b5c0 tx=0x7f1ff0005960 comp rx=0 tx=0).stop 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc06b6b0 msgr2=0x7f1ffc19f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2002269700 1 --2- 172.21.15.98:0/3442834581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ffc06b6b0 0x7f1ffc19f690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c920 tx=0x7f1fec00cc30 comp rx=0 tx=0).stop 2024-04-25T17:34:06.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2000806700 1 -- 172.21.15.98:0/3442834581 reap_dead start 2024-04-25T17:34:06.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 shutdown_connections 2024-04-25T17:34:06.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 >> 172.21.15.98:0/3442834581 conn(0x7f1ffc0788f0 msgr2=0x7f1ffc104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:06.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 shutdown_connections 2024-04-25T17:34:06.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:06.035+0000 7f2002269700 1 -- 172.21.15.98:0/3442834581 wait complete. 2024-04-25T17:34:06.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:05 smithi203 ceph-mon[32834]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:06.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:05 smithi173 ceph-mon[32856]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:06.735 INFO:teuthology.orchestra.run.smithi098.stdout:223338299448 2024-04-25T17:34:06.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T17:34:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.375+0000 7f15ec053700 1 -- 172.21.15.98:0/1726522304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e4107e20 msgr2=0x7f15e406b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.375+0000 7f15ec053700 1 --2- 172.21.15.98:0/1726522304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e4107e20 0x7f15e406b6d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009990 tx=0x7f15d4009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.375+0000 7f15ec053700 1 -- 172.21.15.98:0/1726522304 shutdown_connections 2024-04-25T17:34:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.375+0000 7f15ec053700 1 --2- 172.21.15.98:0/1726522304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e41079b0 0x7f15e406b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.375+0000 7f15ec053700 1 --2- 172.21.15.98:0/1726522304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15e41054c0 0x7f15e4105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.375+0000 7f15ec053700 1 --2- 172.21.15.98:0/1726522304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e4107e20 0x7f15e406b6d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.375+0000 7f15ec053700 1 -- 172.21.15.98:0/1726522304 >> 172.21.15.98:0/1726522304 conn(0x7f15e4078830 msgr2=0x7f15e4078c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.375+0000 7f15ec053700 1 -- 172.21.15.98:0/1726522304 shutdown_connections 2024-04-25T17:34:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.375+0000 7f15ec053700 1 -- 172.21.15.98:0/1726522304 wait complete. 2024-04-25T17:34:07.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.375+0000 7f15ec053700 1 Processor -- start 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15ec053700 1 -- start start 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15ec053700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e41054c0 0x7f15e419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15ec053700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15e41079b0 0x7f15e419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15ec053700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e4107e20 0x7f15e41a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15ec053700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15e4119240 con 0x7f15e41054c0 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15ec053700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41190c0 con 0x7f15e41079b0 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15ec053700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41193c0 con 0x7f15e4107e20 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15e9def700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e41054c0 0x7f15e419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15e9def700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e41054c0 0x7f15e419f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50822/0 (socket says 172.21.15.98:50822) 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15e9def700 1 -- 172.21.15.98:0/600480371 learned_addr learned my addr 172.21.15.98:0/600480371 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15e95ee700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15e41079b0 0x7f15e419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15e9def700 1 -- 172.21.15.98:0/600480371 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e4107e20 msgr2=0x7f15e41a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15e9def700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e4107e20 0x7f15e41a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:07.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15e9def700 1 -- 172.21.15.98:0/600480371 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15e41079b0 msgr2=0x7f15e419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15e9def700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15e41079b0 0x7f15e419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15e9def700 1 -- 172.21.15.98:0/600480371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e41a4730 con 0x7f15e41054c0 2024-04-25T17:34:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15e9def700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e41054c0 0x7f15e419f770 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f15e000c920 tx=0x7f15e000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15daffd700 1 -- 172.21.15.98:0/600480371 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e0007b00 con 0x7f15e41054c0 2024-04-25T17:34:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e41a4a20 con 0x7f15e41054c0 2024-04-25T17:34:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15daffd700 1 -- 172.21.15.98:0/600480371 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15e0004ce0 con 0x7f15e41054c0 2024-04-25T17:34:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e41ac2a0 con 0x7f15e41054c0 2024-04-25T17:34:07.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.376+0000 7f15daffd700 1 -- 172.21.15.98:0/600480371 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e0005650 con 0x7f15e41054c0 2024-04-25T17:34:07.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.377+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f15e4103440 con 0x7f15e41054c0 2024-04-25T17:34:07.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.379+0000 7f15daffd700 1 -- 172.21.15.98:0/600480371 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f15e0007ca0 con 0x7f15e41054c0 2024-04-25T17:34:07.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.379+0000 7f15daffd700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15d00a3cd0 0x7f15d00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:07.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.379+0000 7f15e95ee700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15d00a3cd0 0x7f15d00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:07.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.379+0000 7f15daffd700 1 -- 172.21.15.98:0/600480371 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f15e00d0780 con 0x7f15e41054c0 2024-04-25T17:34:07.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.379+0000 7f15daffd700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] conn(0x7f15d00ad810 0x7f15d00afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:07.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.379+0000 7f15daffd700 1 -- 172.21.15.98:0/600480371 --> [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15e0019e50 con 0x7f15d00ad810 2024-04-25T17:34:07.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.379+0000 7f15daffd700 1 -- 172.21.15.98:0/600480371 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f15e00d0bd0 con 0x7f15e41054c0 2024-04-25T17:34:07.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.380+0000 7f15e95ee700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15d00a3cd0 0x7f15d00a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f15d400b5c0 tx=0x7f15d401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:07.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.380+0000 7f15ea5f0700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] conn(0x7f15d00ad810 0x7f15d00afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:07.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.380+0000 7f15ea5f0700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] conn(0x7f15d00ad810 0x7f15d00afcb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:07.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.381+0000 7f15daffd700 1 -- 172.21.15.98:0/600480371 <== osd.4 v2:172.21.15.173:6808/2870689522 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15e0019e50 con 0x7f15d00ad810 2024-04-25T17:34:07.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.398+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 --> [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f15e4101fa0 con 0x7f15d00ad810 2024-04-25T17:34:07.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15daffd700 1 -- 172.21.15.98:0/600480371 <== osd.4 v2:172.21.15.173:6808/2870689522 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f15e4101fa0 con 0x7f15d00ad810 2024-04-25T17:34:07.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 >> [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] conn(0x7f15d00ad810 msgr2=0x7f15d00afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:07.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ec053700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] conn(0x7f15d00ad810 0x7f15d00afcb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:07.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15d00a3cd0 msgr2=0x7f15d00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:07.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ec053700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15d00a3cd0 0x7f15d00a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f15d400b5c0 tx=0x7f15d401a040 comp rx=0 tx=0).stop 2024-04-25T17:34:07.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e41054c0 msgr2=0x7f15e419f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:07.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ec053700 1 --2- 172.21.15.98:0/600480371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e41054c0 0x7f15e419f770 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f15e000c920 tx=0x7f15e000cce0 comp rx=0 tx=0).stop 2024-04-25T17:34:07.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ea5f0700 1 -- 172.21.15.98:0/600480371 reap_dead start 2024-04-25T17:34:07.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 shutdown_connections 2024-04-25T17:34:07.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 >> 172.21.15.98:0/600480371 conn(0x7f15e4078830 msgr2=0x7f15e41029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:07.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 shutdown_connections 2024-04-25T17:34:07.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:07.399+0000 7f15ec053700 1 -- 172.21.15.98:0/600480371 wait complete. 2024-04-25T17:34:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.177+0000 7f67a566b700 1 -- 172.21.15.98:0/2304697853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a01022f0 msgr2=0x7f67a010e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.177+0000 7f67a566b700 1 --2- 172.21.15.98:0/2304697853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a01022f0 0x7f67a010e7e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f679800b0a0 tx=0x7f679800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.177+0000 7f67a566b700 1 -- 172.21.15.98:0/2304697853 shutdown_connections 2024-04-25T17:34:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.177+0000 7f67a566b700 1 --2- 172.21.15.98:0/2304697853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a01022f0 0x7f67a010e7e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.177+0000 7f67a566b700 1 --2- 172.21.15.98:0/2304697853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67a0107980 0x7f67a0107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.177+0000 7f67a566b700 1 --2- 172.21.15.98:0/2304697853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a0101930 0x7f67a0101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.177+0000 7f67a566b700 1 -- 172.21.15.98:0/2304697853 >> 172.21.15.98:0/2304697853 conn(0x7f67a00788b0 msgr2=0x7f67a0078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.177+0000 7f67a566b700 1 -- 172.21.15.98:0/2304697853 shutdown_connections 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.177+0000 7f67a566b700 1 -- 172.21.15.98:0/2304697853 wait complete. 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f67a566b700 1 Processor -- start 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f67a566b700 1 -- start start 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f67a566b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a0101930 0x7f67a01034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f67a566b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67a01022f0 0x7f67a0103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f67a566b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0107980 0x7f67a0103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f67a566b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f67a0111b40 con 0x7f67a0107980 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f67a566b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f67a01119c0 con 0x7f67a01022f0 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f67a566b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f67a0111cc0 con 0x7f67a0101930 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0107980 0x7f67a0103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0107980 0x7f67a0103f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50846/0 (socket says 172.21.15.98:50846) 2024-04-25T17:34:08.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679f7fe700 1 -- 172.21.15.98:0/477199020 learned_addr learned my addr 172.21.15.98:0/477199020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679e7fc700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67a01022f0 0x7f67a0103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679effd700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a0101930 0x7f67a01034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679f7fe700 1 -- 172.21.15.98:0/477199020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a0101930 msgr2=0x7f67a01034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679f7fe700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a0101930 0x7f67a01034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679f7fe700 1 -- 172.21.15.98:0/477199020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67a01022f0 msgr2=0x7f67a0103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679f7fe700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67a01022f0 0x7f67a0103a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679f7fe700 1 -- 172.21.15.98:0/477199020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a01a7690 con 0x7f67a0107980 2024-04-25T17:34:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679effd700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a0101930 0x7f67a01034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:34:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.178+0000 7f679e7fc700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67a01022f0 0x7f67a0103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:34:08.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.179+0000 7f679f7fe700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0107980 0x7f67a0103f60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f67980062a0 tx=0x7f6798004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.179+0000 7f6787fff700 1 -- 172.21.15.98:0/477199020 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798021070 con 0x7f67a0107980 2024-04-25T17:34:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.179+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a01a7920 con 0x7f67a0107980 2024-04-25T17:34:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.179+0000 7f6787fff700 1 -- 172.21.15.98:0/477199020 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6798007d00 con 0x7f67a0107980 2024-04-25T17:34:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.179+0000 7f6787fff700 1 -- 172.21.15.98:0/477199020 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679801d450 con 0x7f67a0107980 2024-04-25T17:34:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.179+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a01a7dd0 con 0x7f67a0107980 2024-04-25T17:34:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.180+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f67a006b200 con 0x7f67a0107980 2024-04-25T17:34:08.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.181+0000 7f6787fff700 1 -- 172.21.15.98:0/477199020 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6798004000 con 0x7f67a0107980 2024-04-25T17:34:08.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.181+0000 7f6787fff700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f67880a3cd0 0x7f67880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:08.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.182+0000 7f679effd700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f67880a3cd0 0x7f67880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:08.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.182+0000 7f6787fff700 1 -- 172.21.15.98:0/477199020 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f67980d1bd0 con 0x7f67a0107980 2024-04-25T17:34:08.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.182+0000 7f6787fff700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] conn(0x7f67880ad810 0x7f67880afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:08.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.182+0000 7f6787fff700 1 -- 172.21.15.98:0/477199020 --> [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6798003de0 con 0x7f67880ad810 2024-04-25T17:34:08.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.182+0000 7f6787fff700 1 -- 172.21.15.98:0/477199020 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f67980d2020 con 0x7f67a0107980 2024-04-25T17:34:08.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.182+0000 7f679effd700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f67880a3cd0 0x7f67880a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6790005fd0 tx=0x7f6790005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:08.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.182+0000 7f679e7fc700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] conn(0x7f67880ad810 0x7f67880afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:08.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.182+0000 7f679e7fc700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] conn(0x7f67880ad810 0x7f67880afcb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:08.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.183+0000 7f6787fff700 1 -- 172.21.15.98:0/477199020 <== osd.1 v2:172.21.15.173:6800/3686689866 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6798003de0 con 0x7f67880ad810 2024-04-25T17:34:08.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.200+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 --> [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f67a0103050 con 0x7f67880ad810 2024-04-25T17:34:08.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.200+0000 7f6787fff700 1 -- 172.21.15.98:0/477199020 <== osd.1 v2:172.21.15.173:6800/3686689866 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f67a0103050 con 0x7f67880ad810 2024-04-25T17:34:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 >> [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] conn(0x7f67880ad810 msgr2=0x7f67880afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f67a566b700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] conn(0x7f67880ad810 0x7f67880afcb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f67880a3cd0 msgr2=0x7f67880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f67a566b700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f67880a3cd0 0x7f67880a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6790005fd0 tx=0x7f6790005e20 comp rx=0 tx=0).stop 2024-04-25T17:34:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0107980 msgr2=0x7f67a0103f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f67a566b700 1 --2- 172.21.15.98:0/477199020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67a0107980 0x7f67a0103f60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f67980062a0 tx=0x7f6798004ed0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f679f7fe700 1 -- 172.21.15.98:0/477199020 reap_dead start 2024-04-25T17:34:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 shutdown_connections 2024-04-25T17:34:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 >> 172.21.15.98:0/477199020 conn(0x7f67a00788b0 msgr2=0x7f67a01057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 shutdown_connections 2024-04-25T17:34:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.201+0000 7f67a566b700 1 -- 172.21.15.98:0/477199020 wait complete. 2024-04-25T17:34:08.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:08 smithi098 ceph-mon[29192]: pgmap v350: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:34:08.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:08 smithi203 ceph-mon[32834]: pgmap v350: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:34:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:08 smithi173 ceph-mon[32856]: pgmap v350: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T17:34:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.696+0000 7f22bfdd7700 1 -- 172.21.15.98:0/2415206149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22b8101970 msgr2=0x7f22b8101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.696+0000 7f22bfdd7700 1 --2- 172.21.15.98:0/2415206149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22b8101970 0x7f22b8101df0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f22a80099c0 tx=0x7f22a8009cd0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.696+0000 7f22bfdd7700 1 -- 172.21.15.98:0/2415206149 shutdown_connections 2024-04-25T17:34:08.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.696+0000 7f22bfdd7700 1 --2- 172.21.15.98:0/2415206149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b8102330 0x7f22b810e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.696+0000 7f22bfdd7700 1 --2- 172.21.15.98:0/2415206149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b81079c0 0x7f22b8107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.696+0000 7f22bfdd7700 1 --2- 172.21.15.98:0/2415206149 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22b8101970 0x7f22b8101df0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.696+0000 7f22bfdd7700 1 -- 172.21.15.98:0/2415206149 >> 172.21.15.98:0/2415206149 conn(0x7f22b8078980 msgr2=0x7f22b8078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.697+0000 7f22bfdd7700 1 -- 172.21.15.98:0/2415206149 shutdown_connections 2024-04-25T17:34:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.697+0000 7f22bfdd7700 1 -- 172.21.15.98:0/2415206149 wait complete. 2024-04-25T17:34:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.697+0000 7f22bfdd7700 1 Processor -- start 2024-04-25T17:34:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.697+0000 7f22bfdd7700 1 -- start start 2024-04-25T17:34:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22bfdd7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b8101970 0x7f22b8199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22bfdd7700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b8102330 0x7f22b819a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22bfdd7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22b81079c0 0x7f22b81a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22bfdd7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f22b8111b30 con 0x7f22b81079c0 2024-04-25T17:34:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22bfdd7700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f22b81119b0 con 0x7f22b8101970 2024-04-25T17:34:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22bfdd7700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f22b8111cb0 con 0x7f22b8102330 2024-04-25T17:34:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22be374700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22b81079c0 0x7f22b81a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22be374700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22b81079c0 0x7f22b81a4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50860/0 (socket says 172.21.15.98:50860) 2024-04-25T17:34:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22be374700 1 -- 172.21.15.98:0/1871230892 learned_addr learned my addr 172.21.15.98:0/1871230892 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22bd372700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b8102330 0x7f22b819a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22bdb73700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b8101970 0x7f22b8199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22be374700 1 -- 172.21.15.98:0/1871230892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b8102330 msgr2=0x7f22b819a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22be374700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22b8102330 0x7f22b819a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22be374700 1 -- 172.21.15.98:0/1871230892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b8101970 msgr2=0x7f22b8199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22be374700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22b8101970 0x7f22b8199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22be374700 1 -- 172.21.15.98:0/1871230892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b81a5470 con 0x7f22b81079c0 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22be374700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22b81079c0 0x7f22b81a4f30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f22b0007d70 tx=0x7f22b000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22aeffd700 1 -- 172.21.15.98:0/1871230892 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0016020 con 0x7f22b81079c0 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22aeffd700 1 -- 172.21.15.98:0/1871230892 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22b000ad40 con 0x7f22b81079c0 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b81a56a0 con 0x7f22b81079c0 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22aeffd700 1 -- 172.21.15.98:0/1871230892 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b00115e0 con 0x7f22b81079c0 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.698+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b81a5b60 con 0x7f22b81079c0 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.699+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f22b8063110 con 0x7f22b81079c0 2024-04-25T17:34:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.700+0000 7f22aeffd700 1 -- 172.21.15.98:0/1871230892 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f22b0019020 con 0x7f22b81079c0 2024-04-25T17:34:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.701+0000 7f22aeffd700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f22a40a3cd0 0x7f22a40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.701+0000 7f22bdb73700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f22a40a3cd0 0x7f22a40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.701+0000 7f22aeffd700 1 -- 172.21.15.98:0/1871230892 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f22b00d0a20 con 0x7f22b81079c0 2024-04-25T17:34:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.701+0000 7f22aeffd700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] conn(0x7f22a40ad810 0x7f22a40afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.701+0000 7f22aeffd700 1 -- 172.21.15.98:0/1871230892 --> [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22b0027e50 con 0x7f22a40ad810 2024-04-25T17:34:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.701+0000 7f22aeffd700 1 -- 172.21.15.98:0/1871230892 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f22b00d0e70 con 0x7f22b81079c0 2024-04-25T17:34:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.701+0000 7f22bdb73700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f22a40a3cd0 0x7f22a40a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f22b4005fd0 tx=0x7f22b400af20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.702+0000 7f22bd372700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] conn(0x7f22a40ad810 0x7f22a40afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.702+0000 7f22bd372700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] conn(0x7f22a40ad810 0x7f22a40afcb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.703+0000 7f22aeffd700 1 -- 172.21.15.98:0/1871230892 <== osd.8 v2:172.21.15.203:6816/2633569866 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22b0027e50 con 0x7f22a40ad810 2024-04-25T17:34:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.719+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 --> [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f22b8063540 con 0x7f22a40ad810 2024-04-25T17:34:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.719+0000 7f22aeffd700 1 -- 172.21.15.98:0/1871230892 <== osd.8 v2:172.21.15.203:6816/2633569866 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f22b8063540 con 0x7f22a40ad810 2024-04-25T17:34:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 >> [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] conn(0x7f22a40ad810 msgr2=0x7f22a40afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22bfdd7700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] conn(0x7f22a40ad810 0x7f22a40afcb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f22a40a3cd0 msgr2=0x7f22a40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22bfdd7700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f22a40a3cd0 0x7f22a40a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f22b4005fd0 tx=0x7f22b400af20 comp rx=0 tx=0).stop 2024-04-25T17:34:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22b81079c0 msgr2=0x7f22b81a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22bfdd7700 1 --2- 172.21.15.98:0/1871230892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22b81079c0 0x7f22b81a4f30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f22b0007d70 tx=0x7f22b000a340 comp rx=0 tx=0).stop 2024-04-25T17:34:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22be374700 1 -- 172.21.15.98:0/1871230892 reap_dead start 2024-04-25T17:34:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 shutdown_connections 2024-04-25T17:34:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 >> 172.21.15.98:0/1871230892 conn(0x7f22b8078980 msgr2=0x7f22b80ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 shutdown_connections 2024-04-25T17:34:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:08.720+0000 7f22bfdd7700 1 -- 172.21.15.98:0/1871230892 wait complete. 2024-04-25T17:34:08.961 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:08.970 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:08.987 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:08.988 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:09.541 INFO:teuthology.orchestra.run.smithi098.stdout:55834574934 2024-04-25T17:34:09.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T17:34:09.943 INFO:teuthology.orchestra.run.smithi098.stdout:146028888135 2024-04-25T17:34:09.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T17:34:10.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:10 smithi098 ceph-mon[29192]: pgmap v351: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:10.380 INFO:teuthology.orchestra.run.smithi098.stdout:257698037809 2024-04-25T17:34:10.381 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T17:34:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:10 smithi203 ceph-mon[32834]: pgmap v351: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:10.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:10 smithi173 ceph-mon[32856]: pgmap v351: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:12.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:12 smithi098 ceph-mon[29192]: pgmap v352: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:12 smithi203 ceph-mon[32834]: pgmap v352: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:12.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:12 smithi173 ceph-mon[32856]: pgmap v352: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:14.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:14 smithi098 ceph-mon[29192]: pgmap v353: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T17:34:14.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:14 smithi203 ceph-mon[32834]: pgmap v353: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T17:34:14.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:14 smithi173 ceph-mon[32856]: pgmap v353: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T17:34:14.790 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:15.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.000+0000 7f452c232700 1 -- 172.21.15.98:0/3893051405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4524106960 msgr2=0x7f4524106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:15.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.000+0000 7f452c232700 1 --2- 172.21.15.98:0/3893051405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4524106960 0x7f4524106d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4520009990 tx=0x7f4520009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 -- 172.21.15.98:0/3893051405 shutdown_connections 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 --2- 172.21.15.98:0/3893051405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45241025e0 0x7f452410e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 --2- 172.21.15.98:0/3893051405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4524106960 0x7f4524106d40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 --2- 172.21.15.98:0/3893051405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4524101c20 0x7f45241020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 -- 172.21.15.98:0/3893051405 >> 172.21.15.98:0/3893051405 conn(0x7f4524078800 msgr2=0x7f4524078c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 -- 172.21.15.98:0/3893051405 shutdown_connections 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 -- 172.21.15.98:0/3893051405 wait complete. 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 Processor -- start 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 -- start start 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4524101c20 0x7f4524199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45241025e0 0x7f4524199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4524106960 0x7f452419d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:15.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4524111c00 con 0x7f4524101c20 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4524111a80 con 0x7f45241025e0 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.001+0000 7f452c232700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4524111d80 con 0x7f4524106960 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f4529fce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4524101c20 0x7f4524199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f4529fce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4524101c20 0x7f4524199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36154/0 (socket says 172.21.15.98:36154) 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f4529fce700 1 -- 172.21.15.98:0/1900938024 learned_addr learned my addr 172.21.15.98:0/1900938024 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f45297cd700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45241025e0 0x7f4524199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f452a7cf700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4524106960 0x7f452419d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f4529fce700 1 -- 172.21.15.98:0/1900938024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4524106960 msgr2=0x7f452419d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f4529fce700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4524106960 0x7f452419d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f4529fce700 1 -- 172.21.15.98:0/1900938024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45241025e0 msgr2=0x7f4524199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f4529fce700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45241025e0 0x7f4524199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f4529fce700 1 -- 172.21.15.98:0/1900938024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452419e110 con 0x7f4524101c20 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f452a7cf700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4524106960 0x7f452419d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f4529fce700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4524101c20 0x7f4524199150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4520000c00 tx=0x7f45200047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f451affd700 1 -- 172.21.15.98:0/1900938024 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4520027070 con 0x7f4524101c20 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f452c232700 1 -- 172.21.15.98:0/1900938024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452419e3a0 con 0x7f4524101c20 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f451affd700 1 -- 172.21.15.98:0/1900938024 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4520003fa0 con 0x7f4524101c20 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f451affd700 1 -- 172.21.15.98:0/1900938024 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4520022db0 con 0x7f4524101c20 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.002+0000 7f452c232700 1 -- 172.21.15.98:0/1900938024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45241a5c80 con 0x7f4524101c20 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.003+0000 7f452c232700 1 -- 172.21.15.98:0/1900938024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4524069730 con 0x7f4524101c20 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.004+0000 7f451affd700 1 -- 172.21.15.98:0/1900938024 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f452002a070 con 0x7f4524101c20 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.005+0000 7f451affd700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f45100a3cd0 0x7f45100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.005+0000 7f45297cd700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f45100a3cd0 0x7f45100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.005+0000 7f451affd700 1 -- 172.21.15.98:0/1900938024 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f45200d12b0 con 0x7f4524101c20 2024-04-25T17:34:15.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.005+0000 7f45297cd700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f45100a3cd0 0x7f45100a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4514005ea0 tx=0x7f4514005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:15.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.009+0000 7f451affd700 1 -- 172.21.15.98:0/1900938024 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4520013070 con 0x7f4524101c20 2024-04-25T17:34:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.218+0000 7f452c232700 1 -- 172.21.15.98:0/1900938024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f452402df80 con 0x7f4524101c20 2024-04-25T17:34:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.218+0000 7f451affd700 1 -- 172.21.15.98:0/1900938024 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f452009dbd0 con 0x7f4524101c20 2024-04-25T17:34:15.219 INFO:teuthology.orchestra.run.smithi098.stdout:77309411411 2024-04-25T17:34:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.220+0000 7f452c232700 1 -- 172.21.15.98:0/1900938024 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f45100a3cd0 msgr2=0x7f45100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.220+0000 7f452c232700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f45100a3cd0 0x7f45100a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4514005ea0 tx=0x7f4514005e30 comp rx=0 tx=0).stop 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.220+0000 7f452c232700 1 -- 172.21.15.98:0/1900938024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4524101c20 msgr2=0x7f4524199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.220+0000 7f452c232700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4524101c20 0x7f4524199150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4520000c00 tx=0x7f45200047b0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.220+0000 7f452c232700 1 -- 172.21.15.98:0/1900938024 shutdown_connections 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.220+0000 7f452c232700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f45100a3cd0 0x7f45100a6190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.220+0000 7f452c232700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4524106960 0x7f452419d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.220+0000 7f452c232700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4524101c20 0x7f4524199150 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.220+0000 7f452c232700 1 --2- 172.21.15.98:0/1900938024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45241025e0 0x7f4524199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.220+0000 7f452c232700 1 -- 172.21.15.98:0/1900938024 >> 172.21.15.98:0/1900938024 conn(0x7f4524078800 msgr2=0x7f452406b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.221+0000 7f452c232700 1 -- 172.21.15.98:0/1900938024 shutdown_connections 2024-04-25T17:34:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.221+0000 7f452c232700 1 -- 172.21.15.98:0/1900938024 wait complete. 2024-04-25T17:34:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 -- 172.21.15.98:0/3919225181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c60107980 msgr2=0x7f8c60107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 --2- 172.21.15.98:0/3919225181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c60107980 0x7f8c60107d60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8c50009990 tx=0x7f8c50009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 -- 172.21.15.98:0/3919225181 shutdown_connections 2024-04-25T17:34:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 --2- 172.21.15.98:0/3919225181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c601022f0 0x7f8c6010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 --2- 172.21.15.98:0/3919225181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c60107980 0x7f8c60107d60 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 --2- 172.21.15.98:0/3919225181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c60101930 0x7f8c60101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 -- 172.21.15.98:0/3919225181 >> 172.21.15.98:0/3919225181 conn(0x7f8c600788b0 msgr2=0x7f8c60078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 -- 172.21.15.98:0/3919225181 shutdown_connections 2024-04-25T17:34:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 -- 172.21.15.98:0/3919225181 wait complete. 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 Processor -- start 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.261+0000 7f8c66806700 1 -- start start 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c66806700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c60101930 0x7f8c60198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c66806700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c601022f0 0x7f8c60198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c66806700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c60107980 0x7f8c6019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c66806700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c60111ae0 con 0x7f8c60107980 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c66806700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8c60111960 con 0x7f8c60101930 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c66806700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8c60111c60 con 0x7f8c601022f0 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c64da3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c60107980 0x7f8c6019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c64da3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c60107980 0x7f8c6019c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36160/0 (socket says 172.21.15.98:36160) 2024-04-25T17:34:15.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c64da3700 1 -- 172.21.15.98:0/2813874853 learned_addr learned my addr 172.21.15.98:0/2813874853 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c5ffff700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c60101930 0x7f8c60198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c5f7fe700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c601022f0 0x7f8c60198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c64da3700 1 -- 172.21.15.98:0/2813874853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c601022f0 msgr2=0x7f8c60198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c64da3700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c601022f0 0x7f8c60198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c64da3700 1 -- 172.21.15.98:0/2813874853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c60101930 msgr2=0x7f8c60198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c64da3700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c60101930 0x7f8c60198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c64da3700 1 -- 172.21.15.98:0/2813874853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c6019d040 con 0x7f8c60107980 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.262+0000 7f8c64da3700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c60107980 0x7f8c6019c900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8c58007f10 tx=0x7f8c5800f5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.263+0000 7f8c5d7fa700 1 -- 172.21.15.98:0/2813874853 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c58016030 con 0x7f8c60107980 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.263+0000 7f8c5d7fa700 1 -- 172.21.15.98:0/2813874853 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c5800fd30 con 0x7f8c60107980 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.263+0000 7f8c5d7fa700 1 -- 172.21.15.98:0/2813874853 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c58012e50 con 0x7f8c60107980 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.263+0000 7f8c66806700 1 -- 172.21.15.98:0/2813874853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c6019d330 con 0x7f8c60107980 2024-04-25T17:34:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.263+0000 7f8c66806700 1 -- 172.21.15.98:0/2813874853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c6006afb0 con 0x7f8c60107980 2024-04-25T17:34:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.264+0000 7f8c66806700 1 -- 172.21.15.98:0/2813874853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c60069730 con 0x7f8c60107980 2024-04-25T17:34:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.265+0000 7f8c5d7fa700 1 -- 172.21.15.98:0/2813874853 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8c58004c40 con 0x7f8c60107980 2024-04-25T17:34:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.265+0000 7f8c5d7fa700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8c480a3ef0 0x7f8c480a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.265+0000 7f8c5ffff700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8c480a3ef0 0x7f8c480a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.265+0000 7f8c5d7fa700 1 -- 172.21.15.98:0/2813874853 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f8c580d2b00 con 0x7f8c60107980 2024-04-25T17:34:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.266+0000 7f8c5ffff700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8c480a3ef0 0x7f8c480a63b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000b5c0 tx=0x7f8c50005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.269+0000 7f8c5d7fa700 1 -- 172.21.15.98:0/2813874853 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c5809f390 con 0x7f8c60107980 2024-04-25T17:34:15.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.477+0000 7f8c66806700 1 -- 172.21.15.98:0/2813874853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f8c6006b9e0 con 0x7f8c60107980 2024-04-25T17:34:15.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.477+0000 7f8c5d7fa700 1 -- 172.21.15.98:0/2813874853 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8c5809ec90 con 0x7f8c60107980 2024-04-25T17:34:15.478 INFO:teuthology.orchestra.run.smithi098.stdout:292057776175 2024-04-25T17:34:15.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.479+0000 7f8c66806700 1 -- 172.21.15.98:0/2813874853 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8c480a3ef0 msgr2=0x7f8c480a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.479+0000 7f8c66806700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8c480a3ef0 0x7f8c480a63b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000b5c0 tx=0x7f8c50005960 comp rx=0 tx=0).stop 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.479+0000 7f8c66806700 1 -- 172.21.15.98:0/2813874853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c60107980 msgr2=0x7f8c6019c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.479+0000 7f8c66806700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c60107980 0x7f8c6019c900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8c58007f10 tx=0x7f8c5800f5c0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.479+0000 7f8c66806700 1 -- 172.21.15.98:0/2813874853 shutdown_connections 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.479+0000 7f8c66806700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8c480a3ef0 0x7f8c480a63b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.479+0000 7f8c66806700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c60107980 0x7f8c6019c900 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.479+0000 7f8c66806700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c60101930 0x7f8c60198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.479+0000 7f8c66806700 1 --2- 172.21.15.98:0/2813874853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c601022f0 0x7f8c60198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.479+0000 7f8c66806700 1 -- 172.21.15.98:0/2813874853 >> 172.21.15.98:0/2813874853 conn(0x7f8c600788b0 msgr2=0x7f8c60100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.480+0000 7f8c66806700 1 -- 172.21.15.98:0/2813874853 shutdown_connections 2024-04-25T17:34:15.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:15.480+0000 7f8c66806700 1 -- 172.21.15.98:0/2813874853 wait complete. 2024-04-25T17:34:16.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:16 smithi098 ceph-mon[29192]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:16.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:16 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1900938024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T17:34:16.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:16 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2813874853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T17:34:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:16 smithi203 ceph-mon[32834]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:16 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1900938024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T17:34:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:16 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2813874853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T17:34:16.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:16 smithi173 ceph-mon[32856]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:16.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:16 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1900938024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T17:34:16.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:16 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2813874853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T17:34:17.623 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411411 for osd.2 2024-04-25T17:34:17.623 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:18.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:18 smithi098 ceph-mon[29192]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:18.265 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:18.267 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:18.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:18 smithi203 ceph-mon[32834]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:18 smithi173 ceph-mon[32856]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:18.702 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776172 got 292057776175 for osd.9 2024-04-25T17:34:18.702 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:19.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.566+0000 7f778dea2700 1 -- 172.21.15.98:0/4258623941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7788102af0 msgr2=0x7f7788102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.566+0000 7f778dea2700 1 --2- 172.21.15.98:0/4258623941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7788102af0 0x7f7788102f70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7778009990 tx=0x7f7778009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.566+0000 7f778dea2700 1 -- 172.21.15.98:0/4258623941 shutdown_connections 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.566+0000 7f778dea2700 1 --2- 172.21.15.98:0/4258623941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77881034b0 0x7f778810f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.567+0000 7f778dea2700 1 --2- 172.21.15.98:0/4258623941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7788108a80 0x7f7788108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.567+0000 7f778dea2700 1 --2- 172.21.15.98:0/4258623941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7788102af0 0x7f7788102f70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.567+0000 7f778dea2700 1 -- 172.21.15.98:0/4258623941 >> 172.21.15.98:0/4258623941 conn(0x7f7788079b20 msgr2=0x7f7788079f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.567+0000 7f778dea2700 1 -- 172.21.15.98:0/4258623941 shutdown_connections 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.567+0000 7f778dea2700 1 -- 172.21.15.98:0/4258623941 wait complete. 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.567+0000 7f778dea2700 1 Processor -- start 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.567+0000 7f778dea2700 1 -- start start 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f778dea2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7788102af0 0x7f77881991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f778dea2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77881034b0 0x7f77881996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f778dea2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7788108a80 0x7f778819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f778dea2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7788112cb0 con 0x7f7788102af0 2024-04-25T17:34:19.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f778dea2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7788112b30 con 0x7f77881034b0 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f778dea2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7788112e30 con 0x7f7788108a80 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f77877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7788102af0 0x7f77881991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f77877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7788102af0 0x7f77881991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36188/0 (socket says 172.21.15.98:36188) 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f77877fe700 1 -- 172.21.15.98:0/3774309713 learned_addr learned my addr 172.21.15.98:0/3774309713 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f7786ffd700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77881034b0 0x7f77881996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f77877fe700 1 -- 172.21.15.98:0/3774309713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7788108a80 msgr2=0x7f778819da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f77877fe700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7788108a80 0x7f778819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f77877fe700 1 -- 172.21.15.98:0/3774309713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77881034b0 msgr2=0x7f77881996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f77877fe700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77881034b0 0x7f77881996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f77877fe700 1 -- 172.21.15.98:0/3774309713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f778819e1b0 con 0x7f7788102af0 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f77877fe700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7788102af0 0x7f77881991a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f777c00c970 tx=0x7f777c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f7784ff9700 1 -- 172.21.15.98:0/3774309713 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777c007aa0 con 0x7f7788102af0 2024-04-25T17:34:19.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f7784ff9700 1 -- 172.21.15.98:0/3774309713 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f777c004ce0 con 0x7f7788102af0 2024-04-25T17:34:19.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f778dea2700 1 -- 172.21.15.98:0/3774309713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f778819e4a0 con 0x7f7788102af0 2024-04-25T17:34:19.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f7784ff9700 1 -- 172.21.15.98:0/3774309713 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777c0055c0 con 0x7f7788102af0 2024-04-25T17:34:19.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.568+0000 7f778dea2700 1 -- 172.21.15.98:0/3774309713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77881a5d20 con 0x7f7788102af0 2024-04-25T17:34:19.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.569+0000 7f778dea2700 1 -- 172.21.15.98:0/3774309713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7788069730 con 0x7f7788102af0 2024-04-25T17:34:19.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.570+0000 7f7784ff9700 1 -- 172.21.15.98:0/3774309713 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f777c007c40 con 0x7f7788102af0 2024-04-25T17:34:19.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.571+0000 7f7784ff9700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f77700a3cd0 0x7f77700a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:19.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.571+0000 7f7786ffd700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f77700a3cd0 0x7f77700a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:19.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.571+0000 7f7784ff9700 1 -- 172.21.15.98:0/3774309713 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f777c0d0650 con 0x7f7788102af0 2024-04-25T17:34:19.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.572+0000 7f7786ffd700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f77700a3cd0 0x7f77700a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7778005cd0 tx=0x7f7778005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:19.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.575+0000 7f7784ff9700 1 -- 172.21.15.98:0/3774309713 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f777c09d000 con 0x7f7788102af0 2024-04-25T17:34:19.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.781+0000 7f778dea2700 1 -- 172.21.15.98:0/3774309713 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f778802df80 con 0x7f7788102af0 2024-04-25T17:34:19.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.781+0000 7f7784ff9700 1 -- 172.21.15.98:0/3774309713 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f777c09c900 con 0x7f7788102af0 2024-04-25T17:34:19.783 INFO:teuthology.orchestra.run.smithi098.stdout:201863462976 2024-04-25T17:34:19.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.783+0000 7f778dea2700 1 -- 172.21.15.98:0/3774309713 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f77700a3cd0 msgr2=0x7f77700a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:19.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.783+0000 7f778dea2700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f77700a3cd0 0x7f77700a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7778005cd0 tx=0x7f7778005c40 comp rx=0 tx=0).stop 2024-04-25T17:34:19.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.783+0000 7f778dea2700 1 -- 172.21.15.98:0/3774309713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7788102af0 msgr2=0x7f77881991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:19.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.783+0000 7f778dea2700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7788102af0 0x7f77881991a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f777c00c970 tx=0x7f777c00cd30 comp rx=0 tx=0).stop 2024-04-25T17:34:19.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.783+0000 7f778dea2700 1 -- 172.21.15.98:0/3774309713 shutdown_connections 2024-04-25T17:34:19.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.783+0000 7f778dea2700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f77700a3cd0 0x7f77700a6190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:19.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.783+0000 7f778dea2700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7788108a80 0x7f778819da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:19.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.783+0000 7f778dea2700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7788102af0 0x7f77881991a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:19.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.783+0000 7f778dea2700 1 --2- 172.21.15.98:0/3774309713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77881034b0 0x7f77881996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:19.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.783+0000 7f778dea2700 1 -- 172.21.15.98:0/3774309713 >> 172.21.15.98:0/3774309713 conn(0x7f7788079b20 msgr2=0x7f7788101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:19.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.784+0000 7f778dea2700 1 -- 172.21.15.98:0/3774309713 shutdown_connections 2024-04-25T17:34:19.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:19.784+0000 7f778dea2700 1 -- 172.21.15.98:0/3774309713 wait complete. 2024-04-25T17:34:20.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:20 smithi098 ceph-mon[29192]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:20.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:20 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3774309713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T17:34:20.325 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:20 smithi173 ceph-mon[32856]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:20.325 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:20 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3774309713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T17:34:20.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:20 smithi203 ceph-mon[32834]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:20.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:20 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3774309713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T17:34:20.526 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462971 got 201863462976 for osd.6 2024-04-25T17:34:20.527 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.707+0000 7f21b5867700 1 -- 172.21.15.98:0/24524637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b006b190 msgr2=0x7f21b006b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.707+0000 7f21b5867700 1 --2- 172.21.15.98:0/24524637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b006b190 0x7f21b006b5f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009990 tx=0x7f21a4009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.707+0000 7f21b5867700 1 -- 172.21.15.98:0/24524637 shutdown_connections 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.707+0000 7f21b5867700 1 --2- 172.21.15.98:0/24524637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b006bb30 0x7f21b010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.707+0000 7f21b5867700 1 --2- 172.21.15.98:0/24524637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b0108a30 0x7f21b0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.707+0000 7f21b5867700 1 --2- 172.21.15.98:0/24524637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b006b190 0x7f21b006b5f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.707+0000 7f21b5867700 1 -- 172.21.15.98:0/24524637 >> 172.21.15.98:0/24524637 conn(0x7f21b0078910 msgr2=0x7f21b0078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.707+0000 7f21b5867700 1 -- 172.21.15.98:0/24524637 shutdown_connections 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.707+0000 7f21b5867700 1 -- 172.21.15.98:0/24524637 wait complete. 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21b5867700 1 Processor -- start 2024-04-25T17:34:20.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21b5867700 1 -- start start 2024-04-25T17:34:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21b5867700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b006b190 0x7f21b01991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21b5867700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b006bb30 0x7f21b01996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21b5867700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0108a30 0x7f21b019da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:20.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21b5867700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21b0112c50 con 0x7f21b006bb30 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21b5867700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f21b0112ad0 con 0x7f21b006b190 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21b5867700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21b0112dd0 con 0x7f21b0108a30 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21ae7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b006bb30 0x7f21b01996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21ae7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b006bb30 0x7f21b01996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36198/0 (socket says 172.21.15.98:36198) 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.708+0000 7f21ae7fc700 1 -- 172.21.15.98:0/948443321 learned_addr learned my addr 172.21.15.98:0/948443321 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21af7fe700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0108a30 0x7f21b019da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21ae7fc700 1 -- 172.21.15.98:0/948443321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0108a30 msgr2=0x7f21b019da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21aeffd700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b006b190 0x7f21b01991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21ae7fc700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0108a30 0x7f21b019da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21ae7fc700 1 -- 172.21.15.98:0/948443321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b006b190 msgr2=0x7f21b01991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21ae7fc700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b006b190 0x7f21b01991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21ae7fc700 1 -- 172.21.15.98:0/948443321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b019e1b0 con 0x7f21b006bb30 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21aeffd700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b006b190 0x7f21b01991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:34:20.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21ae7fc700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b006bb30 0x7f21b01996e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f21a4000c00 tx=0x7f21a40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21b4865700 1 -- 172.21.15.98:0/948443321 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a4027070 con 0x7f21b006bb30 2024-04-25T17:34:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21b5867700 1 -- 172.21.15.98:0/948443321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b019e440 con 0x7f21b006bb30 2024-04-25T17:34:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21b4865700 1 -- 172.21.15.98:0/948443321 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21a4003fa0 con 0x7f21b006bb30 2024-04-25T17:34:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21b4865700 1 -- 172.21.15.98:0/948443321 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a4022db0 con 0x7f21b006bb30 2024-04-25T17:34:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.709+0000 7f21b5867700 1 -- 172.21.15.98:0/948443321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b01a5d20 con 0x7f21b006bb30 2024-04-25T17:34:20.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.710+0000 7f21b5867700 1 -- 172.21.15.98:0/948443321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21b0104170 con 0x7f21b006bb30 2024-04-25T17:34:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.711+0000 7f21b4865700 1 -- 172.21.15.98:0/948443321 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f21a402a070 con 0x7f21b006bb30 2024-04-25T17:34:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.712+0000 7f21b4865700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f21980a3cd0 0x7f21980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.712+0000 7f21aeffd700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f21980a3cd0 0x7f21980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.712+0000 7f21b4865700 1 -- 172.21.15.98:0/948443321 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f21a40d16a0 con 0x7f21b006bb30 2024-04-25T17:34:20.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.713+0000 7f21aeffd700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f21980a3cd0 0x7f21980a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f21a0007900 tx=0x7f21a0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:20.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.716+0000 7f21b4865700 1 -- 172.21.15.98:0/948443321 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21a4013070 con 0x7f21b006bb30 2024-04-25T17:34:20.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.923+0000 7f21b5867700 1 -- 172.21.15.98:0/948443321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f21b002df80 con 0x7f21b006bb30 2024-04-25T17:34:20.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.924+0000 7f21b4865700 1 -- 172.21.15.98:0/948443321 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f21a409df60 con 0x7f21b006bb30 2024-04-25T17:34:20.925 INFO:teuthology.orchestra.run.smithi098.stdout:313532612651 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 -- 172.21.15.98:0/948443321 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f21980a3cd0 msgr2=0x7f21980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f21980a3cd0 0x7f21980a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f21a0007900 tx=0x7f21a0008040 comp rx=0 tx=0).stop 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 -- 172.21.15.98:0/948443321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b006bb30 msgr2=0x7f21b01996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b006bb30 0x7f21b01996e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f21a4000c00 tx=0x7f21a40047b0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 -- 172.21.15.98:0/948443321 shutdown_connections 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f21980a3cd0 0x7f21980a6190 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0108a30 0x7f21b019da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b006b190 0x7f21b01991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 --2- 172.21.15.98:0/948443321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b006bb30 0x7f21b01996e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 -- 172.21.15.98:0/948443321 >> 172.21.15.98:0/948443321 conn(0x7f21b0078910 msgr2=0x7f21b00ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:20.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 -- 172.21.15.98:0/948443321 shutdown_connections 2024-04-25T17:34:20.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.926+0000 7f21b5867700 1 -- 172.21.15.98:0/948443321 wait complete. 2024-04-25T17:34:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.972+0000 7f63b77d4700 1 -- 172.21.15.98:0/4098487123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b0107990 msgr2=0x7f63b0107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.972+0000 7f63b77d4700 1 --2- 172.21.15.98:0/4098487123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b0107990 0x7f63b0107d70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009990 tx=0x7f63ac009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.972+0000 7f63b77d4700 1 -- 172.21.15.98:0/4098487123 shutdown_connections 2024-04-25T17:34:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.972+0000 7f63b77d4700 1 --2- 172.21.15.98:0/4098487123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63b0102300 0x7f63b010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.972+0000 7f63b77d4700 1 --2- 172.21.15.98:0/4098487123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b0107990 0x7f63b0107d70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.972+0000 7f63b77d4700 1 --2- 172.21.15.98:0/4098487123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63b0101940 0x7f63b0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.972+0000 7f63b77d4700 1 -- 172.21.15.98:0/4098487123 >> 172.21.15.98:0/4098487123 conn(0x7f63b00788f0 msgr2=0x7f63b0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.972+0000 7f63b77d4700 1 -- 172.21.15.98:0/4098487123 shutdown_connections 2024-04-25T17:34:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.972+0000 7f63b77d4700 1 -- 172.21.15.98:0/4098487123 wait complete. 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b77d4700 1 Processor -- start 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b77d4700 1 -- start start 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b77d4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63b0101940 0x7f63b01034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b77d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b0102300 0x7f63b0103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b77d4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63b0107990 0x7f63b0103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b77d4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63b0111ae0 con 0x7f63b0102300 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b77d4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f63b0111960 con 0x7f63b0101940 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b77d4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f63b0111c60 con 0x7f63b0107990 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b4d6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b0102300 0x7f63b0103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b4d6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b0102300 0x7f63b0103a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36216/0 (socket says 172.21.15.98:36216) 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b4d6f700 1 -- 172.21.15.98:0/3811678761 learned_addr learned my addr 172.21.15.98:0/3811678761 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b5570700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63b0101940 0x7f63b01034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b5d71700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63b0107990 0x7f63b0103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b4d6f700 1 -- 172.21.15.98:0/3811678761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63b0107990 msgr2=0x7f63b0103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b4d6f700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63b0107990 0x7f63b0103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b4d6f700 1 -- 172.21.15.98:0/3811678761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63b0101940 msgr2=0x7f63b01034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b4d6f700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63b0101940 0x7f63b01034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.973+0000 7f63b4d6f700 1 -- 172.21.15.98:0/3811678761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b01a7600 con 0x7f63b0102300 2024-04-25T17:34:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.974+0000 7f63b4d6f700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b0102300 0x7f63b0103a30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f63a000c920 tx=0x7f63a000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.974+0000 7f63a67fc700 1 -- 172.21.15.98:0/3811678761 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0007b00 con 0x7f63b0102300 2024-04-25T17:34:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.974+0000 7f63b77d4700 1 -- 172.21.15.98:0/3811678761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63b01a78f0 con 0x7f63b0102300 2024-04-25T17:34:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.974+0000 7f63a67fc700 1 -- 172.21.15.98:0/3811678761 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a0004ce0 con 0x7f63b0102300 2024-04-25T17:34:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.974+0000 7f63a67fc700 1 -- 172.21.15.98:0/3811678761 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0005500 con 0x7f63b0102300 2024-04-25T17:34:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.974+0000 7f63b77d4700 1 -- 172.21.15.98:0/3811678761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63b01a7e30 con 0x7f63b0102300 2024-04-25T17:34:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.975+0000 7f63b77d4700 1 -- 172.21.15.98:0/3811678761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63b0069730 con 0x7f63b0102300 2024-04-25T17:34:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.976+0000 7f63a67fc700 1 -- 172.21.15.98:0/3811678761 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f63a0007ca0 con 0x7f63b0102300 2024-04-25T17:34:20.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.976+0000 7f63a67fc700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f639c0a3cd0 0x7f639c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:20.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.977+0000 7f63b5570700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f639c0a3cd0 0x7f639c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:20.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.977+0000 7f63a67fc700 1 -- 172.21.15.98:0/3811678761 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f63a00d0110 con 0x7f63b0102300 2024-04-25T17:34:20.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.977+0000 7f63b5570700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f639c0a3cd0 0x7f639c0a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00b5c0 tx=0x7f63ac005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:20.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:20.981+0000 7f63a67fc700 1 -- 172.21.15.98:0/3811678761 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63a0014030 con 0x7f63b0102300 2024-04-25T17:34:21.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.182+0000 7f63b77d4700 1 -- 172.21.15.98:0/3811678761 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f63b01a8000 con 0x7f63b0102300 2024-04-25T17:34:21.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.182+0000 7f63a67fc700 1 -- 172.21.15.98:0/3811678761 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f63a009c9a0 con 0x7f63b0102300 2024-04-25T17:34:21.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:21 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/948443321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T17:34:21.184 INFO:teuthology.orchestra.run.smithi098.stdout:111669149775 2024-04-25T17:34:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 -- 172.21.15.98:0/3811678761 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f639c0a3cd0 msgr2=0x7f639c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f639c0a3cd0 0x7f639c0a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00b5c0 tx=0x7f63ac005960 comp rx=0 tx=0).stop 2024-04-25T17:34:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 -- 172.21.15.98:0/3811678761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b0102300 msgr2=0x7f63b0103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b0102300 0x7f63b0103a30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f63a000c920 tx=0x7f63a000cc30 comp rx=0 tx=0).stop 2024-04-25T17:34:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 -- 172.21.15.98:0/3811678761 shutdown_connections 2024-04-25T17:34:21.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f639c0a3cd0 0x7f639c0a6190 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63b0107990 0x7f63b0103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63b0101940 0x7f63b01034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 --2- 172.21.15.98:0/3811678761 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b0102300 0x7f63b0103a30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 -- 172.21.15.98:0/3811678761 >> 172.21.15.98:0/3811678761 conn(0x7f63b00788f0 msgr2=0x7f63b006a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 -- 172.21.15.98:0/3811678761 shutdown_connections 2024-04-25T17:34:21.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:21.184+0000 7f63b77d4700 1 -- 172.21.15.98:0/3811678761 wait complete. 2024-04-25T17:34:21.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:21 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/948443321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T17:34:21.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:21 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/948443321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T17:34:22.098 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149772 got 111669149775 for osd.3 2024-04-25T17:34:22.098 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:22.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:22 smithi098 ceph-mon[29192]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:22.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:22 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3811678761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T17:34:22.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:22 smithi203 ceph-mon[32834]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:22.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:22 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3811678761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T17:34:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:22.513 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:22.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:22 smithi173 ceph-mon[32856]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:22.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:22 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3811678761' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T17:34:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:23.669 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612647 got 313532612651 for osd.10 2024-04-25T17:34:23.669 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:24.322 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:24.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:24 smithi098 ceph-mon[29192]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:34:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:24 smithi203 ceph-mon[32834]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:34:24.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:24 smithi173 ceph-mon[32856]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:34:26.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:26 smithi098 ceph-mon[29192]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:26.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:26 smithi203 ceph-mon[32834]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:26.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:26 smithi173 ceph-mon[32856]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:28.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.121+0000 7fa53cdc1700 1 -- 172.21.15.98:0/3038349850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538101930 msgr2=0x7fa538101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:28.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.121+0000 7fa53cdc1700 1 --2- 172.21.15.98:0/3038349850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538101930 0x7fa538101db0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa52c009990 tx=0x7fa52c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:28.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.121+0000 7fa53cdc1700 1 -- 172.21.15.98:0/3038349850 shutdown_connections 2024-04-25T17:34:28.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.121+0000 7fa53cdc1700 1 --2- 172.21.15.98:0/3038349850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5381022f0 0x7fa53810e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:28.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.121+0000 7fa53cdc1700 1 --2- 172.21.15.98:0/3038349850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa538107980 0x7fa538107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:28.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.121+0000 7fa53cdc1700 1 --2- 172.21.15.98:0/3038349850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538101930 0x7fa538101db0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:28.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.121+0000 7fa53cdc1700 1 -- 172.21.15.98:0/3038349850 >> 172.21.15.98:0/3038349850 conn(0x7fa5380788b0 msgr2=0x7fa538078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:28.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.121+0000 7fa53cdc1700 1 -- 172.21.15.98:0/3038349850 shutdown_connections 2024-04-25T17:34:28.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.121+0000 7fa53cdc1700 1 -- 172.21.15.98:0/3038349850 wait complete. 2024-04-25T17:34:28.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.122+0000 7fa53cdc1700 1 Processor -- start 2024-04-25T17:34:28.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.122+0000 7fa53cdc1700 1 -- start start 2024-04-25T17:34:28.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa53cdc1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa538101930 0x7fa538197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:28.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa53cdc1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5381022f0 0x7fa538198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:28.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa53cdc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538107980 0x7fa53819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:28.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa53cdc1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa538111ae0 con 0x7fa538107980 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa53cdc1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa538111960 con 0x7fa538101930 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa53cdc1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa538111c60 con 0x7fa5381022f0 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa536d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538107980 0x7fa53819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa536d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538107980 0x7fa53819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56040/0 (socket says 172.21.15.98:56040) 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa536d9d700 1 -- 172.21.15.98:0/2462751608 learned_addr learned my addr 172.21.15.98:0/2462751608 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa535d9b700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5381022f0 0x7fa538198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa53659c700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa538101930 0x7fa538197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa536d9d700 1 -- 172.21.15.98:0/2462751608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5381022f0 msgr2=0x7fa538198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa536d9d700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5381022f0 0x7fa538198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa536d9d700 1 -- 172.21.15.98:0/2462751608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa538101930 msgr2=0x7fa538197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:28.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa536d9d700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa538101930 0x7fa538197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:28.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa536d9d700 1 -- 172.21.15.98:0/2462751608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53819cf60 con 0x7fa538107980 2024-04-25T17:34:28.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.123+0000 7fa536d9d700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538107980 0x7fa53819c820 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa53000ada0 tx=0x7fa53000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:28.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.124+0000 7fa5277fe700 1 -- 172.21.15.98:0/2462751608 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530017020 con 0x7fa538107980 2024-04-25T17:34:28.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.124+0000 7fa53cdc1700 1 -- 172.21.15.98:0/2462751608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa53819d250 con 0x7fa538107980 2024-04-25T17:34:28.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.124+0000 7fa5277fe700 1 -- 172.21.15.98:0/2462751608 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa53000ce30 con 0x7fa538107980 2024-04-25T17:34:28.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.124+0000 7fa5277fe700 1 -- 172.21.15.98:0/2462751608 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa530005350 con 0x7fa538107980 2024-04-25T17:34:28.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.124+0000 7fa53cdc1700 1 -- 172.21.15.98:0/2462751608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5381a4ad0 con 0x7fa538107980 2024-04-25T17:34:28.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.125+0000 7fa53cdc1700 1 -- 172.21.15.98:0/2462751608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa538069730 con 0x7fa538107980 2024-04-25T17:34:28.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.126+0000 7fa5277fe700 1 -- 172.21.15.98:0/2462751608 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa53000c980 con 0x7fa538107980 2024-04-25T17:34:28.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.127+0000 7fa5277fe700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa5200a3d50 0x7fa5200a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:28.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.127+0000 7fa53659c700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa5200a3d50 0x7fa5200a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:28.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.127+0000 7fa5277fe700 1 -- 172.21.15.98:0/2462751608 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fa5300d01c0 con 0x7fa538107980 2024-04-25T17:34:28.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.128+0000 7fa53659c700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa5200a3d50 0x7fa5200a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa528005fd0 tx=0x7fa52800af20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:28.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.131+0000 7fa5277fe700 1 -- 172.21.15.98:0/2462751608 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa530021020 con 0x7fa538107980 2024-04-25T17:34:28.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:28 smithi098 ceph-mon[29192]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.343+0000 7fa53cdc1700 1 -- 172.21.15.98:0/2462751608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fa53802df80 con 0x7fa538107980 2024-04-25T17:34:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.343+0000 7fa5277fe700 1 -- 172.21.15.98:0/2462751608 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa53009cae0 con 0x7fa538107980 2024-04-25T17:34:28.345 INFO:teuthology.orchestra.run.smithi098.stdout:347892351013 2024-04-25T17:34:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.345+0000 7fa53cdc1700 1 -- 172.21.15.98:0/2462751608 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa5200a3d50 msgr2=0x7fa5200a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.345+0000 7fa53cdc1700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa5200a3d50 0x7fa5200a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa528005fd0 tx=0x7fa52800af20 comp rx=0 tx=0).stop 2024-04-25T17:34:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.345+0000 7fa53cdc1700 1 -- 172.21.15.98:0/2462751608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538107980 msgr2=0x7fa53819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.345+0000 7fa53cdc1700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538107980 0x7fa53819c820 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa53000ada0 tx=0x7fa53000c330 comp rx=0 tx=0).stop 2024-04-25T17:34:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.345+0000 7fa53cdc1700 1 -- 172.21.15.98:0/2462751608 shutdown_connections 2024-04-25T17:34:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.345+0000 7fa53cdc1700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa5200a3d50 0x7fa5200a6210 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.345+0000 7fa53cdc1700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa538107980 0x7fa53819c820 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.345+0000 7fa53cdc1700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa538101930 0x7fa538197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.345+0000 7fa53cdc1700 1 --2- 172.21.15.98:0/2462751608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5381022f0 0x7fa538198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.345+0000 7fa53cdc1700 1 -- 172.21.15.98:0/2462751608 >> 172.21.15.98:0/2462751608 conn(0x7fa5380788b0 msgr2=0x7fa5380ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.346+0000 7fa53cdc1700 1 -- 172.21.15.98:0/2462751608 shutdown_connections 2024-04-25T17:34:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.346+0000 7fa53cdc1700 1 -- 172.21.15.98:0/2462751608 wait complete. 2024-04-25T17:34:28.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:28 smithi203 ceph-mon[32834]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:28.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:28 smithi173 ceph-mon[32856]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:29.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:29 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3813415863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T17:34:29.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:29 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2462751608' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T17:34:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.012+0000 7f10587d8700 1 -- 172.21.15.98:0/3773809124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1050103480 msgr2=0x7f105010f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.012+0000 7f10587d8700 1 --2- 172.21.15.98:0/3773809124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1050103480 0x7f105010f8d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f104800b0a0 tx=0x7f104800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.013+0000 7f10587d8700 1 -- 172.21.15.98:0/3773809124 shutdown_connections 2024-04-25T17:34:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.013+0000 7f10587d8700 1 --2- 172.21.15.98:0/3773809124 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1050103480 0x7f105010f8d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.013+0000 7f10587d8700 1 --2- 172.21.15.98:0/3773809124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1050108a70 0x7f1050108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.013+0000 7f10587d8700 1 --2- 172.21.15.98:0/3773809124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1050102ac0 0x7f1050102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.013+0000 7f10587d8700 1 -- 172.21.15.98:0/3773809124 >> 172.21.15.98:0/3773809124 conn(0x7f1050079bb0 msgr2=0x7f1050079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.013+0000 7f10587d8700 1 -- 172.21.15.98:0/3773809124 shutdown_connections 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.013+0000 7f10587d8700 1 -- 172.21.15.98:0/3773809124 wait complete. 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.013+0000 7f10587d8700 1 Processor -- start 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.013+0000 7f10587d8700 1 -- start start 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10587d8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1050102ac0 0x7f1050199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10587d8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1050103480 0x7f1050199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10587d8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1050108a70 0x7f105019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10587d8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1050112c00 con 0x7f1050103480 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10587d8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1050112a80 con 0x7f1050108a70 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10587d8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1050112d80 con 0x7f1050102ac0 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1055d73700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1050103480 0x7f1050199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1055d73700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1050103480 0x7f1050199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56012/0 (socket says 172.21.15.98:56012) 2024-04-25T17:34:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1055d73700 1 -- 172.21.15.98:0/3813415863 learned_addr learned my addr 172.21.15.98:0/3813415863 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1056d75700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1050108a70 0x7f105019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1056574700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1050102ac0 0x7f1050199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1055d73700 1 -- 172.21.15.98:0/3813415863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1050102ac0 msgr2=0x7f1050199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1055d73700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1050102ac0 0x7f1050199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1055d73700 1 -- 172.21.15.98:0/3813415863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1050108a70 msgr2=0x7f105019d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1055d73700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1050108a70 0x7f105019d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1055d73700 1 -- 172.21.15.98:0/3813415863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105019e060 con 0x7f1050103480 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1056574700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1050102ac0 0x7f1050199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f1055d73700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1050103480 0x7f1050199590 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f104000c920 tx=0x7f104000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10477fe700 1 -- 172.21.15.98:0/3813415863 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1040007b00 con 0x7f1050103480 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10587d8700 1 -- 172.21.15.98:0/3813415863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f105019e350 con 0x7f1050103480 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10477fe700 1 -- 172.21.15.98:0/3813415863 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1040004ce0 con 0x7f1050103480 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10477fe700 1 -- 172.21.15.98:0/3813415863 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1040005500 con 0x7f1050103480 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.014+0000 7f10587d8700 1 -- 172.21.15.98:0/3813415863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10501a5bd0 con 0x7f1050103480 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.015+0000 7f10587d8700 1 -- 172.21.15.98:0/3813415863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1050069730 con 0x7f1050103480 2024-04-25T17:34:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.016+0000 7f10477fe700 1 -- 172.21.15.98:0/3813415863 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1040007ca0 con 0x7f1050103480 2024-04-25T17:34:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.017+0000 7f10477fe700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f103c0a3cd0 0x7f103c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.017+0000 7f1056574700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f103c0a3cd0 0x7f103c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.017+0000 7f10477fe700 1 -- 172.21.15.98:0/3813415863 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f10400d04d0 con 0x7f1050103480 2024-04-25T17:34:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.018+0000 7f1056574700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f103c0a3cd0 0x7f103c0a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f104c005fd0 tx=0x7f104c005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.022+0000 7f10477fe700 1 -- 172.21.15.98:0/3813415863 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1040014030 con 0x7f1050103480 2024-04-25T17:34:29.341 INFO:teuthology.orchestra.run.smithi098.stdout:34359738463 2024-04-25T17:34:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.234+0000 7f10587d8700 1 -- 172.21.15.98:0/3813415863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f105002df80 con 0x7f1050103480 2024-04-25T17:34:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.235+0000 7f10477fe700 1 -- 172.21.15.98:0/3813415863 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f104009cd60 con 0x7f1050103480 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 -- 172.21.15.98:0/3813415863 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f103c0a3cd0 msgr2=0x7f103c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f103c0a3cd0 0x7f103c0a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f104c005fd0 tx=0x7f104c005db0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 -- 172.21.15.98:0/3813415863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1050103480 msgr2=0x7f1050199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1050103480 0x7f1050199590 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f104000c920 tx=0x7f104000cc30 comp rx=0 tx=0).stop 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 -- 172.21.15.98:0/3813415863 shutdown_connections 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f103c0a3cd0 0x7f103c0a6190 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1050102ac0 0x7f1050199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1050108a70 0x7f105019d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 --2- 172.21.15.98:0/3813415863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1050103480 0x7f1050199590 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 -- 172.21.15.98:0/3813415863 >> 172.21.15.98:0/3813415863 conn(0x7f1050079bb0 msgr2=0x7f1050101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 -- 172.21.15.98:0/3813415863 shutdown_connections 2024-04-25T17:34:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:28.237+0000 7f10587d8700 1 -- 172.21.15.98:0/3813415863 wait complete. 2024-04-25T17:34:29.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:29 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3813415863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T17:34:29.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:29 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2462751608' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T17:34:29.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:29 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3813415863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T17:34:29.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:29 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2462751608' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T17:34:29.554 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:30.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:30 smithi098 ceph-mon[29192]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:30.466 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:30 smithi203 ceph-mon[32834]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.464+0000 7f2590fbd700 1 -- 172.21.15.98:0/3990166953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f258c106400 msgr2=0x7f258c10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.464+0000 7f2590fbd700 1 --2- 172.21.15.98:0/3990166953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f258c106400 0x7f258c10dfd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f258400b0d0 tx=0x7f258400b3e0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.464+0000 7f2590fbd700 1 -- 172.21.15.98:0/3990166953 shutdown_connections 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.464+0000 7f2590fbd700 1 --2- 172.21.15.98:0/3990166953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f258c106400 0x7f258c10dfd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.464+0000 7f2590fbd700 1 --2- 172.21.15.98:0/3990166953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f258c101320 0x7f258c101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.464+0000 7f2590fbd700 1 --2- 172.21.15.98:0/3990166953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f258c101cd0 0x7f258c105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.464+0000 7f2590fbd700 1 -- 172.21.15.98:0/3990166953 >> 172.21.15.98:0/3990166953 conn(0x7f258c077f40 msgr2=0x7f258c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.464+0000 7f2590fbd700 1 -- 172.21.15.98:0/3990166953 shutdown_connections 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.465+0000 7f2590fbd700 1 -- 172.21.15.98:0/3990166953 wait complete. 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.465+0000 7f2590fbd700 1 Processor -- start 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.465+0000 7f2590fbd700 1 -- start start 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.465+0000 7f2590fbd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f258c101320 0x7f258c0fd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.465+0000 7f2590fbd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f258c101cd0 0x7f258c0fd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.465+0000 7f2590fbd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f258c106400 0x7f258c0fde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.465+0000 7f2590fbd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f258c110a30 con 0x7f258c101cd0 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.465+0000 7f2590fbd700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f258c1108b0 con 0x7f258c101320 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2590fbd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f258c110bb0 con 0x7f258c106400 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2589d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f258c101cd0 0x7f258c0fd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2589d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f258c101cd0 0x7f258c0fd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56064/0 (socket says 172.21.15.98:56064) 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2589d9b700 1 -- 172.21.15.98:0/1801453320 learned_addr learned my addr 172.21.15.98:0/1801453320 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f258ad9d700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f258c106400 0x7f258c0fde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2589d9b700 1 -- 172.21.15.98:0/1801453320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f258c106400 msgr2=0x7f258c0fde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f258a59c700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f258c101320 0x7f258c0fd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2589d9b700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f258c106400 0x7f258c0fde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2589d9b700 1 -- 172.21.15.98:0/1801453320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f258c101320 msgr2=0x7f258c0fd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2589d9b700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f258c101320 0x7f258c0fd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2589d9b700 1 -- 172.21.15.98:0/1801453320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258c1a76b0 con 0x7f258c101cd0 2024-04-25T17:34:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f258ad9d700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f258c106400 0x7f258c0fde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:34:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2589d9b700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f258c101cd0 0x7f258c0fd8c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f258000d950 tx=0x7f258000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f257b7fe700 1 -- 172.21.15.98:0/1801453320 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2580017070 con 0x7f258c101cd0 2024-04-25T17:34:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f257b7fe700 1 -- 172.21.15.98:0/1801453320 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f258000de30 con 0x7f258c101cd0 2024-04-25T17:34:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2590fbd700 1 -- 172.21.15.98:0/1801453320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f258c1a79a0 con 0x7f258c101cd0 2024-04-25T17:34:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f257b7fe700 1 -- 172.21.15.98:0/1801453320 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2580005440 con 0x7f258c101cd0 2024-04-25T17:34:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.466+0000 7f2590fbd700 1 -- 172.21.15.98:0/1801453320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f258c1a7e90 con 0x7f258c101cd0 2024-04-25T17:34:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.467+0000 7f2590fbd700 1 -- 172.21.15.98:0/1801453320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f258c069730 con 0x7f258c101cd0 2024-04-25T17:34:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.468+0000 7f257b7fe700 1 -- 172.21.15.98:0/1801453320 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2580004750 con 0x7f258c101cd0 2024-04-25T17:34:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.469+0000 7f257b7fe700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25740a3cd0 0x7f25740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.469+0000 7f258a59c700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25740a3cd0 0x7f25740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.469+0000 7f257b7fe700 1 -- 172.21.15.98:0/1801453320 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f25800d01a0 con 0x7f258c101cd0 2024-04-25T17:34:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.469+0000 7f258a59c700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25740a3cd0 0x7f25740a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f257c005950 tx=0x7f257c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.473+0000 7f257b7fe700 1 -- 172.21.15.98:0/1801453320 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f258009ca30 con 0x7f258c101cd0 2024-04-25T17:34:30.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:30 smithi173 ceph-mon[32856]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:30.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.672+0000 7f2590fbd700 1 -- 172.21.15.98:0/1801453320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f258c02e140 con 0x7f258c101cd0 2024-04-25T17:34:30.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.672+0000 7f257b7fe700 1 -- 172.21.15.98:0/1801453320 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f258009c330 con 0x7f258c101cd0 2024-04-25T17:34:30.673 INFO:teuthology.orchestra.run.smithi098.stdout:257698037813 2024-04-25T17:34:30.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 -- 172.21.15.98:0/1801453320 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25740a3cd0 msgr2=0x7f25740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:30.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25740a3cd0 0x7f25740a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f257c005950 tx=0x7f257c0058e0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 -- 172.21.15.98:0/1801453320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f258c101cd0 msgr2=0x7f258c0fd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:30.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f258c101cd0 0x7f258c0fd8c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f258000d950 tx=0x7f258000dd10 comp rx=0 tx=0).stop 2024-04-25T17:34:30.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 -- 172.21.15.98:0/1801453320 shutdown_connections 2024-04-25T17:34:30.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25740a3cd0 0x7f25740a6190 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f258c106400 0x7f258c0fde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f258c101320 0x7f258c0fd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 --2- 172.21.15.98:0/1801453320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f258c101cd0 0x7f258c0fd8c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:30.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 -- 172.21.15.98:0/1801453320 >> 172.21.15.98:0/1801453320 conn(0x7f258c077f40 msgr2=0x7f258c0ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:30.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 -- 172.21.15.98:0/1801453320 shutdown_connections 2024-04-25T17:34:30.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:30.674+0000 7f2590fbd700 1 -- 172.21.15.98:0/1801453320 wait complete. 2024-04-25T17:34:31.382 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:31 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1801453320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T17:34:31.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:31 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1801453320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T17:34:31.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:31 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1801453320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T17:34:31.779 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037809 got 257698037813 for osd.8 2024-04-25T17:34:31.779 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:32.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.260+0000 7f6e3ba85700 1 -- 172.21.15.98:0/1683263799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e34101930 msgr2=0x7f6e34101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.260+0000 7f6e3ba85700 1 --2- 172.21.15.98:0/1683263799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e34101930 0x7f6e34101db0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e24009990 tx=0x7f6e24009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.260+0000 7f6e3ba85700 1 -- 172.21.15.98:0/1683263799 shutdown_connections 2024-04-25T17:34:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.260+0000 7f6e3ba85700 1 --2- 172.21.15.98:0/1683263799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e34107980 0x7f6e34107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.260+0000 7f6e3ba85700 1 --2- 172.21.15.98:0/1683263799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e341022f0 0x7f6e3410e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.260+0000 7f6e3ba85700 1 --2- 172.21.15.98:0/1683263799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e34101930 0x7f6e34101db0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.260+0000 7f6e3ba85700 1 -- 172.21.15.98:0/1683263799 >> 172.21.15.98:0/1683263799 conn(0x7f6e340788b0 msgr2=0x7f6e34078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e3ba85700 1 -- 172.21.15.98:0/1683263799 shutdown_connections 2024-04-25T17:34:32.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e3ba85700 1 -- 172.21.15.98:0/1683263799 wait complete. 2024-04-25T17:34:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e3ba85700 1 Processor -- start 2024-04-25T17:34:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e3ba85700 1 -- start start 2024-04-25T17:34:32.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e3ba85700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e34101930 0x7f6e341034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e3ba85700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e341022f0 0x7f6e34103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e3ba85700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e34107980 0x7f6e34103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e3ba85700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e34111b40 con 0x7f6e341022f0 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e3ba85700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e341119c0 con 0x7f6e34101930 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e3ba85700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e34111cc0 con 0x7f6e34107980 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.261+0000 7f6e39020700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e341022f0 0x7f6e34103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e39020700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e341022f0 0x7f6e34103a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56090/0 (socket says 172.21.15.98:56090) 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e39020700 1 -- 172.21.15.98:0/4066031365 learned_addr learned my addr 172.21.15.98:0/4066031365 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e3a022700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e34107980 0x7f6e34103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e39821700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e34101930 0x7f6e341034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e39020700 1 -- 172.21.15.98:0/4066031365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e34107980 msgr2=0x7f6e34103f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e39020700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e34107980 0x7f6e34103f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e39020700 1 -- 172.21.15.98:0/4066031365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e34101930 msgr2=0x7f6e341034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:32.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e39020700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e34101930 0x7f6e341034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e39020700 1 -- 172.21.15.98:0/4066031365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e341a76b0 con 0x7f6e341022f0 2024-04-25T17:34:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e39020700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e341022f0 0x7f6e34103a20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6e24005910 tx=0x7f6e2400b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e2affd700 1 -- 172.21.15.98:0/4066031365 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e24027070 con 0x7f6e341022f0 2024-04-25T17:34:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e2affd700 1 -- 172.21.15.98:0/4066031365 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e24022470 con 0x7f6e341022f0 2024-04-25T17:34:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e3ba85700 1 -- 172.21.15.98:0/4066031365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e341a7940 con 0x7f6e341022f0 2024-04-25T17:34:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e2affd700 1 -- 172.21.15.98:0/4066031365 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e24017650 con 0x7f6e341022f0 2024-04-25T17:34:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.262+0000 7f6e3ba85700 1 -- 172.21.15.98:0/4066031365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e341a7d90 con 0x7f6e341022f0 2024-04-25T17:34:32.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.263+0000 7f6e3ba85700 1 -- 172.21.15.98:0/4066031365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e34069730 con 0x7f6e341022f0 2024-04-25T17:34:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.264+0000 7f6e2affd700 1 -- 172.21.15.98:0/4066031365 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6e2402a070 con 0x7f6e341022f0 2024-04-25T17:34:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.265+0000 7f6e2affd700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e200a3cd0 0x7f6e200a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.265+0000 7f6e39821700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e200a3cd0 0x7f6e200a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.265+0000 7f6e2affd700 1 -- 172.21.15.98:0/4066031365 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f6e240d1220 con 0x7f6e341022f0 2024-04-25T17:34:32.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.265+0000 7f6e39821700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e200a3cd0 0x7f6e200a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009be0 tx=0x7f6e30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:32.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.269+0000 7f6e2affd700 1 -- 172.21.15.98:0/4066031365 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e24013070 con 0x7f6e341022f0 2024-04-25T17:34:32.399 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:32 smithi098 ceph-mon[29192]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:32.400 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:32 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:34:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:32 smithi203 ceph-mon[32834]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:32 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:34:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.478+0000 7f6e3ba85700 1 -- 172.21.15.98:0/4066031365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f6e3402e140 con 0x7f6e341022f0 2024-04-25T17:34:32.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.478+0000 7f6e2affd700 1 -- 172.21.15.98:0/4066031365 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6e2409db40 con 0x7f6e341022f0 2024-04-25T17:34:32.480 INFO:teuthology.orchestra.run.smithi098.stdout:223338299453 2024-04-25T17:34:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 -- 172.21.15.98:0/4066031365 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e200a3cd0 msgr2=0x7f6e200a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e200a3cd0 0x7f6e200a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009be0 tx=0x7f6e30009260 comp rx=0 tx=0).stop 2024-04-25T17:34:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 -- 172.21.15.98:0/4066031365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e341022f0 msgr2=0x7f6e34103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e341022f0 0x7f6e34103a20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6e24005910 tx=0x7f6e2400b670 comp rx=0 tx=0).stop 2024-04-25T17:34:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 -- 172.21.15.98:0/4066031365 shutdown_connections 2024-04-25T17:34:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f6e200a3cd0 0x7f6e200a6190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e34107980 0x7f6e34103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e34101930 0x7f6e341034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 --2- 172.21.15.98:0/4066031365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e341022f0 0x7f6e34103a20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 -- 172.21.15.98:0/4066031365 >> 172.21.15.98:0/4066031365 conn(0x7f6e340788b0 msgr2=0x7f6e3406b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 -- 172.21.15.98:0/4066031365 shutdown_connections 2024-04-25T17:34:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.482+0000 7f6e3ba85700 1 -- 172.21.15.98:0/4066031365 wait complete. 2024-04-25T17:34:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.508+0000 7f0c86fed700 1 -- 172.21.15.98:0/2935396167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c80102300 msgr2=0x7f0c8010e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.508+0000 7f0c86fed700 1 --2- 172.21.15.98:0/2935396167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c80102300 0x7f0c8010e7f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800b0a0 tx=0x7f0c7800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.508+0000 7f0c86fed700 1 -- 172.21.15.98:0/2935396167 shutdown_connections 2024-04-25T17:34:32.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.508+0000 7f0c86fed700 1 --2- 172.21.15.98:0/2935396167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c80102300 0x7f0c8010e7f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.508+0000 7f0c86fed700 1 --2- 172.21.15.98:0/2935396167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c80107990 0x7f0c80107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.508+0000 7f0c86fed700 1 --2- 172.21.15.98:0/2935396167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c80101940 0x7f0c80101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.508+0000 7f0c86fed700 1 -- 172.21.15.98:0/2935396167 >> 172.21.15.98:0/2935396167 conn(0x7f0c800788f0 msgr2=0x7f0c80078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.508+0000 7f0c86fed700 1 -- 172.21.15.98:0/2935396167 shutdown_connections 2024-04-25T17:34:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.508+0000 7f0c86fed700 1 -- 172.21.15.98:0/2935396167 wait complete. 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c86fed700 1 Processor -- start 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c86fed700 1 -- start start 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c86fed700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c80101940 0x7f0c80199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c86fed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c80102300 0x7f0c8019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c86fed700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c80107990 0x7f0c801a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c86fed700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c80111af0 con 0x7f0c80102300 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c86fed700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0c80111970 con 0x7f0c80107990 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c86fed700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0c80111c70 con 0x7f0c80101940 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c7ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c80102300 0x7f0c8019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c7ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c80102300 0x7f0c8019a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56108/0 (socket says 172.21.15.98:56108) 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c7ffff700 1 -- 172.21.15.98:0/2629469215 learned_addr learned my addr 172.21.15.98:0/2629469215 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.509+0000 7f0c84d89700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c80101940 0x7f0c80199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c8558a700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c80107990 0x7f0c801a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c7ffff700 1 -- 172.21.15.98:0/2629469215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c80101940 msgr2=0x7f0c80199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c7ffff700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c80101940 0x7f0c80199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c7ffff700 1 -- 172.21.15.98:0/2629469215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c80107990 msgr2=0x7f0c801a4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c7ffff700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c80107990 0x7f0c801a4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c7ffff700 1 -- 172.21.15.98:0/2629469215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c801a5490 con 0x7f0c80102300 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c7ffff700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c80102300 0x7f0c8019a410 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00d980 tx=0x7f0c6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c7dffb700 1 -- 172.21.15.98:0/2629469215 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c017070 con 0x7f0c80102300 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c86fed700 1 -- 172.21.15.98:0/2629469215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c801a5660 con 0x7f0c80102300 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c7dffb700 1 -- 172.21.15.98:0/2629469215 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c6c005020 con 0x7f0c80102300 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.510+0000 7f0c7dffb700 1 -- 172.21.15.98:0/2629469215 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c009b50 con 0x7f0c80102300 2024-04-25T17:34:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.511+0000 7f0c86fed700 1 -- 172.21.15.98:0/2629469215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c801a5b70 con 0x7f0c80102300 2024-04-25T17:34:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.512+0000 7f0c86fed700 1 -- 172.21.15.98:0/2629469215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c80069730 con 0x7f0c80102300 2024-04-25T17:34:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.513+0000 7f0c7dffb700 1 -- 172.21.15.98:0/2629469215 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0c6c0051d0 con 0x7f0c80102300 2024-04-25T17:34:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.513+0000 7f0c7dffb700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0c700a3d50 0x7f0c700a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.513+0000 7f0c7dffb700 1 -- 172.21.15.98:0/2629469215 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f0c6c0d0420 con 0x7f0c80102300 2024-04-25T17:34:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.513+0000 7f0c84d89700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0c700a3d50 0x7f0c700a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:32.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.514+0000 7f0c84d89700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0c700a3d50 0x7f0c700a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400abd0 tx=0x7f0c7400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:32.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.518+0000 7f0c7dffb700 1 -- 172.21.15.98:0/2629469215 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c6c020020 con 0x7f0c80102300 2024-04-25T17:34:32.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:32 smithi173 ceph-mon[32856]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:32.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:32 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:34:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.722+0000 7f0c86fed700 1 -- 172.21.15.98:0/2629469215 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0c8002df80 con 0x7f0c80102300 2024-04-25T17:34:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.722+0000 7f0c7dffb700 1 -- 172.21.15.98:0/2629469215 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0c6c09ccb0 con 0x7f0c80102300 2024-04-25T17:34:32.723 INFO:teuthology.orchestra.run.smithi098.stdout:167503724615 2024-04-25T17:34:32.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.724+0000 7f0c86fed700 1 -- 172.21.15.98:0/2629469215 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0c700a3d50 msgr2=0x7f0c700a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.724+0000 7f0c86fed700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0c700a3d50 0x7f0c700a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400abd0 tx=0x7f0c7400a380 comp rx=0 tx=0).stop 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.724+0000 7f0c86fed700 1 -- 172.21.15.98:0/2629469215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c80102300 msgr2=0x7f0c8019a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.724+0000 7f0c86fed700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c80102300 0x7f0c8019a410 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00d980 tx=0x7f0c6c00dc90 comp rx=0 tx=0).stop 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.724+0000 7f0c86fed700 1 -- 172.21.15.98:0/2629469215 shutdown_connections 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.724+0000 7f0c86fed700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0c700a3d50 0x7f0c700a6210 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.724+0000 7f0c86fed700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c80101940 0x7f0c80199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.724+0000 7f0c86fed700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c80107990 0x7f0c801a4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.724+0000 7f0c86fed700 1 --2- 172.21.15.98:0/2629469215 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c80102300 0x7f0c8019a410 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.724+0000 7f0c86fed700 1 -- 172.21.15.98:0/2629469215 >> 172.21.15.98:0/2629469215 conn(0x7f0c800788f0 msgr2=0x7f0c800ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.725+0000 7f0c86fed700 1 -- 172.21.15.98:0/2629469215 shutdown_connections 2024-04-25T17:34:32.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:32.725+0000 7f0c86fed700 1 -- 172.21.15.98:0/2629469215 wait complete. 2024-04-25T17:34:33.204 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299448 got 223338299453 for osd.7 2024-04-25T17:34:33.204 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:33.212 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:33 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/4066031365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T17:34:33.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:33 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2629469215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T17:34:33.351 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:33 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4066031365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T17:34:33.351 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:33 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2629469215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T17:34:33.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:33 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/4066031365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T17:34:33.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:33 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2629469215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T17:34:34.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:34 smithi098 ceph-mon[29192]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:34:34.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:34 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:34 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:34 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:34 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.294 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724609 got 167503724615 for osd.5 2024-04-25T17:34:34.294 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:34 smithi203 ceph-mon[32834]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:34:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:34 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:34 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:34 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:34 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:34 smithi173 ceph-mon[32856]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:34:34.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:34 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:34 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:34 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:34 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:34.708 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738457 got 34359738463 for osd.0 2024-04-25T17:34:34.708 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:35.246 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351007 got 347892351013 for osd.11 2024-04-25T17:34:35.246 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:35.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.469+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3764433360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9c4102a90 msgr2=0x7ff9c4102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:35.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.469+0000 7ff9cbf97700 1 --2- 172.21.15.98:0/3764433360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9c4102a90 0x7ff9c4102f10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009990 tx=0x7ff9c0009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.469+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3764433360 shutdown_connections 2024-04-25T17:34:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.469+0000 7ff9cbf97700 1 --2- 172.21.15.98:0/3764433360 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9c4108a50 0x7ff9c4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.469+0000 7ff9cbf97700 1 --2- 172.21.15.98:0/3764433360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9c4103450 0x7ff9c410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.469+0000 7ff9cbf97700 1 --2- 172.21.15.98:0/3764433360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9c4102a90 0x7ff9c4102f10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.469+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3764433360 >> 172.21.15.98:0/3764433360 conn(0x7ff9c4079ba0 msgr2=0x7ff9c4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.469+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3764433360 shutdown_connections 2024-04-25T17:34:35.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.469+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3764433360 wait complete. 2024-04-25T17:34:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.470+0000 7ff9cbf97700 1 Processor -- start 2024-04-25T17:34:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.470+0000 7ff9cbf97700 1 -- start start 2024-04-25T17:34:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.470+0000 7ff9cbf97700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9c4102a90 0x7ff9c41991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.470+0000 7ff9cbf97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9c4103450 0x7ff9c41996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.470+0000 7ff9cbf97700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9c4108a50 0x7ff9c419da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.470+0000 7ff9cbf97700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c4112be0 con 0x7ff9c4103450 2024-04-25T17:34:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.470+0000 7ff9cbf97700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c4112a60 con 0x7ff9c4102a90 2024-04-25T17:34:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.470+0000 7ff9cbf97700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c4112d60 con 0x7ff9c4108a50 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9c9532700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9c4103450 0x7ff9c41996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9c9532700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9c4103450 0x7ff9c41996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51794/0 (socket says 172.21.15.98:51794) 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9c9532700 1 -- 172.21.15.98:0/3741466186 learned_addr learned my addr 172.21.15.98:0/3741466186 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9c9d33700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9c4102a90 0x7ff9c41991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9ca534700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9c4108a50 0x7ff9c419da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9c9532700 1 -- 172.21.15.98:0/3741466186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9c4108a50 msgr2=0x7ff9c419da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9c9532700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9c4108a50 0x7ff9c419da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9c9532700 1 -- 172.21.15.98:0/3741466186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9c4102a90 msgr2=0x7ff9c41991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9c9532700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9c4102a90 0x7ff9c41991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9c9532700 1 -- 172.21.15.98:0/3741466186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c419e1c0 con 0x7ff9c4103450 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9c9532700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9c4103450 0x7ff9c41996f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0000c00 tx=0x7ff9c00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9baffd700 1 -- 172.21.15.98:0/3741466186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0027070 con 0x7ff9c4103450 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3741466186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9c419e450 con 0x7ff9c4103450 2024-04-25T17:34:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9baffd700 1 -- 172.21.15.98:0/3741466186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9c0003f20 con 0x7ff9c4103450 2024-04-25T17:34:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9baffd700 1 -- 172.21.15.98:0/3741466186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0022ca0 con 0x7ff9c4103450 2024-04-25T17:34:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.471+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3741466186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9c4074c70 con 0x7ff9c4103450 2024-04-25T17:34:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.473+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3741466186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9c4104180 con 0x7ff9c4103450 2024-04-25T17:34:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.473+0000 7ff9baffd700 1 -- 172.21.15.98:0/3741466186 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff9c002a070 con 0x7ff9c4103450 2024-04-25T17:34:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.474+0000 7ff9baffd700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff9b00a3cd0 0x7ff9b00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.474+0000 7ff9c9d33700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff9b00a3cd0 0x7ff9b00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.474+0000 7ff9baffd700 1 -- 172.21.15.98:0/3741466186 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7ff9c00d13e0 con 0x7ff9c4103450 2024-04-25T17:34:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.474+0000 7ff9c9d33700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff9b00a3cd0 0x7ff9b00a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4005950 tx=0x7ff9b400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:35.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.479+0000 7ff9baffd700 1 -- 172.21.15.98:0/3741466186 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff9c005eff0 con 0x7ff9c4103450 2024-04-25T17:34:35.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.685+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3741466186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff9c419a510 con 0x7ff9c4103450 2024-04-25T17:34:35.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.685+0000 7ff9baffd700 1 -- 172.21.15.98:0/3741466186 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff9c0013070 con 0x7ff9c4103450 2024-04-25T17:34:35.687 INFO:teuthology.orchestra.run.smithi098.stdout:146028888141 2024-04-25T17:34:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.687+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3741466186 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff9b00a3cd0 msgr2=0x7ff9b00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.687+0000 7ff9cbf97700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff9b00a3cd0 0x7ff9b00a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4005950 tx=0x7ff9b400b410 comp rx=0 tx=0).stop 2024-04-25T17:34:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.687+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3741466186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9c4103450 msgr2=0x7ff9c41996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.687+0000 7ff9cbf97700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9c4103450 0x7ff9c41996f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0000c00 tx=0x7ff9c00047b0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.688+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3741466186 shutdown_connections 2024-04-25T17:34:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.688+0000 7ff9cbf97700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff9b00a3cd0 0x7ff9b00a6190 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.688+0000 7ff9cbf97700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9c4108a50 0x7ff9c419da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.688+0000 7ff9cbf97700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9c4102a90 0x7ff9c41991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.688+0000 7ff9cbf97700 1 --2- 172.21.15.98:0/3741466186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9c4103450 0x7ff9c41996f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.688+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3741466186 >> 172.21.15.98:0/3741466186 conn(0x7ff9c4079ba0 msgr2=0x7ff9c4101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.688+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3741466186 shutdown_connections 2024-04-25T17:34:35.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:35.688+0000 7ff9cbf97700 1 -- 172.21.15.98:0/3741466186 wait complete. 2024-04-25T17:34:36.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:36 smithi098 ceph-mon[29192]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:36.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:36 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3741466186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T17:34:36.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:36 smithi203 ceph-mon[32834]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:36.466 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:36 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3741466186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T17:34:36.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:36 smithi173 ceph-mon[32856]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:34:36.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:36 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3741466186' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T17:34:36.623 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888135 got 146028888141 for osd.4 2024-04-25T17:34:36.623 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.687+0000 7f3f54a42700 1 -- 172.21.15.98:0/2417736024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f50101950 msgr2=0x7f3f5010f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.687+0000 7f3f54a42700 1 --2- 172.21.15.98:0/2417736024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f50101950 0x7f3f5010f920 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800b0d0 tx=0x7f3f4800b3e0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.687+0000 7f3f54a42700 1 -- 172.21.15.98:0/2417736024 shutdown_connections 2024-04-25T17:34:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.687+0000 7f3f54a42700 1 --2- 172.21.15.98:0/2417736024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50107a60 0x7f3f50107e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.687+0000 7f3f54a42700 1 --2- 172.21.15.98:0/2417736024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f50101950 0x7f3f5010f920 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.687+0000 7f3f54a42700 1 --2- 172.21.15.98:0/2417736024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f50100f90 0x7f3f50101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.687+0000 7f3f54a42700 1 -- 172.21.15.98:0/2417736024 >> 172.21.15.98:0/2417736024 conn(0x7f3f50077f40 msgr2=0x7f3f500ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.687+0000 7f3f54a42700 1 -- 172.21.15.98:0/2417736024 shutdown_connections 2024-04-25T17:34:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.687+0000 7f3f54a42700 1 -- 172.21.15.98:0/2417736024 wait complete. 2024-04-25T17:34:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f54a42700 1 Processor -- start 2024-04-25T17:34:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f54a42700 1 -- start start 2024-04-25T17:34:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f54a42700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100f90 0x7f3f501991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f54a42700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f50101950 0x7f3f501996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f54a42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f50107a60 0x7f3f5019da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f54a42700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3f50112d70 con 0x7f3f50107a60 2024-04-25T17:34:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f54a42700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3f50112bf0 con 0x7f3f50101950 2024-04-25T17:34:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f54a42700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3f50112ef0 con 0x7f3f50100f90 2024-04-25T17:34:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f4ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f50107a60 0x7f3f5019da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f4ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f50107a60 0x7f3f5019da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51812/0 (socket says 172.21.15.98:51812) 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f4ed9d700 1 -- 172.21.15.98:0/1757895925 learned_addr learned my addr 172.21.15.98:0/1757895925 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f4e59c700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100f90 0x7f3f501991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f45d9b700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f50101950 0x7f3f501996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f4ed9d700 1 -- 172.21.15.98:0/1757895925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100f90 msgr2=0x7f3f501991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f4ed9d700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100f90 0x7f3f501991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f4ed9d700 1 -- 172.21.15.98:0/1757895925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f50101950 msgr2=0x7f3f501996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f4ed9d700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f50101950 0x7f3f501996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.688+0000 7f3f4ed9d700 1 -- 172.21.15.98:0/1757895925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5019e1b0 con 0x7f3f50107a60 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.689+0000 7f3f4ed9d700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f50107a60 0x7f3f5019da70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3f480095c0 tx=0x7f3f48007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.689+0000 7f3f47fff700 1 -- 172.21.15.98:0/1757895925 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4801f070 con 0x7f3f50107a60 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.689+0000 7f3f47fff700 1 -- 172.21.15.98:0/1757895925 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f48003c00 con 0x7f3f50107a60 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.689+0000 7f3f54a42700 1 -- 172.21.15.98:0/1757895925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f5019e440 con 0x7f3f50107a60 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.689+0000 7f3f47fff700 1 -- 172.21.15.98:0/1757895925 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4801b730 con 0x7f3f50107a60 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.689+0000 7f3f54a42700 1 -- 172.21.15.98:0/1757895925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f501a5d20 con 0x7f3f50107a60 2024-04-25T17:34:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.690+0000 7f3f54a42700 1 -- 172.21.15.98:0/1757895925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f5010afe0 con 0x7f3f50107a60 2024-04-25T17:34:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.691+0000 7f3f47fff700 1 -- 172.21.15.98:0/1757895925 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3f48022070 con 0x7f3f50107a60 2024-04-25T17:34:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.692+0000 7f3f47fff700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3f3c0a3cd0 0x7f3f3c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.692+0000 7f3f4e59c700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3f3c0a3cd0 0x7f3f3c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.692+0000 7f3f47fff700 1 -- 172.21.15.98:0/1757895925 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f3f4800f020 con 0x7f3f50107a60 2024-04-25T17:34:37.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.692+0000 7f3f4e59c700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3f3c0a3cd0 0x7f3f3c0a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3f40006fd0 tx=0x7f3f40008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.695+0000 7f3f47fff700 1 -- 172.21.15.98:0/1757895925 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f48029020 con 0x7f3f50107a60 2024-04-25T17:34:37.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.897+0000 7f3f54a42700 1 -- 172.21.15.98:0/1757895925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f3f5019a500 con 0x7f3f50107a60 2024-04-25T17:34:37.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.897+0000 7f3f47fff700 1 -- 172.21.15.98:0/1757895925 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3f4809e4e0 con 0x7f3f50107a60 2024-04-25T17:34:37.903 INFO:teuthology.orchestra.run.smithi098.stdout:55834574939 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.899+0000 7f3f54a42700 1 -- 172.21.15.98:0/1757895925 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3f3c0a3cd0 msgr2=0x7f3f3c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.899+0000 7f3f54a42700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3f3c0a3cd0 0x7f3f3c0a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3f40006fd0 tx=0x7f3f40008040 comp rx=0 tx=0).stop 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.899+0000 7f3f54a42700 1 -- 172.21.15.98:0/1757895925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f50107a60 msgr2=0x7f3f5019da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.899+0000 7f3f54a42700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f50107a60 0x7f3f5019da70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3f480095c0 tx=0x7f3f48007ad0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.899+0000 7f3f54a42700 1 -- 172.21.15.98:0/1757895925 shutdown_connections 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.900+0000 7f3f54a42700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f3f3c0a3cd0 0x7f3f3c0a6190 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.900+0000 7f3f54a42700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f50100f90 0x7f3f501991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.900+0000 7f3f54a42700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f50107a60 0x7f3f5019da70 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.900+0000 7f3f54a42700 1 --2- 172.21.15.98:0/1757895925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f50101950 0x7f3f501996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.900+0000 7f3f54a42700 1 -- 172.21.15.98:0/1757895925 >> 172.21.15.98:0/1757895925 conn(0x7f3f50077f40 msgr2=0x7f3f50105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.900+0000 7f3f54a42700 1 -- 172.21.15.98:0/1757895925 shutdown_connections 2024-04-25T17:34:37.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:37.900+0000 7f3f54a42700 1 -- 172.21.15.98:0/1757895925 wait complete. 2024-04-25T17:34:38.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:38 smithi098 ceph-mon[29192]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:38.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:38 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1757895925' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T17:34:38.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:38 smithi203 ceph-mon[32834]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:38.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:38 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1757895925' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T17:34:38.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:38 smithi173 ceph-mon[32856]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:38.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:38 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1757895925' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T17:34:38.719 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574934 got 55834574939 for osd.1 2024-04-25T17:34:38.719 DEBUG:teuthology.parallel:result is None 2024-04-25T17:34:38.719 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T17:34:38.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T17:34:39.103 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:40.367 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:40 smithi098 ceph-mon[29192]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:40.367 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:40 smithi098 ceph-mon[29192]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T17:34:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:40 smithi203 ceph-mon[32834]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:40.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:40 smithi203 ceph-mon[32834]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T17:34:40.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:40 smithi173 ceph-mon[32856]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:34:40.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:40 smithi173 ceph-mon[32856]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T17:34:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.912+0000 7f1255860700 1 -- 172.21.15.98:0/1153748004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1250102af0 msgr2=0x7f1250102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.912+0000 7f1255860700 1 --2- 172.21.15.98:0/1153748004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1250102af0 0x7f1250102f70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f12440099c0 tx=0x7f1244009cd0 comp rx=0 tx=0).stop 2024-04-25T17:34:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.912+0000 7f1255860700 1 -- 172.21.15.98:0/1153748004 shutdown_connections 2024-04-25T17:34:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.912+0000 7f1255860700 1 --2- 172.21.15.98:0/1153748004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12501034b0 0x7f125010f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.912+0000 7f1255860700 1 --2- 172.21.15.98:0/1153748004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1250107940 0x7f1250107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.912+0000 7f1255860700 1 --2- 172.21.15.98:0/1153748004 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1250102af0 0x7f1250102f70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.912+0000 7f1255860700 1 -- 172.21.15.98:0/1153748004 >> 172.21.15.98:0/1153748004 conn(0x7f1250079ba0 msgr2=0x7f1250079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.912+0000 7f1255860700 1 -- 172.21.15.98:0/1153748004 shutdown_connections 2024-04-25T17:34:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.912+0000 7f1255860700 1 -- 172.21.15.98:0/1153748004 wait complete. 2024-04-25T17:34:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.912+0000 7f1255860700 1 Processor -- start 2024-04-25T17:34:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f1255860700 1 -- start start 2024-04-25T17:34:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f1255860700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1250102af0 0x7f1250074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f1255860700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12501034b0 0x7f1250075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f1255860700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1250107940 0x7f12500757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f1255860700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1250112c30 con 0x7f1250102af0 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f1255860700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1250112ab0 con 0x7f1250107940 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f1255860700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1250112db0 con 0x7f12501034b0 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f124effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1250102af0 0x7f1250074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f124effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1250102af0 0x7f1250074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51836/0 (socket says 172.21.15.98:51836) 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f124effd700 1 -- 172.21.15.98:0/4081174037 learned_addr learned my addr 172.21.15.98:0/4081174037 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f124f7fe700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1250107940 0x7f12500757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f124e7fc700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12501034b0 0x7f1250075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f124effd700 1 -- 172.21.15.98:0/4081174037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12501034b0 msgr2=0x7f1250075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f124effd700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12501034b0 0x7f1250075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:40.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f124effd700 1 -- 172.21.15.98:0/4081174037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1250107940 msgr2=0x7f12500757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f124effd700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1250107940 0x7f12500757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.913+0000 7f124effd700 1 -- 172.21.15.98:0/4081174037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12501a7620 con 0x7f1250102af0 2024-04-25T17:34:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.914+0000 7f124effd700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1250102af0 0x7f1250074d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f124000d950 tx=0x7f124000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.914+0000 7f125485e700 1 -- 172.21.15.98:0/4081174037 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1240016070 con 0x7f1250102af0 2024-04-25T17:34:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.914+0000 7f1255860700 1 -- 172.21.15.98:0/4081174037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12501a7910 con 0x7f1250102af0 2024-04-25T17:34:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.914+0000 7f125485e700 1 -- 172.21.15.98:0/4081174037 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f124000fd40 con 0x7f1250102af0 2024-04-25T17:34:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.914+0000 7f125485e700 1 -- 172.21.15.98:0/4081174037 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124000e3b0 con 0x7f1250102af0 2024-04-25T17:34:40.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.914+0000 7f1255860700 1 -- 172.21.15.98:0/4081174037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12501a7e20 con 0x7f1250102af0 2024-04-25T17:34:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.915+0000 7f1255860700 1 -- 172.21.15.98:0/4081174037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1250069730 con 0x7f1250102af0 2024-04-25T17:34:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.916+0000 7f125485e700 1 -- 172.21.15.98:0/4081174037 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1240019070 con 0x7f1250102af0 2024-04-25T17:34:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.916+0000 7f125485e700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f12380a3c40 0x7f12380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.917+0000 7f124e7fc700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f12380a3c40 0x7f12380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.917+0000 7f125485e700 1 -- 172.21.15.98:0/4081174037 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f12400cfdf0 con 0x7f1250102af0 2024-04-25T17:34:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.917+0000 7f124e7fc700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f12380a3c40 0x7f12380a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1244009990 tx=0x7f1244009650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:40.921+0000 7f125485e700 1 -- 172.21.15.98:0/4081174037 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f124009c7e0 con 0x7f1250102af0 2024-04-25T17:34:41.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.134+0000 7f1255860700 1 -- 172.21.15.98:0/4081174037 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1250062e80 con 0x7f12380a3c40 2024-04-25T17:34:41.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.149+0000 7f125485e700 1 -- 172.21.15.98:0/4081174037 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258967 (secure 0 0 0) 0x7f1250062e80 con 0x7f12380a3c40 2024-04-25T17:34:41.154 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:34:41.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 -- 172.21.15.98:0/4081174037 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f12380a3c40 msgr2=0x7f12380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:41.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f12380a3c40 0x7f12380a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1244009990 tx=0x7f1244009650 comp rx=0 tx=0).stop 2024-04-25T17:34:41.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 -- 172.21.15.98:0/4081174037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1250102af0 msgr2=0x7f1250074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:41.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1250102af0 0x7f1250074d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f124000d950 tx=0x7f124000dc60 comp rx=0 tx=0).stop 2024-04-25T17:34:41.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 -- 172.21.15.98:0/4081174037 shutdown_connections 2024-04-25T17:34:41.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f12380a3c40 0x7f12380a6100 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:41.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1250107940 0x7f12500757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:41.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1250102af0 0x7f1250074d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:41.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 --2- 172.21.15.98:0/4081174037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f12501034b0 0x7f1250075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:41.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 -- 172.21.15.98:0/4081174037 >> 172.21.15.98:0/4081174037 conn(0x7f1250079ba0 msgr2=0x7f1250100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:41.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 -- 172.21.15.98:0/4081174037 shutdown_connections 2024-04-25T17:34:41.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:41.152+0000 7f1255860700 1 -- 172.21.15.98:0/4081174037 wait complete. 2024-04-25T17:34:41.157 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T17:34:41.860 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-04-25T17:34:40.996680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656938,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":774,"num_read_kb":508,"num_write":349,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":410,"ondisk_log_size":410,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357136,"kb_used_data":12108,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584688,"statfs":{"total":1151940427776,"available":1151574720512,"internally_reserved":0,"allocated":12398592,"data_stored":7759059,"data_compressed":106941,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003284"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.997821+0000","last_change":"2024-04-25T17:32:35.075338+0000","last_active":"2024-04-25T17:34:38.997821+0000","last_peered":"2024-04-25T17:34:38.997821+0000","last_clean":"2024-04-25T17:34:38.997821+0000","last_became_active":"2024-04-25T17:32:35.075029+0000","last_became_peered":"2024-04-25T17:32:35.075029+0000","last_unstale":"2024-04-25T17:34:38.997821+0000","last_undegraded":"2024-04-25T17:34:38.997821+0000","last_fullsized":"2024-04-25T17:34:38.997821+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:23.837337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647523+0000","last_change":"2024-04-25T17:32:35.075422+0000","last_active":"2024-04-25T17:33:10.647523+0000","last_peered":"2024-04-25T17:33:10.647523+0000","last_clean":"2024-04-25T17:33:10.647523+0000","last_became_active":"2024-04-25T17:32:35.071975+0000","last_became_peered":"2024-04-25T17:32:35.071975+0000","last_unstale":"2024-04-25T17:33:10.647523+0000","last_undegraded":"2024-04-25T17:33:10.647523+0000","last_fullsized":"2024-04-25T17:33:10.647523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:20.548869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647315+0000","last_change":"2024-04-25T17:32:35.068153+0000","last_active":"2024-04-25T17:33:10.647315+0000","last_peered":"2024-04-25T17:33:10.647315+0000","last_clean":"2024-04-25T17:33:10.647315+0000","last_became_active":"2024-04-25T17:32:35.065873+0000","last_became_peered":"2024-04-25T17:32:35.065873+0000","last_unstale":"2024-04-25T17:33:10.647315+0000","last_undegraded":"2024-04-25T17:33:10.647315+0000","last_fullsized":"2024-04-25T17:33:10.647315+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:31.381875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605690+0000","last_change":"2024-04-25T17:32:35.073055+0000","last_active":"2024-04-25T17:33:10.605690+0000","last_peered":"2024-04-25T17:33:10.605690+0000","last_clean":"2024-04-25T17:33:10.605690+0000","last_became_active":"2024-04-25T17:32:35.072596+0000","last_became_peered":"2024-04-25T17:32:35.072596+0000","last_unstale":"2024-04-25T17:33:10.605690+0000","last_undegraded":"2024-04-25T17:33:10.605690+0000","last_fullsized":"2024-04-25T17:33:10.605690+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:37.529698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.978679+0000","last_change":"2024-04-25T17:32:35.072721+0000","last_active":"2024-04-25T17:34:38.978679+0000","last_peered":"2024-04-25T17:34:38.978679+0000","last_clean":"2024-04-25T17:34:38.978679+0000","last_became_active":"2024-04-25T17:32:35.072370+0000","last_became_peered":"2024-04-25T17:32:35.072370+0000","last_unstale":"2024-04-25T17:34:38.978679+0000","last_undegraded":"2024-04-25T17:34:38.978679+0000","last_fullsized":"2024-04-25T17:34:38.978679+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:34.429985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850042+0000","last_change":"2024-04-25T17:32:35.071538+0000","last_active":"2024-04-25T17:33:10.850042+0000","last_peered":"2024-04-25T17:33:10.850042+0000","last_clean":"2024-04-25T17:33:10.850042+0000","last_became_active":"2024-04-25T17:32:35.067427+0000","last_became_peered":"2024-04-25T17:32:35.067427+0000","last_unstale":"2024-04-25T17:33:10.850042+0000","last_undegraded":"2024-04-25T17:33:10.850042+0000","last_fullsized":"2024-04-25T17:33:10.850042+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:36.052762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.992163+0000","last_change":"2024-04-25T17:32:35.079476+0000","last_active":"2024-04-25T17:34:38.992163+0000","last_peered":"2024-04-25T17:34:38.992163+0000","last_clean":"2024-04-25T17:34:38.992163+0000","last_became_active":"2024-04-25T17:32:35.079170+0000","last_became_peered":"2024-04-25T17:32:35.079170+0000","last_unstale":"2024-04-25T17:34:38.992163+0000","last_undegraded":"2024-04-25T17:34:38.992163+0000","last_fullsized":"2024-04-25T17:34:38.992163+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:30.937670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189780+0000","last_change":"2024-04-25T17:32:35.077261+0000","last_active":"2024-04-25T17:33:10.189780+0000","last_peered":"2024-04-25T17:33:10.189780+0000","last_clean":"2024-04-25T17:33:10.189780+0000","last_became_active":"2024-04-25T17:32:35.076896+0000","last_became_peered":"2024-04-25T17:32:35.076896+0000","last_unstale":"2024-04-25T17:33:10.189780+0000","last_undegraded":"2024-04-25T17:33:10.189780+0000","last_fullsized":"2024-04-25T17:33:10.189780+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:24.082550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187931+0000","last_change":"2024-04-25T17:32:35.072229+0000","last_active":"2024-04-25T17:33:10.187931+0000","last_peered":"2024-04-25T17:33:10.187931+0000","last_clean":"2024-04-25T17:33:10.187931+0000","last_became_active":"2024-04-25T17:32:35.071586+0000","last_became_peered":"2024-04-25T17:32:35.071586+0000","last_unstale":"2024-04-25T17:33:10.187931+0000","last_undegraded":"2024-04-25T17:33:10.187931+0000","last_fullsized":"2024-04-25T17:33:10.187931+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:01.195638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850464+0000","last_change":"2024-04-25T17:32:35.075274+0000","last_active":"2024-04-25T17:33:10.850464+0000","last_peered":"2024-04-25T17:33:10.850464+0000","last_clean":"2024-04-25T17:33:10.850464+0000","last_became_active":"2024-04-25T17:32:35.073821+0000","last_became_peered":"2024-04-25T17:32:35.073821+0000","last_unstale":"2024-04-25T17:33:10.850464+0000","last_undegraded":"2024-04-25T17:33:10.850464+0000","last_fullsized":"2024-04-25T17:33:10.850464+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:43:04.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.995557+0000","last_change":"2024-04-25T17:32:35.080747+0000","last_active":"2024-04-25T17:34:38.995557+0000","last_peered":"2024-04-25T17:34:38.995557+0000","last_clean":"2024-04-25T17:34:38.995557+0000","last_became_active":"2024-04-25T17:32:35.080383+0000","last_became_peered":"2024-04-25T17:32:35.080383+0000","last_unstale":"2024-04-25T17:34:38.995557+0000","last_undegraded":"2024-04-25T17:34:38.995557+0000","last_fullsized":"2024-04-25T17:34:38.995557+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:42.128892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605662+0000","last_change":"2024-04-25T17:32:35.061442+0000","last_active":"2024-04-25T17:33:10.605662+0000","last_peered":"2024-04-25T17:33:10.605662+0000","last_clean":"2024-04-25T17:33:10.605662+0000","last_became_active":"2024-04-25T17:32:35.061030+0000","last_became_peered":"2024-04-25T17:32:35.061030+0000","last_unstale":"2024-04-25T17:33:10.605662+0000","last_undegraded":"2024-04-25T17:33:10.605662+0000","last_fullsized":"2024-04-25T17:33:10.605662+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:03.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187927+0000","last_change":"2024-04-25T17:32:35.069698+0000","last_active":"2024-04-25T17:33:10.187927+0000","last_peered":"2024-04-25T17:33:10.187927+0000","last_clean":"2024-04-25T17:33:10.187927+0000","last_became_active":"2024-04-25T17:32:35.069374+0000","last_became_peered":"2024-04-25T17:32:35.069374+0000","last_unstale":"2024-04-25T17:33:10.187927+0000","last_undegraded":"2024-04-25T17:33:10.187927+0000","last_fullsized":"2024-04-25T17:33:10.187927+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:03.214525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647126+0000","last_change":"2024-04-25T17:32:35.069001+0000","last_active":"2024-04-25T17:33:10.647126+0000","last_peered":"2024-04-25T17:33:10.647126+0000","last_clean":"2024-04-25T17:33:10.647126+0000","last_became_active":"2024-04-25T17:32:35.064882+0000","last_became_peered":"2024-04-25T17:32:35.064882+0000","last_unstale":"2024-04-25T17:33:10.647126+0000","last_undegraded":"2024-04-25T17:33:10.647126+0000","last_fullsized":"2024-04-25T17:33:10.647126+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:32.122251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647097+0000","last_change":"2024-04-25T17:32:35.065042+0000","last_active":"2024-04-25T17:33:10.647097+0000","last_peered":"2024-04-25T17:33:10.647097+0000","last_clean":"2024-04-25T17:33:10.647097+0000","last_became_active":"2024-04-25T17:32:35.064719+0000","last_became_peered":"2024-04-25T17:32:35.064719+0000","last_unstale":"2024-04-25T17:33:10.647097+0000","last_undegraded":"2024-04-25T17:33:10.647097+0000","last_fullsized":"2024-04-25T17:33:10.647097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:54.715385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187383+0000","last_change":"2024-04-25T17:32:35.067390+0000","last_active":"2024-04-25T17:33:10.187383+0000","last_peered":"2024-04-25T17:33:10.187383+0000","last_clean":"2024-04-25T17:33:10.187383+0000","last_became_active":"2024-04-25T17:32:35.066846+0000","last_became_peered":"2024-04-25T17:32:35.066846+0000","last_unstale":"2024-04-25T17:33:10.187383+0000","last_undegraded":"2024-04-25T17:33:10.187383+0000","last_fullsized":"2024-04-25T17:33:10.187383+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:24.305767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.999597+0000","last_change":"2024-04-25T17:32:35.075349+0000","last_active":"2024-04-25T17:34:38.999597+0000","last_peered":"2024-04-25T17:34:38.999597+0000","last_clean":"2024-04-25T17:34:38.999597+0000","last_became_active":"2024-04-25T17:32:35.075046+0000","last_became_peered":"2024-04-25T17:32:35.075046+0000","last_unstale":"2024-04-25T17:34:38.999597+0000","last_undegraded":"2024-04-25T17:34:38.999597+0000","last_fullsized":"2024-04-25T17:34:38.999597+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:51.703406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185881+0000","last_change":"2024-04-25T17:32:35.077597+0000","last_active":"2024-04-25T17:33:10.185881+0000","last_peered":"2024-04-25T17:33:10.185881+0000","last_clean":"2024-04-25T17:33:10.185881+0000","last_became_active":"2024-04-25T17:32:35.077232+0000","last_became_peered":"2024-04-25T17:32:35.077232+0000","last_unstale":"2024-04-25T17:33:10.185881+0000","last_undegraded":"2024-04-25T17:33:10.185881+0000","last_fullsized":"2024-04-25T17:33:10.185881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:46.842673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.188192+0000","last_change":"2024-04-25T17:32:35.078229+0000","last_active":"2024-04-25T17:33:10.188192+0000","last_peered":"2024-04-25T17:33:10.188192+0000","last_clean":"2024-04-25T17:33:10.188192+0000","last_became_active":"2024-04-25T17:32:35.077899+0000","last_became_peered":"2024-04-25T17:32:35.077899+0000","last_unstale":"2024-04-25T17:33:10.188192+0000","last_undegraded":"2024-04-25T17:33:10.188192+0000","last_fullsized":"2024-04-25T17:33:10.188192+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:22.776371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185950+0000","last_change":"2024-04-25T17:32:32.045385+0000","last_active":"2024-04-25T17:33:10.185950+0000","last_peered":"2024-04-25T17:33:10.185950+0000","last_clean":"2024-04-25T17:33:10.185950+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:33:10.185950+0000","last_undegraded":"2024-04-25T17:33:10.185950+0000","last_fullsized":"2024-04-25T17:33:10.185950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:29.732001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190580+0000","last_change":"2024-04-25T17:32:35.079966+0000","last_active":"2024-04-25T17:33:10.190580+0000","last_peered":"2024-04-25T17:33:10.190580+0000","last_clean":"2024-04-25T17:33:10.190580+0000","last_became_active":"2024-04-25T17:32:35.079649+0000","last_became_peered":"2024-04-25T17:32:35.079649+0000","last_unstale":"2024-04-25T17:33:10.190580+0000","last_undegraded":"2024-04-25T17:33:10.190580+0000","last_fullsized":"2024-04-25T17:33:10.190580+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:19:45.679273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190761+0000","last_change":"2024-04-25T17:32:32.285667+0000","last_active":"2024-04-25T17:33:10.190761+0000","last_peered":"2024-04-25T17:33:10.190761+0000","last_clean":"2024-04-25T17:33:10.190761+0000","last_became_active":"2024-04-25T17:32:32.285330+0000","last_became_peered":"2024-04-25T17:32:32.285330+0000","last_unstale":"2024-04-25T17:33:10.190761+0000","last_undegraded":"2024-04-25T17:33:10.190761+0000","last_fullsized":"2024-04-25T17:33:10.190761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.975280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.686361+0000","last_change":"2024-04-25T17:32:35.068841+0000","last_active":"2024-04-25T17:33:10.686361+0000","last_peered":"2024-04-25T17:33:10.686361+0000","last_clean":"2024-04-25T17:33:10.686361+0000","last_became_active":"2024-04-25T17:32:35.068518+0000","last_became_peered":"2024-04-25T17:32:35.068518+0000","last_unstale":"2024-04-25T17:33:10.686361+0000","last_undegraded":"2024-04-25T17:33:10.686361+0000","last_fullsized":"2024-04-25T17:33:10.686361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:23.290990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195797+0000","last_change":"2024-04-25T17:32:32.045627+0000","last_active":"2024-04-25T17:33:10.195797+0000","last_peered":"2024-04-25T17:33:10.195797+0000","last_clean":"2024-04-25T17:33:10.195797+0000","last_became_active":"2024-04-25T17:32:32.044971+0000","last_became_peered":"2024-04-25T17:32:32.044971+0000","last_unstale":"2024-04-25T17:33:10.195797+0000","last_undegraded":"2024-04-25T17:33:10.195797+0000","last_fullsized":"2024-04-25T17:33:10.195797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:50.590435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187551+0000","last_change":"2024-04-25T17:32:35.070862+0000","last_active":"2024-04-25T17:33:10.187551+0000","last_peered":"2024-04-25T17:33:10.187551+0000","last_clean":"2024-04-25T17:33:10.187551+0000","last_became_active":"2024-04-25T17:32:35.069927+0000","last_became_peered":"2024-04-25T17:32:35.069927+0000","last_unstale":"2024-04-25T17:33:10.187551+0000","last_undegraded":"2024-04-25T17:33:10.187551+0000","last_fullsized":"2024-04-25T17:33:10.187551+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:40:01.799015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195097+0000","last_change":"2024-04-25T17:32:32.046062+0000","last_active":"2024-04-25T17:33:10.195097+0000","last_peered":"2024-04-25T17:33:10.195097+0000","last_clean":"2024-04-25T17:33:10.195097+0000","last_became_active":"2024-04-25T17:32:32.044903+0000","last_became_peered":"2024-04-25T17:32:32.044903+0000","last_unstale":"2024-04-25T17:33:10.195097+0000","last_undegraded":"2024-04-25T17:33:10.195097+0000","last_fullsized":"2024-04-25T17:33:10.195097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:51.099460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193396+0000","last_change":"2024-04-25T17:32:35.075452+0000","last_active":"2024-04-25T17:33:10.193396+0000","last_peered":"2024-04-25T17:33:10.193396+0000","last_clean":"2024-04-25T17:33:10.193396+0000","last_became_active":"2024-04-25T17:32:35.075136+0000","last_became_peered":"2024-04-25T17:32:35.075136+0000","last_unstale":"2024-04-25T17:33:10.193396+0000","last_undegraded":"2024-04-25T17:33:10.193396+0000","last_fullsized":"2024-04-25T17:33:10.193396+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:02.290926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":134,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:35.297047+0000","last_change":"2024-04-25T17:32:32.042019+0000","last_active":"2024-04-25T17:34:35.297047+0000","last_peered":"2024-04-25T17:34:35.297047+0000","last_clean":"2024-04-25T17:34:35.297047+0000","last_became_active":"2024-04-25T17:32:32.041641+0000","last_became_peered":"2024-04-25T17:32:32.041641+0000","last_unstale":"2024-04-25T17:34:35.297047+0000","last_undegraded":"2024-04-25T17:34:35.297047+0000","last_fullsized":"2024-04-25T17:34:35.297047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:02.025915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.998149+0000","last_change":"2024-04-25T17:32:35.069715+0000","last_active":"2024-04-25T17:34:38.998149+0000","last_peered":"2024-04-25T17:34:38.998149+0000","last_clean":"2024-04-25T17:34:38.998149+0000","last_became_active":"2024-04-25T17:32:35.069339+0000","last_became_peered":"2024-04-25T17:32:35.069339+0000","last_unstale":"2024-04-25T17:34:38.998149+0000","last_undegraded":"2024-04-25T17:34:38.998149+0000","last_fullsized":"2024-04-25T17:34:38.998149+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:48:13.572261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.849969+0000","last_change":"2024-04-25T17:32:32.042445+0000","last_active":"2024-04-25T17:33:10.849969+0000","last_peered":"2024-04-25T17:33:10.849969+0000","last_clean":"2024-04-25T17:33:10.849969+0000","last_became_active":"2024-04-25T17:32:32.042033+0000","last_became_peered":"2024-04-25T17:32:32.042033+0000","last_unstale":"2024-04-25T17:33:10.849969+0000","last_undegraded":"2024-04-25T17:33:10.849969+0000","last_fullsized":"2024-04-25T17:33:10.849969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.968383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193657+0000","last_change":"2024-04-25T17:32:35.074492+0000","last_active":"2024-04-25T17:33:10.193657+0000","last_peered":"2024-04-25T17:33:10.193657+0000","last_clean":"2024-04-25T17:33:10.193657+0000","last_became_active":"2024-04-25T17:32:35.073652+0000","last_became_peered":"2024-04-25T17:32:35.073652+0000","last_unstale":"2024-04-25T17:33:10.193657+0000","last_undegraded":"2024-04-25T17:33:10.193657+0000","last_fullsized":"2024-04-25T17:33:10.193657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:00.300453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187848+0000","last_change":"2024-04-25T17:32:32.039568+0000","last_active":"2024-04-25T17:33:10.187848+0000","last_peered":"2024-04-25T17:33:10.187848+0000","last_clean":"2024-04-25T17:33:10.187848+0000","last_became_active":"2024-04-25T17:32:32.039228+0000","last_became_peered":"2024-04-25T17:32:32.039228+0000","last_unstale":"2024-04-25T17:33:10.187848+0000","last_undegraded":"2024-04-25T17:33:10.187848+0000","last_fullsized":"2024-04-25T17:33:10.187848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:17:29.079749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189877+0000","last_change":"2024-04-25T17:32:35.077619+0000","last_active":"2024-04-25T17:33:10.189877+0000","last_peered":"2024-04-25T17:33:10.189877+0000","last_clean":"2024-04-25T17:33:10.189877+0000","last_became_active":"2024-04-25T17:32:35.077309+0000","last_became_peered":"2024-04-25T17:32:35.077309+0000","last_unstale":"2024-04-25T17:33:10.189877+0000","last_undegraded":"2024-04-25T17:33:10.189877+0000","last_fullsized":"2024-04-25T17:33:10.189877+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:41.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193055+0000","last_change":"2024-04-25T17:32:32.044650+0000","last_active":"2024-04-25T17:33:10.193055+0000","last_peered":"2024-04-25T17:33:10.193055+0000","last_clean":"2024-04-25T17:33:10.193055+0000","last_became_active":"2024-04-25T17:32:32.043999+0000","last_became_peered":"2024-04-25T17:32:32.043999+0000","last_unstale":"2024-04-25T17:33:10.193055+0000","last_undegraded":"2024-04-25T17:33:10.193055+0000","last_fullsized":"2024-04-25T17:33:10.193055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:19.463941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185738+0000","last_change":"2024-04-25T17:32:35.080598+0000","last_active":"2024-04-25T17:33:10.185738+0000","last_peered":"2024-04-25T17:33:10.185738+0000","last_clean":"2024-04-25T17:33:10.185738+0000","last_became_active":"2024-04-25T17:32:35.080300+0000","last_became_peered":"2024-04-25T17:32:35.080300+0000","last_unstale":"2024-04-25T17:33:10.185738+0000","last_undegraded":"2024-04-25T17:33:10.185738+0000","last_fullsized":"2024-04-25T17:33:10.185738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:42.149746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":131,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:36.297980+0000","last_change":"2024-04-25T17:32:32.043889+0000","last_active":"2024-04-25T17:34:36.297980+0000","last_peered":"2024-04-25T17:34:36.297980+0000","last_clean":"2024-04-25T17:34:36.297980+0000","last_became_active":"2024-04-25T17:32:32.043520+0000","last_became_peered":"2024-04-25T17:32:32.043520+0000","last_unstale":"2024-04-25T17:34:36.297980+0000","last_undegraded":"2024-04-25T17:34:36.297980+0000","last_fullsized":"2024-04-25T17:34:36.297980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:25.134797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187839+0000","last_change":"2024-04-25T17:32:35.071733+0000","last_active":"2024-04-25T17:33:10.187839+0000","last_peered":"2024-04-25T17:33:10.187839+0000","last_clean":"2024-04-25T17:33:10.187839+0000","last_became_active":"2024-04-25T17:32:35.071383+0000","last_became_peered":"2024-04-25T17:32:35.071383+0000","last_unstale":"2024-04-25T17:33:10.187839+0000","last_undegraded":"2024-04-25T17:33:10.187839+0000","last_fullsized":"2024-04-25T17:33:10.187839+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:09:49.433675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190101+0000","last_change":"2024-04-25T17:32:32.044823+0000","last_active":"2024-04-25T17:33:10.190101+0000","last_peered":"2024-04-25T17:33:10.190101+0000","last_clean":"2024-04-25T17:33:10.190101+0000","last_became_active":"2024-04-25T17:32:32.044497+0000","last_became_peered":"2024-04-25T17:32:32.044497+0000","last_unstale":"2024-04-25T17:33:10.190101+0000","last_undegraded":"2024-04-25T17:33:10.190101+0000","last_fullsized":"2024-04-25T17:33:10.190101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:47.811172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.686285+0000","last_change":"2024-04-25T17:32:35.071109+0000","last_active":"2024-04-25T17:33:10.686285+0000","last_peered":"2024-04-25T17:33:10.686285+0000","last_clean":"2024-04-25T17:33:10.686285+0000","last_became_active":"2024-04-25T17:32:35.070782+0000","last_became_peered":"2024-04-25T17:32:35.070782+0000","last_unstale":"2024-04-25T17:33:10.686285+0000","last_undegraded":"2024-04-25T17:33:10.686285+0000","last_fullsized":"2024-04-25T17:33:10.686285+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:37.247274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193437+0000","last_change":"2024-04-25T17:32:32.288683+0000","last_active":"2024-04-25T17:33:10.193437+0000","last_peered":"2024-04-25T17:33:10.193437+0000","last_clean":"2024-04-25T17:33:10.193437+0000","last_became_active":"2024-04-25T17:32:32.288261+0000","last_became_peered":"2024-04-25T17:32:32.288261+0000","last_unstale":"2024-04-25T17:33:10.193437+0000","last_undegraded":"2024-04-25T17:33:10.193437+0000","last_fullsized":"2024-04-25T17:33:10.193437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:36.502434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187579+0000","last_change":"2024-04-25T17:32:35.060828+0000","last_active":"2024-04-25T17:33:10.187579+0000","last_peered":"2024-04-25T17:33:10.187579+0000","last_clean":"2024-04-25T17:33:10.187579+0000","last_became_active":"2024-04-25T17:32:35.060433+0000","last_became_peered":"2024-04-25T17:32:35.060433+0000","last_unstale":"2024-04-25T17:33:10.187579+0000","last_undegraded":"2024-04-25T17:33:10.187579+0000","last_fullsized":"2024-04-25T17:33:10.187579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:32.382042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'29","reported_seq":269,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:37.299378+0000","last_change":"2024-04-25T17:32:32.042979+0000","last_active":"2024-04-25T17:34:37.299378+0000","last_peered":"2024-04-25T17:34:37.299378+0000","last_clean":"2024-04-25T17:34:37.299378+0000","last_became_active":"2024-04-25T17:32:32.042257+0000","last_became_peered":"2024-04-25T17:32:32.042257+0000","last_unstale":"2024-04-25T17:34:37.299378+0000","last_undegraded":"2024-04-25T17:34:37.299378+0000","last_fullsized":"2024-04-25T17:34:37.299378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:19.316803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62335,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":214,"num_read_kb":168,"num_write":30,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.985081+0000","last_change":"2024-04-25T17:32:35.069994+0000","last_active":"2024-04-25T17:34:38.985081+0000","last_peered":"2024-04-25T17:34:38.985081+0000","last_clean":"2024-04-25T17:34:38.985081+0000","last_became_active":"2024-04-25T17:32:35.069632+0000","last_became_peered":"2024-04-25T17:32:35.069632+0000","last_unstale":"2024-04-25T17:34:38.985081+0000","last_undegraded":"2024-04-25T17:34:38.985081+0000","last_fullsized":"2024-04-25T17:34:38.985081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:51.169489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187904+0000","last_change":"2024-04-25T17:32:32.042652+0000","last_active":"2024-04-25T17:33:10.187904+0000","last_peered":"2024-04-25T17:33:10.187904+0000","last_clean":"2024-04-25T17:33:10.187904+0000","last_became_active":"2024-04-25T17:32:32.042214+0000","last_became_peered":"2024-04-25T17:32:32.042214+0000","last_unstale":"2024-04-25T17:33:10.187904+0000","last_undegraded":"2024-04-25T17:33:10.187904+0000","last_fullsized":"2024-04-25T17:33:10.187904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.662580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187756+0000","last_change":"2024-04-25T17:32:35.074449+0000","last_active":"2024-04-25T17:33:10.187756+0000","last_peered":"2024-04-25T17:33:10.187756+0000","last_clean":"2024-04-25T17:33:10.187756+0000","last_became_active":"2024-04-25T17:32:35.074116+0000","last_became_peered":"2024-04-25T17:32:35.074116+0000","last_unstale":"2024-04-25T17:33:10.187756+0000","last_undegraded":"2024-04-25T17:33:10.187756+0000","last_fullsized":"2024-04-25T17:33:10.187756+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:09.233206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193473+0000","last_change":"2024-04-25T17:32:32.041368+0000","last_active":"2024-04-25T17:33:10.193473+0000","last_peered":"2024-04-25T17:33:10.193473+0000","last_clean":"2024-04-25T17:33:10.193473+0000","last_became_active":"2024-04-25T17:32:32.040940+0000","last_became_peered":"2024-04-25T17:32:32.040940+0000","last_unstale":"2024-04-25T17:33:10.193473+0000","last_undegraded":"2024-04-25T17:33:10.193473+0000","last_fullsized":"2024-04-25T17:33:10.193473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:38.871786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.004496+0000","last_change":"2024-04-25T17:32:35.073535+0000","last_active":"2024-04-25T17:34:39.004496+0000","last_peered":"2024-04-25T17:34:39.004496+0000","last_clean":"2024-04-25T17:34:39.004496+0000","last_became_active":"2024-04-25T17:32:35.073190+0000","last_became_peered":"2024-04-25T17:32:35.073190+0000","last_unstale":"2024-04-25T17:34:39.004496+0000","last_undegraded":"2024-04-25T17:34:39.004496+0000","last_fullsized":"2024-04-25T17:34:39.004496+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:19.841978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":135,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:36.298156+0000","last_change":"2024-04-25T17:32:32.044359+0000","last_active":"2024-04-25T17:34:36.298156+0000","last_peered":"2024-04-25T17:34:36.298156+0000","last_clean":"2024-04-25T17:34:36.298156+0000","last_became_active":"2024-04-25T17:32:32.044024+0000","last_became_peered":"2024-04-25T17:32:32.044024+0000","last_unstale":"2024-04-25T17:34:36.298156+0000","last_undegraded":"2024-04-25T17:34:36.298156+0000","last_fullsized":"2024-04-25T17:34:36.298156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:58.940554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.004819+0000","last_change":"2024-04-25T17:33:10.851665+0000","last_active":"2024-04-25T17:34:39.004819+0000","last_peered":"2024-04-25T17:34:39.004819+0000","last_clean":"2024-04-25T17:34:39.004819+0000","last_became_active":"2024-04-25T17:33:10.851217+0000","last_became_peered":"2024-04-25T17:33:10.851217+0000","last_unstale":"2024-04-25T17:34:39.004819+0000","last_undegraded":"2024-04-25T17:34:39.004819+0000","last_fullsized":"2024-04-25T17:34:39.004819+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:58.436715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.194439+0000","last_change":"2024-04-25T17:32:32.285823+0000","last_active":"2024-04-25T17:33:10.194439+0000","last_peered":"2024-04-25T17:33:10.194439+0000","last_clean":"2024-04-25T17:33:10.194439+0000","last_became_active":"2024-04-25T17:32:32.285416+0000","last_became_peered":"2024-04-25T17:32:32.285416+0000","last_unstale":"2024-04-25T17:33:10.194439+0000","last_undegraded":"2024-04-25T17:33:10.194439+0000","last_fullsized":"2024-04-25T17:33:10.194439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:24.473406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193119+0000","last_change":"2024-04-25T17:32:35.072626+0000","last_active":"2024-04-25T17:33:10.193119+0000","last_peered":"2024-04-25T17:33:10.193119+0000","last_clean":"2024-04-25T17:33:10.193119+0000","last_became_active":"2024-04-25T17:32:35.072264+0000","last_became_peered":"2024-04-25T17:32:35.072264+0000","last_unstale":"2024-04-25T17:33:10.193119+0000","last_undegraded":"2024-04-25T17:33:10.193119+0000","last_fullsized":"2024-04-25T17:33:10.193119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:54.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195985+0000","last_change":"2024-04-25T17:32:32.045589+0000","last_active":"2024-04-25T17:33:10.195985+0000","last_peered":"2024-04-25T17:33:10.195985+0000","last_clean":"2024-04-25T17:33:10.195985+0000","last_became_active":"2024-04-25T17:32:32.044838+0000","last_became_peered":"2024-04-25T17:32:32.044838+0000","last_unstale":"2024-04-25T17:33:10.195985+0000","last_undegraded":"2024-04-25T17:33:10.195985+0000","last_fullsized":"2024-04-25T17:33:10.195985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:05.748335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.993117+0000","last_change":"2024-04-25T17:32:35.075077+0000","last_active":"2024-04-25T17:34:38.993117+0000","last_peered":"2024-04-25T17:34:38.993117+0000","last_clean":"2024-04-25T17:34:38.993117+0000","last_became_active":"2024-04-25T17:32:35.074622+0000","last_became_peered":"2024-04-25T17:32:35.074622+0000","last_unstale":"2024-04-25T17:34:38.993117+0000","last_undegraded":"2024-04-25T17:34:38.993117+0000","last_fullsized":"2024-04-25T17:34:38.993117+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:53.435637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185818+0000","last_change":"2024-04-25T17:32:32.288707+0000","last_active":"2024-04-25T17:33:10.185818+0000","last_peered":"2024-04-25T17:33:10.185818+0000","last_clean":"2024-04-25T17:33:10.185818+0000","last_became_active":"2024-04-25T17:32:32.288321+0000","last_became_peered":"2024-04-25T17:32:32.288321+0000","last_unstale":"2024-04-25T17:33:10.185818+0000","last_undegraded":"2024-04-25T17:33:10.185818+0000","last_fullsized":"2024-04-25T17:33:10.185818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:00.113662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'1","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850203+0000","last_change":"2024-04-25T17:32:35.072100+0000","last_active":"2024-04-25T17:33:10.850203+0000","last_peered":"2024-04-25T17:33:10.850203+0000","last_clean":"2024-04-25T17:33:10.850203+0000","last_became_active":"2024-04-25T17:32:35.070706+0000","last_became_peered":"2024-04-25T17:32:35.070706+0000","last_unstale":"2024-04-25T17:33:10.850203+0000","last_undegraded":"2024-04-25T17:33:10.850203+0000","last_fullsized":"2024-04-25T17:33:10.850203+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:18.717385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190264+0000","last_change":"2024-04-25T17:32:32.042700+0000","last_active":"2024-04-25T17:33:10.190264+0000","last_peered":"2024-04-25T17:33:10.190264+0000","last_clean":"2024-04-25T17:33:10.190264+0000","last_became_active":"2024-04-25T17:32:32.042383+0000","last_became_peered":"2024-04-25T17:32:32.042383+0000","last_unstale":"2024-04-25T17:33:10.190264+0000","last_undegraded":"2024-04-25T17:33:10.190264+0000","last_fullsized":"2024-04-25T17:33:10.190264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:11.128322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.850056+0000","last_change":"2024-04-25T17:32:35.072336+0000","last_active":"2024-04-25T17:33:10.850056+0000","last_peered":"2024-04-25T17:33:10.850056+0000","last_clean":"2024-04-25T17:33:10.850056+0000","last_became_active":"2024-04-25T17:32:35.069830+0000","last_became_peered":"2024-04-25T17:32:35.069830+0000","last_unstale":"2024-04-25T17:33:10.850056+0000","last_undegraded":"2024-04-25T17:33:10.850056+0000","last_fullsized":"2024-04-25T17:33:10.850056+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:12.887428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193423+0000","last_change":"2024-04-25T17:32:35.071375+0000","last_active":"2024-04-25T17:33:10.193423+0000","last_peered":"2024-04-25T17:33:10.193423+0000","last_clean":"2024-04-25T17:33:10.193423+0000","last_became_active":"2024-04-25T17:32:35.071063+0000","last_became_peered":"2024-04-25T17:32:35.071063+0000","last_unstale":"2024-04-25T17:33:10.193423+0000","last_undegraded":"2024-04-25T17:33:10.193423+0000","last_fullsized":"2024-04-25T17:33:10.193423+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:41.917038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":133,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:37.298911+0000","last_change":"2024-04-25T17:32:32.286941+0000","last_active":"2024-04-25T17:34:37.298911+0000","last_peered":"2024-04-25T17:34:37.298911+0000","last_clean":"2024-04-25T17:34:37.298911+0000","last_became_active":"2024-04-25T17:32:32.286608+0000","last_became_peered":"2024-04-25T17:32:32.286608+0000","last_unstale":"2024-04-25T17:34:37.298911+0000","last_undegraded":"2024-04-25T17:34:37.298911+0000","last_fullsized":"2024-04-25T17:34:37.298911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.559724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190113+0000","last_change":"2024-04-25T17:32:35.068701+0000","last_active":"2024-04-25T17:33:10.190113+0000","last_peered":"2024-04-25T17:33:10.190113+0000","last_clean":"2024-04-25T17:33:10.190113+0000","last_became_active":"2024-04-25T17:32:35.068418+0000","last_became_peered":"2024-04-25T17:32:35.068418+0000","last_unstale":"2024-04-25T17:33:10.190113+0000","last_undegraded":"2024-04-25T17:33:10.190113+0000","last_fullsized":"2024-04-25T17:33:10.190113+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:42:25.431527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605706+0000","last_change":"2024-04-25T17:32:32.042167+0000","last_active":"2024-04-25T17:33:10.605706+0000","last_peered":"2024-04-25T17:33:10.605706+0000","last_clean":"2024-04-25T17:33:10.605706+0000","last_became_active":"2024-04-25T17:32:32.041765+0000","last_became_peered":"2024-04-25T17:32:32.041765+0000","last_unstale":"2024-04-25T17:33:10.605706+0000","last_undegraded":"2024-04-25T17:33:10.605706+0000","last_fullsized":"2024-04-25T17:33:10.605706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:28.901648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":360,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190045+0000","last_change":"2024-04-25T17:31:05.391870+0000","last_active":"2024-04-25T17:33:10.190045+0000","last_peered":"2024-04-25T17:33:10.190045+0000","last_clean":"2024-04-25T17:33:10.190045+0000","last_became_active":"2024-04-25T17:31:05.391426+0000","last_became_peered":"2024-04-25T17:31:05.391426+0000","last_unstale":"2024-04-25T17:33:10.190045+0000","last_undegraded":"2024-04-25T17:33:10.190045+0000","last_fullsized":"2024-04-25T17:33:10.190045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_clean_scrub_stamp":"2024-04-25T17:27:37.393149+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:18.400543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190518+0000","last_change":"2024-04-25T17:33:10.190518+0000","last_active":"2024-04-25T17:33:10.190518+0000","last_peered":"2024-04-25T17:33:10.190518+0000","last_clean":"2024-04-25T17:33:10.190518+0000","last_became_active":"2024-04-25T17:33:10.190166+0000","last_became_peered":"2024-04-25T17:33:10.190166+0000","last_unstale":"2024-04-25T17:33:10.190518+0000","last_undegraded":"2024-04-25T17:33:10.190518+0000","last_fullsized":"2024-04-25T17:33:10.190518+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:45.707078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.197863+0000","last_change":"2024-04-25T17:33:10.197412+0000","last_active":"2024-04-25T17:33:10.197863+0000","last_peered":"2024-04-25T17:33:10.197863+0000","last_clean":"2024-04-25T17:33:10.197863+0000","last_became_active":"2024-04-25T17:33:10.194388+0000","last_became_peered":"2024-04-25T17:33:10.194388+0000","last_unstale":"2024-04-25T17:33:10.197863+0000","last_undegraded":"2024-04-25T17:33:10.197863+0000","last_fullsized":"2024-04-25T17:33:10.197863+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:35.455072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195750+0000","last_change":"2024-04-25T17:32:32.285845+0000","last_active":"2024-04-25T17:33:10.195750+0000","last_peered":"2024-04-25T17:33:10.195750+0000","last_clean":"2024-04-25T17:33:10.195750+0000","last_became_active":"2024-04-25T17:32:32.285386+0000","last_became_peered":"2024-04-25T17:32:32.285386+0000","last_unstale":"2024-04-25T17:33:10.195750+0000","last_undegraded":"2024-04-25T17:33:10.195750+0000","last_fullsized":"2024-04-25T17:33:10.195750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:12.115900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187921+0000","last_change":"2024-04-25T17:32:35.065808+0000","last_active":"2024-04-25T17:33:10.187921+0000","last_peered":"2024-04-25T17:33:10.187921+0000","last_clean":"2024-04-25T17:33:10.187921+0000","last_became_active":"2024-04-25T17:32:35.065504+0000","last_became_peered":"2024-04-25T17:32:35.065504+0000","last_unstale":"2024-04-25T17:33:10.187921+0000","last_undegraded":"2024-04-25T17:33:10.187921+0000","last_fullsized":"2024-04-25T17:33:10.187921+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:33.438281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:18.083369+0000","last_change":"2024-04-25T17:32:32.045357+0000","last_active":"2024-04-25T17:33:18.083369+0000","last_peered":"2024-04-25T17:33:18.083369+0000","last_clean":"2024-04-25T17:33:18.083369+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:33:18.083369+0000","last_undegraded":"2024-04-25T17:33:18.083369+0000","last_fullsized":"2024-04-25T17:33:18.083369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:36.113130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190361+0000","last_change":"2024-04-25T17:32:35.077889+0000","last_active":"2024-04-25T17:33:10.190361+0000","last_peered":"2024-04-25T17:33:10.190361+0000","last_clean":"2024-04-25T17:33:10.190361+0000","last_became_active":"2024-04-25T17:32:35.076470+0000","last_became_peered":"2024-04-25T17:32:35.076470+0000","last_unstale":"2024-04-25T17:33:10.190361+0000","last_undegraded":"2024-04-25T17:33:10.190361+0000","last_fullsized":"2024-04-25T17:33:10.190361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:45.457146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605847+0000","last_change":"2024-04-25T17:32:32.406826+0000","last_active":"2024-04-25T17:33:10.605847+0000","last_peered":"2024-04-25T17:33:10.605847+0000","last_clean":"2024-04-25T17:33:10.605847+0000","last_became_active":"2024-04-25T17:32:32.406454+0000","last_became_peered":"2024-04-25T17:32:32.406454+0000","last_unstale":"2024-04-25T17:33:10.605847+0000","last_undegraded":"2024-04-25T17:33:10.605847+0000","last_fullsized":"2024-04-25T17:33:10.605847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:21:11.730942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647517+0000","last_change":"2024-04-25T17:32:35.075961+0000","last_active":"2024-04-25T17:33:10.647517+0000","last_peered":"2024-04-25T17:33:10.647517+0000","last_clean":"2024-04-25T17:33:10.647517+0000","last_became_active":"2024-04-25T17:32:35.075595+0000","last_became_peered":"2024-04-25T17:32:35.075595+0000","last_unstale":"2024-04-25T17:33:10.647517+0000","last_undegraded":"2024-04-25T17:33:10.647517+0000","last_fullsized":"2024-04-25T17:33:10.647517+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:25.513096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187390+0000","last_change":"2024-04-25T17:32:32.289123+0000","last_active":"2024-04-25T17:33:10.187390+0000","last_peered":"2024-04-25T17:33:10.187390+0000","last_clean":"2024-04-25T17:33:10.187390+0000","last_became_active":"2024-04-25T17:32:32.288783+0000","last_became_peered":"2024-04-25T17:32:32.288783+0000","last_unstale":"2024-04-25T17:33:10.187390+0000","last_undegraded":"2024-04-25T17:33:10.187390+0000","last_fullsized":"2024-04-25T17:33:10.187390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:52.222440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.965579+0000","last_change":"2024-04-25T17:32:35.066364+0000","last_active":"2024-04-25T17:34:38.965579+0000","last_peered":"2024-04-25T17:34:38.965579+0000","last_clean":"2024-04-25T17:34:38.965579+0000","last_became_active":"2024-04-25T17:32:35.065489+0000","last_became_peered":"2024-04-25T17:32:35.065489+0000","last_unstale":"2024-04-25T17:34:38.965579+0000","last_undegraded":"2024-04-25T17:34:38.965579+0000","last_fullsized":"2024-04-25T17:34:38.965579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:16.817796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189874+0000","last_change":"2024-04-25T17:32:32.040662+0000","last_active":"2024-04-25T17:33:10.189874+0000","last_peered":"2024-04-25T17:33:10.189874+0000","last_clean":"2024-04-25T17:33:10.189874+0000","last_became_active":"2024-04-25T17:32:32.040334+0000","last_became_peered":"2024-04-25T17:32:32.040334+0000","last_unstale":"2024-04-25T17:33:10.189874+0000","last_undegraded":"2024-04-25T17:33:10.189874+0000","last_fullsized":"2024-04-25T17:33:10.189874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:03.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.999502+0000","last_change":"2024-04-25T17:32:35.079183+0000","last_active":"2024-04-25T17:34:38.999502+0000","last_peered":"2024-04-25T17:34:38.999502+0000","last_clean":"2024-04-25T17:34:38.999502+0000","last_became_active":"2024-04-25T17:32:35.078472+0000","last_became_peered":"2024-04-25T17:32:35.078472+0000","last_unstale":"2024-04-25T17:34:38.999502+0000","last_undegraded":"2024-04-25T17:34:38.999502+0000","last_fullsized":"2024-04-25T17:34:38.999502+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:39:59.120020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.201283+0000","last_change":"2024-04-25T17:33:10.201283+0000","last_active":"2024-04-25T17:33:10.201283+0000","last_peered":"2024-04-25T17:33:10.201283+0000","last_clean":"2024-04-25T17:33:10.201283+0000","last_became_active":"2024-04-25T17:33:10.196192+0000","last_became_peered":"2024-04-25T17:33:10.196192+0000","last_unstale":"2024-04-25T17:33:10.201283+0000","last_undegraded":"2024-04-25T17:33:10.201283+0000","last_fullsized":"2024-04-25T17:33:10.201283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:03.716366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190342+0000","last_change":"2024-04-25T17:32:35.081593+0000","last_active":"2024-04-25T17:33:10.190342+0000","last_peered":"2024-04-25T17:33:10.190342+0000","last_clean":"2024-04-25T17:33:10.190342+0000","last_became_active":"2024-04-25T17:32:35.081248+0000","last_became_peered":"2024-04-25T17:32:35.081248+0000","last_unstale":"2024-04-25T17:33:10.190342+0000","last_undegraded":"2024-04-25T17:33:10.190342+0000","last_fullsized":"2024-04-25T17:33:10.190342+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:41:01.177899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.198901+0000","last_change":"2024-04-25T17:33:10.198901+0000","last_active":"2024-04-25T17:33:10.198901+0000","last_peered":"2024-04-25T17:33:10.198901+0000","last_clean":"2024-04-25T17:33:10.198901+0000","last_became_active":"2024-04-25T17:33:10.192191+0000","last_became_peered":"2024-04-25T17:33:10.192191+0000","last_unstale":"2024-04-25T17:33:10.198901+0000","last_undegraded":"2024-04-25T17:33:10.198901+0000","last_fullsized":"2024-04-25T17:33:10.198901+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:39.760642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.003272+0000","last_change":"2024-04-25T17:32:35.068592+0000","last_active":"2024-04-25T17:34:39.003272+0000","last_peered":"2024-04-25T17:34:39.003272+0000","last_clean":"2024-04-25T17:34:39.003272+0000","last_became_active":"2024-04-25T17:32:35.068240+0000","last_became_peered":"2024-04-25T17:32:35.068240+0000","last_unstale":"2024-04-25T17:34:39.003272+0000","last_undegraded":"2024-04-25T17:34:39.003272+0000","last_fullsized":"2024-04-25T17:34:39.003272+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:48.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.188180+0000","last_change":"2024-04-25T17:32:32.406702+0000","last_active":"2024-04-25T17:33:10.188180+0000","last_peered":"2024-04-25T17:33:10.188180+0000","last_clean":"2024-04-25T17:33:10.188180+0000","last_became_active":"2024-04-25T17:32:32.406263+0000","last_became_peered":"2024-04-25T17:32:32.406263+0000","last_unstale":"2024-04-25T17:33:10.188180+0000","last_undegraded":"2024-04-25T17:33:10.188180+0000","last_fullsized":"2024-04-25T17:33:10.188180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:27.627002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.000540+0000","last_change":"2024-04-25T17:32:35.073585+0000","last_active":"2024-04-25T17:34:39.000540+0000","last_peered":"2024-04-25T17:34:39.000540+0000","last_clean":"2024-04-25T17:34:39.000540+0000","last_became_active":"2024-04-25T17:32:35.072902+0000","last_became_peered":"2024-04-25T17:32:35.072902+0000","last_unstale":"2024-04-25T17:34:39.000540+0000","last_undegraded":"2024-04-25T17:34:39.000540+0000","last_fullsized":"2024-04-25T17:34:39.000540+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:17.277129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193299+0000","last_change":"2024-04-25T17:32:32.042546+0000","last_active":"2024-04-25T17:33:10.193299+0000","last_peered":"2024-04-25T17:33:10.193299+0000","last_clean":"2024-04-25T17:33:10.193299+0000","last_became_active":"2024-04-25T17:32:32.042191+0000","last_became_peered":"2024-04-25T17:32:32.042191+0000","last_unstale":"2024-04-25T17:33:10.193299+0000","last_undegraded":"2024-04-25T17:33:10.193299+0000","last_fullsized":"2024-04-25T17:33:10.193299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:58.126260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.993721+0000","last_change":"2024-04-25T17:32:35.077067+0000","last_active":"2024-04-25T17:34:38.993721+0000","last_peered":"2024-04-25T17:34:38.993721+0000","last_clean":"2024-04-25T17:34:38.993721+0000","last_became_active":"2024-04-25T17:32:35.076701+0000","last_became_peered":"2024-04-25T17:32:35.076701+0000","last_unstale":"2024-04-25T17:34:38.993721+0000","last_undegraded":"2024-04-25T17:34:38.993721+0000","last_fullsized":"2024-04-25T17:34:38.993721+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:06.646419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.610032+0000","last_change":"2024-04-25T17:33:10.610032+0000","last_active":"2024-04-25T17:33:10.610032+0000","last_peered":"2024-04-25T17:33:10.610032+0000","last_clean":"2024-04-25T17:33:10.610032+0000","last_became_active":"2024-04-25T17:33:10.607260+0000","last_became_peered":"2024-04-25T17:33:10.607260+0000","last_unstale":"2024-04-25T17:33:10.610032+0000","last_undegraded":"2024-04-25T17:33:10.610032+0000","last_fullsized":"2024-04-25T17:33:10.610032+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:42:41.509257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.001696+0000","last_change":"2024-04-25T17:32:35.071351+0000","last_active":"2024-04-25T17:34:39.001696+0000","last_peered":"2024-04-25T17:34:39.001696+0000","last_clean":"2024-04-25T17:34:39.001696+0000","last_became_active":"2024-04-25T17:32:35.070559+0000","last_became_peered":"2024-04-25T17:32:35.070559+0000","last_unstale":"2024-04-25T17:34:39.001696+0000","last_undegraded":"2024-04-25T17:34:39.001696+0000","last_fullsized":"2024-04-25T17:34:39.001696+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:23.864826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.000743+0000","last_change":"2024-04-25T17:32:35.073326+0000","last_active":"2024-04-25T17:34:39.000743+0000","last_peered":"2024-04-25T17:34:39.000743+0000","last_clean":"2024-04-25T17:34:39.000743+0000","last_became_active":"2024-04-25T17:32:35.072904+0000","last_became_peered":"2024-04-25T17:32:35.072904+0000","last_unstale":"2024-04-25T17:34:39.000743+0000","last_undegraded":"2024-04-25T17:34:39.000743+0000","last_fullsized":"2024-04-25T17:34:39.000743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:50.586127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195444+0000","last_change":"2024-04-25T17:33:10.195444+0000","last_active":"2024-04-25T17:33:10.195444+0000","last_peered":"2024-04-25T17:33:10.195444+0000","last_clean":"2024-04-25T17:33:10.195444+0000","last_became_active":"2024-04-25T17:33:10.195048+0000","last_became_peered":"2024-04-25T17:33:10.195048+0000","last_unstale":"2024-04-25T17:33:10.195444+0000","last_undegraded":"2024-04-25T17:33:10.195444+0000","last_fullsized":"2024-04-25T17:33:10.195444+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:03.098618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187678+0000","last_change":"2024-04-25T17:32:35.072053+0000","last_active":"2024-04-25T17:33:10.187678+0000","last_peered":"2024-04-25T17:33:10.187678+0000","last_clean":"2024-04-25T17:33:10.187678+0000","last_became_active":"2024-04-25T17:32:35.071712+0000","last_became_peered":"2024-04-25T17:32:35.071712+0000","last_unstale":"2024-04-25T17:33:10.187678+0000","last_undegraded":"2024-04-25T17:33:10.187678+0000","last_fullsized":"2024-04-25T17:33:10.187678+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:42.928854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.647627+0000","last_change":"2024-04-25T17:32:35.061408+0000","last_active":"2024-04-25T17:33:10.647627+0000","last_peered":"2024-04-25T17:33:10.647627+0000","last_clean":"2024-04-25T17:33:10.647627+0000","last_became_active":"2024-04-25T17:32:35.061105+0000","last_became_peered":"2024-04-25T17:32:35.061105+0000","last_unstale":"2024-04-25T17:33:10.647627+0000","last_undegraded":"2024-04-25T17:33:10.647627+0000","last_fullsized":"2024-04-25T17:33:10.647627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:48.242157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.687122+0000","last_change":"2024-04-25T17:33:10.687122+0000","last_active":"2024-04-25T17:33:10.687122+0000","last_peered":"2024-04-25T17:33:10.687122+0000","last_clean":"2024-04-25T17:33:10.687122+0000","last_became_active":"2024-04-25T17:33:10.686755+0000","last_became_peered":"2024-04-25T17:33:10.686755+0000","last_unstale":"2024-04-25T17:33:10.687122+0000","last_undegraded":"2024-04-25T17:33:10.687122+0000","last_fullsized":"2024-04-25T17:33:10.687122+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.999655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.996432+0000","last_change":"2024-04-25T17:32:35.077433+0000","last_active":"2024-04-25T17:34:38.996432+0000","last_peered":"2024-04-25T17:34:38.996432+0000","last_clean":"2024-04-25T17:34:38.996432+0000","last_became_active":"2024-04-25T17:32:35.076201+0000","last_became_peered":"2024-04-25T17:32:35.076201+0000","last_unstale":"2024-04-25T17:34:38.996432+0000","last_undegraded":"2024-04-25T17:34:38.996432+0000","last_fullsized":"2024-04-25T17:34:38.996432+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:42.677022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.188374+0000","last_change":"2024-04-25T17:32:35.068017+0000","last_active":"2024-04-25T17:33:10.188374+0000","last_peered":"2024-04-25T17:33:10.188374+0000","last_clean":"2024-04-25T17:33:10.188374+0000","last_became_active":"2024-04-25T17:32:35.067647+0000","last_became_peered":"2024-04-25T17:32:35.067647+0000","last_unstale":"2024-04-25T17:33:10.188374+0000","last_undegraded":"2024-04-25T17:33:10.188374+0000","last_fullsized":"2024-04-25T17:33:10.188374+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:28:11.339402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190441+0000","last_change":"2024-04-25T17:32:35.075192+0000","last_active":"2024-04-25T17:33:10.190441+0000","last_peered":"2024-04-25T17:33:10.190441+0000","last_clean":"2024-04-25T17:33:10.190441+0000","last_became_active":"2024-04-25T17:32:35.074850+0000","last_became_peered":"2024-04-25T17:32:35.074850+0000","last_unstale":"2024-04-25T17:33:10.190441+0000","last_undegraded":"2024-04-25T17:33:10.190441+0000","last_fullsized":"2024-04-25T17:33:10.190441+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:04:41.343312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193064+0000","last_change":"2024-04-25T17:32:35.078324+0000","last_active":"2024-04-25T17:33:10.193064+0000","last_peered":"2024-04-25T17:33:10.193064+0000","last_clean":"2024-04-25T17:33:10.193064+0000","last_became_active":"2024-04-25T17:32:35.077958+0000","last_became_peered":"2024-04-25T17:32:35.077958+0000","last_unstale":"2024-04-25T17:33:10.193064+0000","last_undegraded":"2024-04-25T17:33:10.193064+0000","last_fullsized":"2024-04-25T17:33:10.193064+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:10.148060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189834+0000","last_change":"2024-04-25T17:32:35.078023+0000","last_active":"2024-04-25T17:33:10.189834+0000","last_peered":"2024-04-25T17:33:10.189834+0000","last_clean":"2024-04-25T17:33:10.189834+0000","last_became_active":"2024-04-25T17:32:35.077702+0000","last_became_peered":"2024-04-25T17:32:35.077702+0000","last_unstale":"2024-04-25T17:33:10.189834+0000","last_undegraded":"2024-04-25T17:33:10.189834+0000","last_fullsized":"2024-04-25T17:33:10.189834+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:44:37.933179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187569+0000","last_change":"2024-04-25T17:32:35.078388+0000","last_active":"2024-04-25T17:33:10.187569+0000","last_peered":"2024-04-25T17:33:10.187569+0000","last_clean":"2024-04-25T17:33:10.187569+0000","last_became_active":"2024-04-25T17:32:35.078058+0000","last_became_peered":"2024-04-25T17:32:35.078058+0000","last_unstale":"2024-04-25T17:33:10.187569+0000","last_undegraded":"2024-04-25T17:33:10.187569+0000","last_fullsized":"2024-04-25T17:33:10.187569+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:05.699542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187895+0000","last_change":"2024-04-25T17:32:35.068532+0000","last_active":"2024-04-25T17:33:10.187895+0000","last_peered":"2024-04-25T17:33:10.187895+0000","last_clean":"2024-04-25T17:33:10.187895+0000","last_became_active":"2024-04-25T17:32:35.068225+0000","last_became_peered":"2024-04-25T17:32:35.068225+0000","last_unstale":"2024-04-25T17:33:10.187895+0000","last_undegraded":"2024-04-25T17:33:10.187895+0000","last_fullsized":"2024-04-25T17:33:10.187895+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:09.911999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.999091+0000","last_change":"2024-04-25T17:32:35.065526+0000","last_active":"2024-04-25T17:34:38.999091+0000","last_peered":"2024-04-25T17:34:38.999091+0000","last_clean":"2024-04-25T17:34:38.999091+0000","last_became_active":"2024-04-25T17:32:35.065146+0000","last_became_peered":"2024-04-25T17:32:35.065146+0000","last_unstale":"2024-04-25T17:34:38.999091+0000","last_undegraded":"2024-04-25T17:34:38.999091+0000","last_fullsized":"2024-04-25T17:34:38.999091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:33.662055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133,"ondisk_log_size":133,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280256,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66336,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":638,"num_read_kb":389,"num_write":166,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":199149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351016,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":482924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1137131,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776180,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":483160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037816,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":483080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299455,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":1075151,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462980,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":482549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":544677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":483820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":546027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":482860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":483377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1064960,"data_stored":1074303,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T17:34:41.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T17:34:42.112 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:42.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:42 smithi098 ceph-mon[29192]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:42.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:42.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:42 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:42.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:42 smithi098 ceph-mon[29192]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:34:42.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:42 smithi173 ceph-mon[32856]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:42.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:42.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:42 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:42.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:42 smithi173 ceph-mon[32856]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:34:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:42 smithi203 ceph-mon[32834]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:42 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:42.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:42 smithi203 ceph-mon[32834]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:34:43.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:34:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:34:43.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:43.384 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:34:43.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:34:43.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:43.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:34:43.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:34:43.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:34:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.982+0000 7ff78bf5b700 1 -- 172.21.15.98:0/3368995122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff78406b0f0 msgr2=0x7ff78410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.982+0000 7ff78bf5b700 1 --2- 172.21.15.98:0/3368995122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff78406b0f0 0x7ff78410f900 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00b0a0 tx=0x7ff77c00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:34:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.982+0000 7ff78bf5b700 1 -- 172.21.15.98:0/3368995122 shutdown_connections 2024-04-25T17:34:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.982+0000 7ff78bf5b700 1 --2- 172.21.15.98:0/3368995122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff78406b0f0 0x7ff78410f900 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.982+0000 7ff78bf5b700 1 --2- 172.21.15.98:0/3368995122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784108a30 0x7ff784108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.982+0000 7ff78bf5b700 1 --2- 172.21.15.98:0/3368995122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff78406a750 0x7ff78406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.982+0000 7ff78bf5b700 1 -- 172.21.15.98:0/3368995122 >> 172.21.15.98:0/3368995122 conn(0x7ff7840788f0 msgr2=0x7ff784078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.982+0000 7ff78bf5b700 1 -- 172.21.15.98:0/3368995122 shutdown_connections 2024-04-25T17:34:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.982+0000 7ff78bf5b700 1 -- 172.21.15.98:0/3368995122 wait complete. 2024-04-25T17:34:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff78bf5b700 1 Processor -- start 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff78bf5b700 1 -- start start 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff78bf5b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff78406a750 0x7ff7841990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff78bf5b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff78406b0f0 0x7ff784199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff78bf5b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784108a30 0x7ff78419d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff78bf5b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff784112bd0 con 0x7ff78406b0f0 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff78bf5b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff784112a50 con 0x7ff784108a30 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff78bf5b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff784112d50 con 0x7ff78406a750 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff7894f6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff78406b0f0 0x7ff784199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff7894f6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff78406b0f0 0x7ff784199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47134/0 (socket says 172.21.15.98:47134) 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff7894f6700 1 -- 172.21.15.98:0/2735128259 learned_addr learned my addr 172.21.15.98:0/2735128259 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff78a4f8700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784108a30 0x7ff78419d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff789cf7700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff78406a750 0x7ff7841990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff7894f6700 1 -- 172.21.15.98:0/2735128259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff78406a750 msgr2=0x7ff7841990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff7894f6700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff78406a750 0x7ff7841990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff7894f6700 1 -- 172.21.15.98:0/2735128259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784108a30 msgr2=0x7ff78419d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff7894f6700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784108a30 0x7ff78419d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff7894f6700 1 -- 172.21.15.98:0/2735128259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78419e0a0 con 0x7ff78406b0f0 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff78a4f8700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784108a30 0x7ff78419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff7894f6700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff78406b0f0 0x7ff784199620 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff77400c920 tx=0x7ff77400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.983+0000 7ff77affd700 1 -- 172.21.15.98:0/2735128259 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774007b00 con 0x7ff78406b0f0 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.984+0000 7ff77affd700 1 -- 172.21.15.98:0/2735128259 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff774004ce0 con 0x7ff78406b0f0 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.984+0000 7ff78bf5b700 1 -- 172.21.15.98:0/2735128259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78419e390 con 0x7ff78406b0f0 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.984+0000 7ff77affd700 1 -- 172.21.15.98:0/2735128259 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774005650 con 0x7ff78406b0f0 2024-04-25T17:34:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.984+0000 7ff78bf5b700 1 -- 172.21.15.98:0/2735128259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7841a5c10 con 0x7ff78406b0f0 2024-04-25T17:34:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.984+0000 7ff78bf5b700 1 -- 172.21.15.98:0/2735128259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff784069730 con 0x7ff78406b0f0 2024-04-25T17:34:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.986+0000 7ff77affd700 1 -- 172.21.15.98:0/2735128259 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff774004750 con 0x7ff78406b0f0 2024-04-25T17:34:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.986+0000 7ff77affd700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7700a3c40 0x7ff7700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.986+0000 7ff789cf7700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7700a3c40 0x7ff7700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.986+0000 7ff77affd700 1 -- 172.21.15.98:0/2735128259 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7ff7740cffa0 con 0x7ff78406b0f0 2024-04-25T17:34:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.987+0000 7ff789cf7700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7700a3c40 0x7ff7700a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff780005fd0 tx=0x7ff780005ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:43.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:43.991+0000 7ff77affd700 1 -- 172.21.15.98:0/2735128259 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff774014030 con 0x7ff78406b0f0 2024-04-25T17:34:44.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.185+0000 7ff78bf5b700 1 -- 172.21.15.98:0/2735128259 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff784062e80 con 0x7ff7700a3c40 2024-04-25T17:34:44.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.192+0000 7ff77affd700 1 -- 172.21.15.98:0/2735128259 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259126 (secure 0 0 0) 0x7ff784062e80 con 0x7ff7700a3c40 2024-04-25T17:34:44.197 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:34:44.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 -- 172.21.15.98:0/2735128259 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7700a3c40 msgr2=0x7ff7700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7700a3c40 0x7ff7700a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff780005fd0 tx=0x7ff780005ce0 comp rx=0 tx=0).stop 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 -- 172.21.15.98:0/2735128259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff78406b0f0 msgr2=0x7ff784199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff78406b0f0 0x7ff784199620 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff77400c920 tx=0x7ff77400cce0 comp rx=0 tx=0).stop 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 -- 172.21.15.98:0/2735128259 shutdown_connections 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7700a3c40 0x7ff7700a6100 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff78406a750 0x7ff7841990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff784108a30 0x7ff78419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 --2- 172.21.15.98:0/2735128259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff78406b0f0 0x7ff784199620 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 -- 172.21.15.98:0/2735128259 >> 172.21.15.98:0/2735128259 conn(0x7ff7840788f0 msgr2=0x7ff7840ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 -- 172.21.15.98:0/2735128259 shutdown_connections 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:44.195+0000 7ff78bf5b700 1 -- 172.21.15.98:0/2735128259 wait complete. 2024-04-25T17:34:44.199 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T17:34:44.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:44 smithi098 ceph-mon[29192]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-25T17:34:44.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:44 smithi203 ceph-mon[32834]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-25T17:34:44.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:44 smithi173 ceph-mon[32856]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-25T17:34:44.890 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-04-25T17:34:42.997257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657328,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":801,"num_read_kb":520,"num_write":355,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":736,"ondisk_log_size":736,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357116,"kb_used_data":12108,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584708,"statfs":{"total":1151940427776,"available":1151574740992,"internally_reserved":0,"allocated":12398592,"data_stored":7760088,"data_compressed":106941,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":87,"num_read_kb":42,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003301"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.997821+0000","last_change":"2024-04-25T17:32:35.075338+0000","last_active":"2024-04-25T17:34:38.997821+0000","last_peered":"2024-04-25T17:34:38.997821+0000","last_clean":"2024-04-25T17:34:38.997821+0000","last_became_active":"2024-04-25T17:32:35.075029+0000","last_became_peered":"2024-04-25T17:32:35.075029+0000","last_unstale":"2024-04-25T17:34:38.997821+0000","last_undegraded":"2024-04-25T17:34:38.997821+0000","last_fullsized":"2024-04-25T17:34:38.997821+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:23.837337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.996187+0000","last_change":"2024-04-25T17:32:35.075422+0000","last_active":"2024-04-25T17:34:38.996187+0000","last_peered":"2024-04-25T17:34:38.996187+0000","last_clean":"2024-04-25T17:34:38.996187+0000","last_became_active":"2024-04-25T17:32:35.071975+0000","last_became_peered":"2024-04-25T17:32:35.071975+0000","last_unstale":"2024-04-25T17:34:38.996187+0000","last_undegraded":"2024-04-25T17:34:38.996187+0000","last_fullsized":"2024-04-25T17:34:38.996187+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:20.548869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.005763+0000","last_change":"2024-04-25T17:32:35.068153+0000","last_active":"2024-04-25T17:34:39.005763+0000","last_peered":"2024-04-25T17:34:39.005763+0000","last_clean":"2024-04-25T17:34:39.005763+0000","last_became_active":"2024-04-25T17:32:35.065873+0000","last_became_peered":"2024-04-25T17:32:35.065873+0000","last_unstale":"2024-04-25T17:34:39.005763+0000","last_undegraded":"2024-04-25T17:34:39.005763+0000","last_fullsized":"2024-04-25T17:34:39.005763+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:31.381875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.999667+0000","last_change":"2024-04-25T17:32:35.073055+0000","last_active":"2024-04-25T17:34:38.999667+0000","last_peered":"2024-04-25T17:34:38.999667+0000","last_clean":"2024-04-25T17:34:38.999667+0000","last_became_active":"2024-04-25T17:32:35.072596+0000","last_became_peered":"2024-04-25T17:32:35.072596+0000","last_unstale":"2024-04-25T17:34:38.999667+0000","last_undegraded":"2024-04-25T17:34:38.999667+0000","last_fullsized":"2024-04-25T17:34:38.999667+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:37.529698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.978679+0000","last_change":"2024-04-25T17:32:35.072721+0000","last_active":"2024-04-25T17:34:38.978679+0000","last_peered":"2024-04-25T17:34:38.978679+0000","last_clean":"2024-04-25T17:34:38.978679+0000","last_became_active":"2024-04-25T17:32:35.072370+0000","last_became_peered":"2024-04-25T17:32:35.072370+0000","last_unstale":"2024-04-25T17:34:38.978679+0000","last_undegraded":"2024-04-25T17:34:38.978679+0000","last_fullsized":"2024-04-25T17:34:38.978679+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:34.429985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.003305+0000","last_change":"2024-04-25T17:32:35.071538+0000","last_active":"2024-04-25T17:34:39.003305+0000","last_peered":"2024-04-25T17:34:39.003305+0000","last_clean":"2024-04-25T17:34:39.003305+0000","last_became_active":"2024-04-25T17:32:35.067427+0000","last_became_peered":"2024-04-25T17:32:35.067427+0000","last_unstale":"2024-04-25T17:34:39.003305+0000","last_undegraded":"2024-04-25T17:34:39.003305+0000","last_fullsized":"2024-04-25T17:34:39.003305+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:36.052762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.992163+0000","last_change":"2024-04-25T17:32:35.079476+0000","last_active":"2024-04-25T17:34:38.992163+0000","last_peered":"2024-04-25T17:34:38.992163+0000","last_clean":"2024-04-25T17:34:38.992163+0000","last_became_active":"2024-04-25T17:32:35.079170+0000","last_became_peered":"2024-04-25T17:32:35.079170+0000","last_unstale":"2024-04-25T17:34:38.992163+0000","last_undegraded":"2024-04-25T17:34:38.992163+0000","last_fullsized":"2024-04-25T17:34:38.992163+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:30.937670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.000660+0000","last_change":"2024-04-25T17:32:35.077261+0000","last_active":"2024-04-25T17:34:39.000660+0000","last_peered":"2024-04-25T17:34:39.000660+0000","last_clean":"2024-04-25T17:34:39.000660+0000","last_became_active":"2024-04-25T17:32:35.076896+0000","last_became_peered":"2024-04-25T17:32:35.076896+0000","last_unstale":"2024-04-25T17:34:39.000660+0000","last_undegraded":"2024-04-25T17:34:39.000660+0000","last_fullsized":"2024-04-25T17:34:39.000660+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:24.082550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.005902+0000","last_change":"2024-04-25T17:32:35.072229+0000","last_active":"2024-04-25T17:34:39.005902+0000","last_peered":"2024-04-25T17:34:39.005902+0000","last_clean":"2024-04-25T17:34:39.005902+0000","last_became_active":"2024-04-25T17:32:35.071586+0000","last_became_peered":"2024-04-25T17:32:35.071586+0000","last_unstale":"2024-04-25T17:34:39.005902+0000","last_undegraded":"2024-04-25T17:34:39.005902+0000","last_fullsized":"2024-04-25T17:34:39.005902+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:01.195638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.003762+0000","last_change":"2024-04-25T17:32:35.075274+0000","last_active":"2024-04-25T17:34:39.003762+0000","last_peered":"2024-04-25T17:34:39.003762+0000","last_clean":"2024-04-25T17:34:39.003762+0000","last_became_active":"2024-04-25T17:32:35.073821+0000","last_became_peered":"2024-04-25T17:32:35.073821+0000","last_unstale":"2024-04-25T17:34:39.003762+0000","last_undegraded":"2024-04-25T17:34:39.003762+0000","last_fullsized":"2024-04-25T17:34:39.003762+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:43:04.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.995557+0000","last_change":"2024-04-25T17:32:35.080747+0000","last_active":"2024-04-25T17:34:38.995557+0000","last_peered":"2024-04-25T17:34:38.995557+0000","last_clean":"2024-04-25T17:34:38.995557+0000","last_became_active":"2024-04-25T17:32:35.080383+0000","last_became_peered":"2024-04-25T17:32:35.080383+0000","last_unstale":"2024-04-25T17:34:38.995557+0000","last_undegraded":"2024-04-25T17:34:38.995557+0000","last_fullsized":"2024-04-25T17:34:38.995557+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:42.128892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.995584+0000","last_change":"2024-04-25T17:32:35.061442+0000","last_active":"2024-04-25T17:34:38.995584+0000","last_peered":"2024-04-25T17:34:38.995584+0000","last_clean":"2024-04-25T17:34:38.995584+0000","last_became_active":"2024-04-25T17:32:35.061030+0000","last_became_peered":"2024-04-25T17:32:35.061030+0000","last_unstale":"2024-04-25T17:34:38.995584+0000","last_undegraded":"2024-04-25T17:34:38.995584+0000","last_fullsized":"2024-04-25T17:34:38.995584+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:03.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187927+0000","last_change":"2024-04-25T17:32:35.069698+0000","last_active":"2024-04-25T17:33:10.187927+0000","last_peered":"2024-04-25T17:33:10.187927+0000","last_clean":"2024-04-25T17:33:10.187927+0000","last_became_active":"2024-04-25T17:32:35.069374+0000","last_became_peered":"2024-04-25T17:32:35.069374+0000","last_unstale":"2024-04-25T17:33:10.187927+0000","last_undegraded":"2024-04-25T17:33:10.187927+0000","last_fullsized":"2024-04-25T17:33:10.187927+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:03.214525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.997081+0000","last_change":"2024-04-25T17:32:35.069001+0000","last_active":"2024-04-25T17:34:38.997081+0000","last_peered":"2024-04-25T17:34:38.997081+0000","last_clean":"2024-04-25T17:34:38.997081+0000","last_became_active":"2024-04-25T17:32:35.064882+0000","last_became_peered":"2024-04-25T17:32:35.064882+0000","last_unstale":"2024-04-25T17:34:38.997081+0000","last_undegraded":"2024-04-25T17:34:38.997081+0000","last_fullsized":"2024-04-25T17:34:38.997081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:32.122251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.002188+0000","last_change":"2024-04-25T17:32:35.065042+0000","last_active":"2024-04-25T17:34:39.002188+0000","last_peered":"2024-04-25T17:34:39.002188+0000","last_clean":"2024-04-25T17:34:39.002188+0000","last_became_active":"2024-04-25T17:32:35.064719+0000","last_became_peered":"2024-04-25T17:32:35.064719+0000","last_unstale":"2024-04-25T17:34:39.002188+0000","last_undegraded":"2024-04-25T17:34:39.002188+0000","last_fullsized":"2024-04-25T17:34:39.002188+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:54.715385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.000673+0000","last_change":"2024-04-25T17:32:35.067390+0000","last_active":"2024-04-25T17:34:39.000673+0000","last_peered":"2024-04-25T17:34:39.000673+0000","last_clean":"2024-04-25T17:34:39.000673+0000","last_became_active":"2024-04-25T17:32:35.066846+0000","last_became_peered":"2024-04-25T17:32:35.066846+0000","last_unstale":"2024-04-25T17:34:39.000673+0000","last_undegraded":"2024-04-25T17:34:39.000673+0000","last_fullsized":"2024-04-25T17:34:39.000673+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:24.305767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.999597+0000","last_change":"2024-04-25T17:32:35.075349+0000","last_active":"2024-04-25T17:34:38.999597+0000","last_peered":"2024-04-25T17:34:38.999597+0000","last_clean":"2024-04-25T17:34:38.999597+0000","last_became_active":"2024-04-25T17:32:35.075046+0000","last_became_peered":"2024-04-25T17:32:35.075046+0000","last_unstale":"2024-04-25T17:34:38.999597+0000","last_undegraded":"2024-04-25T17:34:38.999597+0000","last_fullsized":"2024-04-25T17:34:38.999597+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:51.703406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.991822+0000","last_change":"2024-04-25T17:32:35.077597+0000","last_active":"2024-04-25T17:34:38.991822+0000","last_peered":"2024-04-25T17:34:38.991822+0000","last_clean":"2024-04-25T17:34:38.991822+0000","last_became_active":"2024-04-25T17:32:35.077232+0000","last_became_peered":"2024-04-25T17:32:35.077232+0000","last_unstale":"2024-04-25T17:34:38.991822+0000","last_undegraded":"2024-04-25T17:34:38.991822+0000","last_fullsized":"2024-04-25T17:34:38.991822+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:46.842673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.003086+0000","last_change":"2024-04-25T17:32:35.078229+0000","last_active":"2024-04-25T17:34:39.003086+0000","last_peered":"2024-04-25T17:34:39.003086+0000","last_clean":"2024-04-25T17:34:39.003086+0000","last_became_active":"2024-04-25T17:32:35.077899+0000","last_became_peered":"2024-04-25T17:32:35.077899+0000","last_unstale":"2024-04-25T17:34:39.003086+0000","last_undegraded":"2024-04-25T17:34:39.003086+0000","last_fullsized":"2024-04-25T17:34:39.003086+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:22.776371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.186009+0000","last_change":"2024-04-25T17:32:32.045385+0000","last_active":"2024-04-25T17:33:10.186009+0000","last_peered":"2024-04-25T17:33:10.186009+0000","last_clean":"2024-04-25T17:33:10.186009+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:33:10.186009+0000","last_undegraded":"2024-04-25T17:33:10.186009+0000","last_fullsized":"2024-04-25T17:33:10.186009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:29.732001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.998246+0000","last_change":"2024-04-25T17:32:35.079966+0000","last_active":"2024-04-25T17:34:38.998246+0000","last_peered":"2024-04-25T17:34:38.998246+0000","last_clean":"2024-04-25T17:34:38.998246+0000","last_became_active":"2024-04-25T17:32:35.079649+0000","last_became_peered":"2024-04-25T17:32:35.079649+0000","last_unstale":"2024-04-25T17:34:38.998246+0000","last_undegraded":"2024-04-25T17:34:38.998246+0000","last_fullsized":"2024-04-25T17:34:38.998246+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:19:45.679273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190761+0000","last_change":"2024-04-25T17:32:32.285667+0000","last_active":"2024-04-25T17:33:10.190761+0000","last_peered":"2024-04-25T17:33:10.190761+0000","last_clean":"2024-04-25T17:33:10.190761+0000","last_became_active":"2024-04-25T17:32:32.285330+0000","last_became_peered":"2024-04-25T17:32:32.285330+0000","last_unstale":"2024-04-25T17:33:10.190761+0000","last_undegraded":"2024-04-25T17:33:10.190761+0000","last_fullsized":"2024-04-25T17:33:10.190761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.975280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.006677+0000","last_change":"2024-04-25T17:32:35.068841+0000","last_active":"2024-04-25T17:34:39.006677+0000","last_peered":"2024-04-25T17:34:39.006677+0000","last_clean":"2024-04-25T17:34:39.006677+0000","last_became_active":"2024-04-25T17:32:35.068518+0000","last_became_peered":"2024-04-25T17:32:35.068518+0000","last_unstale":"2024-04-25T17:34:39.006677+0000","last_undegraded":"2024-04-25T17:34:39.006677+0000","last_fullsized":"2024-04-25T17:34:39.006677+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:23.290990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195797+0000","last_change":"2024-04-25T17:32:32.045627+0000","last_active":"2024-04-25T17:33:10.195797+0000","last_peered":"2024-04-25T17:33:10.195797+0000","last_clean":"2024-04-25T17:33:10.195797+0000","last_became_active":"2024-04-25T17:32:32.044971+0000","last_became_peered":"2024-04-25T17:32:32.044971+0000","last_unstale":"2024-04-25T17:33:10.195797+0000","last_undegraded":"2024-04-25T17:33:10.195797+0000","last_fullsized":"2024-04-25T17:33:10.195797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:50.590435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.005986+0000","last_change":"2024-04-25T17:32:35.070862+0000","last_active":"2024-04-25T17:34:39.005986+0000","last_peered":"2024-04-25T17:34:39.005986+0000","last_clean":"2024-04-25T17:34:39.005986+0000","last_became_active":"2024-04-25T17:32:35.069927+0000","last_became_peered":"2024-04-25T17:32:35.069927+0000","last_unstale":"2024-04-25T17:34:39.005986+0000","last_undegraded":"2024-04-25T17:34:39.005986+0000","last_fullsized":"2024-04-25T17:34:39.005986+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:40:01.799015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195097+0000","last_change":"2024-04-25T17:32:32.046062+0000","last_active":"2024-04-25T17:33:10.195097+0000","last_peered":"2024-04-25T17:33:10.195097+0000","last_clean":"2024-04-25T17:33:10.195097+0000","last_became_active":"2024-04-25T17:32:32.044903+0000","last_became_peered":"2024-04-25T17:32:32.044903+0000","last_unstale":"2024-04-25T17:33:10.195097+0000","last_undegraded":"2024-04-25T17:33:10.195097+0000","last_fullsized":"2024-04-25T17:33:10.195097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:51.099460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.002568+0000","last_change":"2024-04-25T17:32:35.075452+0000","last_active":"2024-04-25T17:34:39.002568+0000","last_peered":"2024-04-25T17:34:39.002568+0000","last_clean":"2024-04-25T17:34:39.002568+0000","last_became_active":"2024-04-25T17:32:35.075136+0000","last_became_peered":"2024-04-25T17:32:35.075136+0000","last_unstale":"2024-04-25T17:34:39.002568+0000","last_undegraded":"2024-04-25T17:34:39.002568+0000","last_fullsized":"2024-04-25T17:34:39.002568+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:02.290926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":139,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:40.304950+0000","last_change":"2024-04-25T17:32:32.042019+0000","last_active":"2024-04-25T17:34:40.304950+0000","last_peered":"2024-04-25T17:34:40.304950+0000","last_clean":"2024-04-25T17:34:40.304950+0000","last_became_active":"2024-04-25T17:32:32.041641+0000","last_became_peered":"2024-04-25T17:32:32.041641+0000","last_unstale":"2024-04-25T17:34:40.304950+0000","last_undegraded":"2024-04-25T17:34:40.304950+0000","last_fullsized":"2024-04-25T17:34:40.304950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:02.025915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.998149+0000","last_change":"2024-04-25T17:32:35.069715+0000","last_active":"2024-04-25T17:34:38.998149+0000","last_peered":"2024-04-25T17:34:38.998149+0000","last_clean":"2024-04-25T17:34:38.998149+0000","last_became_active":"2024-04-25T17:32:35.069339+0000","last_became_peered":"2024-04-25T17:32:35.069339+0000","last_unstale":"2024-04-25T17:34:38.998149+0000","last_undegraded":"2024-04-25T17:34:38.998149+0000","last_fullsized":"2024-04-25T17:34:38.998149+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:48:13.572261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.849969+0000","last_change":"2024-04-25T17:32:32.042445+0000","last_active":"2024-04-25T17:33:10.849969+0000","last_peered":"2024-04-25T17:33:10.849969+0000","last_clean":"2024-04-25T17:33:10.849969+0000","last_became_active":"2024-04-25T17:32:32.042033+0000","last_became_peered":"2024-04-25T17:32:32.042033+0000","last_unstale":"2024-04-25T17:33:10.849969+0000","last_undegraded":"2024-04-25T17:33:10.849969+0000","last_fullsized":"2024-04-25T17:33:10.849969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.968383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.000301+0000","last_change":"2024-04-25T17:32:35.074492+0000","last_active":"2024-04-25T17:34:39.000301+0000","last_peered":"2024-04-25T17:34:39.000301+0000","last_clean":"2024-04-25T17:34:39.000301+0000","last_became_active":"2024-04-25T17:32:35.073652+0000","last_became_peered":"2024-04-25T17:32:35.073652+0000","last_unstale":"2024-04-25T17:34:39.000301+0000","last_undegraded":"2024-04-25T17:34:39.000301+0000","last_fullsized":"2024-04-25T17:34:39.000301+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:00.300453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187848+0000","last_change":"2024-04-25T17:32:32.039568+0000","last_active":"2024-04-25T17:33:10.187848+0000","last_peered":"2024-04-25T17:33:10.187848+0000","last_clean":"2024-04-25T17:33:10.187848+0000","last_became_active":"2024-04-25T17:32:32.039228+0000","last_became_peered":"2024-04-25T17:32:32.039228+0000","last_unstale":"2024-04-25T17:33:10.187848+0000","last_undegraded":"2024-04-25T17:33:10.187848+0000","last_fullsized":"2024-04-25T17:33:10.187848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:17:29.079749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.994678+0000","last_change":"2024-04-25T17:32:35.077619+0000","last_active":"2024-04-25T17:34:38.994678+0000","last_peered":"2024-04-25T17:34:38.994678+0000","last_clean":"2024-04-25T17:34:38.994678+0000","last_became_active":"2024-04-25T17:32:35.077309+0000","last_became_peered":"2024-04-25T17:32:35.077309+0000","last_unstale":"2024-04-25T17:34:38.994678+0000","last_undegraded":"2024-04-25T17:34:38.994678+0000","last_fullsized":"2024-04-25T17:34:38.994678+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:41.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193055+0000","last_change":"2024-04-25T17:32:32.044650+0000","last_active":"2024-04-25T17:33:10.193055+0000","last_peered":"2024-04-25T17:33:10.193055+0000","last_clean":"2024-04-25T17:33:10.193055+0000","last_became_active":"2024-04-25T17:32:32.043999+0000","last_became_peered":"2024-04-25T17:32:32.043999+0000","last_unstale":"2024-04-25T17:33:10.193055+0000","last_undegraded":"2024-04-25T17:33:10.193055+0000","last_fullsized":"2024-04-25T17:33:10.193055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:19.463941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.999351+0000","last_change":"2024-04-25T17:32:35.080598+0000","last_active":"2024-04-25T17:34:38.999351+0000","last_peered":"2024-04-25T17:34:38.999351+0000","last_clean":"2024-04-25T17:34:38.999351+0000","last_became_active":"2024-04-25T17:32:35.080300+0000","last_became_peered":"2024-04-25T17:32:35.080300+0000","last_unstale":"2024-04-25T17:34:38.999351+0000","last_undegraded":"2024-04-25T17:34:38.999351+0000","last_fullsized":"2024-04-25T17:34:38.999351+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:42.149746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":136,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:41.305755+0000","last_change":"2024-04-25T17:32:32.043889+0000","last_active":"2024-04-25T17:34:41.305755+0000","last_peered":"2024-04-25T17:34:41.305755+0000","last_clean":"2024-04-25T17:34:41.305755+0000","last_became_active":"2024-04-25T17:32:32.043520+0000","last_became_peered":"2024-04-25T17:32:32.043520+0000","last_unstale":"2024-04-25T17:34:41.305755+0000","last_undegraded":"2024-04-25T17:34:41.305755+0000","last_fullsized":"2024-04-25T17:34:41.305755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:25.134797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.996631+0000","last_change":"2024-04-25T17:32:35.071733+0000","last_active":"2024-04-25T17:34:38.996631+0000","last_peered":"2024-04-25T17:34:38.996631+0000","last_clean":"2024-04-25T17:34:38.996631+0000","last_became_active":"2024-04-25T17:32:35.071383+0000","last_became_peered":"2024-04-25T17:32:35.071383+0000","last_unstale":"2024-04-25T17:34:38.996631+0000","last_undegraded":"2024-04-25T17:34:38.996631+0000","last_fullsized":"2024-04-25T17:34:38.996631+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:09:49.433675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190101+0000","last_change":"2024-04-25T17:32:32.044823+0000","last_active":"2024-04-25T17:33:10.190101+0000","last_peered":"2024-04-25T17:33:10.190101+0000","last_clean":"2024-04-25T17:33:10.190101+0000","last_became_active":"2024-04-25T17:32:32.044497+0000","last_became_peered":"2024-04-25T17:32:32.044497+0000","last_unstale":"2024-04-25T17:33:10.190101+0000","last_undegraded":"2024-04-25T17:33:10.190101+0000","last_fullsized":"2024-04-25T17:33:10.190101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:47.811172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.997652+0000","last_change":"2024-04-25T17:32:35.071109+0000","last_active":"2024-04-25T17:34:38.997652+0000","last_peered":"2024-04-25T17:34:38.997652+0000","last_clean":"2024-04-25T17:34:38.997652+0000","last_became_active":"2024-04-25T17:32:35.070782+0000","last_became_peered":"2024-04-25T17:32:35.070782+0000","last_unstale":"2024-04-25T17:34:38.997652+0000","last_undegraded":"2024-04-25T17:34:38.997652+0000","last_fullsized":"2024-04-25T17:34:38.997652+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:37.247274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193437+0000","last_change":"2024-04-25T17:32:32.288683+0000","last_active":"2024-04-25T17:33:10.193437+0000","last_peered":"2024-04-25T17:33:10.193437+0000","last_clean":"2024-04-25T17:33:10.193437+0000","last_became_active":"2024-04-25T17:32:32.288261+0000","last_became_peered":"2024-04-25T17:32:32.288261+0000","last_unstale":"2024-04-25T17:33:10.193437+0000","last_undegraded":"2024-04-25T17:33:10.193437+0000","last_fullsized":"2024-04-25T17:33:10.193437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:36.502434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.005853+0000","last_change":"2024-04-25T17:32:35.060828+0000","last_active":"2024-04-25T17:34:39.005853+0000","last_peered":"2024-04-25T17:34:39.005853+0000","last_clean":"2024-04-25T17:34:39.005853+0000","last_became_active":"2024-04-25T17:32:35.060433+0000","last_became_peered":"2024-04-25T17:32:35.060433+0000","last_unstale":"2024-04-25T17:34:39.005853+0000","last_undegraded":"2024-04-25T17:34:39.005853+0000","last_fullsized":"2024-04-25T17:34:39.005853+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:32.382042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'32","reported_seq":283,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:42.307405+0000","last_change":"2024-04-25T17:32:32.042979+0000","last_active":"2024-04-25T17:34:42.307405+0000","last_peered":"2024-04-25T17:34:42.307405+0000","last_clean":"2024-04-25T17:34:42.307405+0000","last_became_active":"2024-04-25T17:32:32.042257+0000","last_became_peered":"2024-04-25T17:32:32.042257+0000","last_unstale":"2024-04-25T17:34:42.307405+0000","last_undegraded":"2024-04-25T17:34:42.307405+0000","last_fullsized":"2024-04-25T17:34:42.307405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:19.316803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62584,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":225,"num_read_kb":174,"num_write":33,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.985081+0000","last_change":"2024-04-25T17:32:35.069994+0000","last_active":"2024-04-25T17:34:38.985081+0000","last_peered":"2024-04-25T17:34:38.985081+0000","last_clean":"2024-04-25T17:34:38.985081+0000","last_became_active":"2024-04-25T17:32:35.069632+0000","last_became_peered":"2024-04-25T17:32:35.069632+0000","last_unstale":"2024-04-25T17:34:38.985081+0000","last_undegraded":"2024-04-25T17:34:38.985081+0000","last_fullsized":"2024-04-25T17:34:38.985081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:51.169489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187904+0000","last_change":"2024-04-25T17:32:32.042652+0000","last_active":"2024-04-25T17:33:10.187904+0000","last_peered":"2024-04-25T17:33:10.187904+0000","last_clean":"2024-04-25T17:33:10.187904+0000","last_became_active":"2024-04-25T17:32:32.042214+0000","last_became_peered":"2024-04-25T17:32:32.042214+0000","last_unstale":"2024-04-25T17:33:10.187904+0000","last_undegraded":"2024-04-25T17:33:10.187904+0000","last_fullsized":"2024-04-25T17:33:10.187904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.662580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187756+0000","last_change":"2024-04-25T17:32:35.074449+0000","last_active":"2024-04-25T17:33:10.187756+0000","last_peered":"2024-04-25T17:33:10.187756+0000","last_clean":"2024-04-25T17:33:10.187756+0000","last_became_active":"2024-04-25T17:32:35.074116+0000","last_became_peered":"2024-04-25T17:32:35.074116+0000","last_unstale":"2024-04-25T17:33:10.187756+0000","last_undegraded":"2024-04-25T17:33:10.187756+0000","last_fullsized":"2024-04-25T17:33:10.187756+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:09.233206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193473+0000","last_change":"2024-04-25T17:32:32.041368+0000","last_active":"2024-04-25T17:33:10.193473+0000","last_peered":"2024-04-25T17:33:10.193473+0000","last_clean":"2024-04-25T17:33:10.193473+0000","last_became_active":"2024-04-25T17:32:32.040940+0000","last_became_peered":"2024-04-25T17:32:32.040940+0000","last_unstale":"2024-04-25T17:33:10.193473+0000","last_undegraded":"2024-04-25T17:33:10.193473+0000","last_fullsized":"2024-04-25T17:33:10.193473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:38.871786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.004496+0000","last_change":"2024-04-25T17:32:35.073535+0000","last_active":"2024-04-25T17:34:39.004496+0000","last_peered":"2024-04-25T17:34:39.004496+0000","last_clean":"2024-04-25T17:34:39.004496+0000","last_became_active":"2024-04-25T17:32:35.073190+0000","last_became_peered":"2024-04-25T17:32:35.073190+0000","last_unstale":"2024-04-25T17:34:39.004496+0000","last_undegraded":"2024-04-25T17:34:39.004496+0000","last_fullsized":"2024-04-25T17:34:39.004496+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:19.841978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":142,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:41.306226+0000","last_change":"2024-04-25T17:32:32.044359+0000","last_active":"2024-04-25T17:34:41.306226+0000","last_peered":"2024-04-25T17:34:41.306226+0000","last_clean":"2024-04-25T17:34:41.306226+0000","last_became_active":"2024-04-25T17:32:32.044024+0000","last_became_peered":"2024-04-25T17:32:32.044024+0000","last_unstale":"2024-04-25T17:34:41.306226+0000","last_undegraded":"2024-04-25T17:34:41.306226+0000","last_fullsized":"2024-04-25T17:34:41.306226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:58.940554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.004819+0000","last_change":"2024-04-25T17:33:10.851665+0000","last_active":"2024-04-25T17:34:39.004819+0000","last_peered":"2024-04-25T17:34:39.004819+0000","last_clean":"2024-04-25T17:34:39.004819+0000","last_became_active":"2024-04-25T17:33:10.851217+0000","last_became_peered":"2024-04-25T17:33:10.851217+0000","last_unstale":"2024-04-25T17:34:39.004819+0000","last_undegraded":"2024-04-25T17:34:39.004819+0000","last_fullsized":"2024-04-25T17:34:39.004819+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:58.436715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.194439+0000","last_change":"2024-04-25T17:32:32.285823+0000","last_active":"2024-04-25T17:33:10.194439+0000","last_peered":"2024-04-25T17:33:10.194439+0000","last_clean":"2024-04-25T17:33:10.194439+0000","last_became_active":"2024-04-25T17:32:32.285416+0000","last_became_peered":"2024-04-25T17:32:32.285416+0000","last_unstale":"2024-04-25T17:33:10.194439+0000","last_undegraded":"2024-04-25T17:33:10.194439+0000","last_fullsized":"2024-04-25T17:33:10.194439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:24.473406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.001850+0000","last_change":"2024-04-25T17:32:35.072626+0000","last_active":"2024-04-25T17:34:39.001850+0000","last_peered":"2024-04-25T17:34:39.001850+0000","last_clean":"2024-04-25T17:34:39.001850+0000","last_became_active":"2024-04-25T17:32:35.072264+0000","last_became_peered":"2024-04-25T17:32:35.072264+0000","last_unstale":"2024-04-25T17:34:39.001850+0000","last_undegraded":"2024-04-25T17:34:39.001850+0000","last_fullsized":"2024-04-25T17:34:39.001850+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:54.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195985+0000","last_change":"2024-04-25T17:32:32.045589+0000","last_active":"2024-04-25T17:33:10.195985+0000","last_peered":"2024-04-25T17:33:10.195985+0000","last_clean":"2024-04-25T17:33:10.195985+0000","last_became_active":"2024-04-25T17:32:32.044838+0000","last_became_peered":"2024-04-25T17:32:32.044838+0000","last_unstale":"2024-04-25T17:33:10.195985+0000","last_undegraded":"2024-04-25T17:33:10.195985+0000","last_fullsized":"2024-04-25T17:33:10.195985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:05.748335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.993117+0000","last_change":"2024-04-25T17:32:35.075077+0000","last_active":"2024-04-25T17:34:38.993117+0000","last_peered":"2024-04-25T17:34:38.993117+0000","last_clean":"2024-04-25T17:34:38.993117+0000","last_became_active":"2024-04-25T17:32:35.074622+0000","last_became_peered":"2024-04-25T17:32:35.074622+0000","last_unstale":"2024-04-25T17:34:38.993117+0000","last_undegraded":"2024-04-25T17:34:38.993117+0000","last_fullsized":"2024-04-25T17:34:38.993117+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:53.435637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.185863+0000","last_change":"2024-04-25T17:32:32.288707+0000","last_active":"2024-04-25T17:33:10.185863+0000","last_peered":"2024-04-25T17:33:10.185863+0000","last_clean":"2024-04-25T17:33:10.185863+0000","last_became_active":"2024-04-25T17:32:32.288321+0000","last_became_peered":"2024-04-25T17:32:32.288321+0000","last_unstale":"2024-04-25T17:33:10.185863+0000","last_undegraded":"2024-04-25T17:33:10.185863+0000","last_fullsized":"2024-04-25T17:33:10.185863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:00.113662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'5","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.005205+0000","last_change":"2024-04-25T17:32:35.072100+0000","last_active":"2024-04-25T17:34:39.005205+0000","last_peered":"2024-04-25T17:34:39.005205+0000","last_clean":"2024-04-25T17:34:39.005205+0000","last_became_active":"2024-04-25T17:32:35.070706+0000","last_became_peered":"2024-04-25T17:32:35.070706+0000","last_unstale":"2024-04-25T17:34:39.005205+0000","last_undegraded":"2024-04-25T17:34:39.005205+0000","last_fullsized":"2024-04-25T17:34:39.005205+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:18.717385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190264+0000","last_change":"2024-04-25T17:32:32.042700+0000","last_active":"2024-04-25T17:33:10.190264+0000","last_peered":"2024-04-25T17:33:10.190264+0000","last_clean":"2024-04-25T17:33:10.190264+0000","last_became_active":"2024-04-25T17:32:32.042383+0000","last_became_peered":"2024-04-25T17:32:32.042383+0000","last_unstale":"2024-04-25T17:33:10.190264+0000","last_undegraded":"2024-04-25T17:33:10.190264+0000","last_fullsized":"2024-04-25T17:33:10.190264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:11.128322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.003691+0000","last_change":"2024-04-25T17:32:35.072336+0000","last_active":"2024-04-25T17:34:39.003691+0000","last_peered":"2024-04-25T17:34:39.003691+0000","last_clean":"2024-04-25T17:34:39.003691+0000","last_became_active":"2024-04-25T17:32:35.069830+0000","last_became_peered":"2024-04-25T17:32:35.069830+0000","last_unstale":"2024-04-25T17:34:39.003691+0000","last_undegraded":"2024-04-25T17:34:39.003691+0000","last_fullsized":"2024-04-25T17:34:39.003691+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:12.887428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.996487+0000","last_change":"2024-04-25T17:32:35.071375+0000","last_active":"2024-04-25T17:34:38.996487+0000","last_peered":"2024-04-25T17:34:38.996487+0000","last_clean":"2024-04-25T17:34:38.996487+0000","last_became_active":"2024-04-25T17:32:35.071063+0000","last_became_peered":"2024-04-25T17:32:35.071063+0000","last_unstale":"2024-04-25T17:34:38.996487+0000","last_undegraded":"2024-04-25T17:34:38.996487+0000","last_fullsized":"2024-04-25T17:34:38.996487+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:41.917038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":133,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:37.298911+0000","last_change":"2024-04-25T17:32:32.286941+0000","last_active":"2024-04-25T17:34:37.298911+0000","last_peered":"2024-04-25T17:34:37.298911+0000","last_clean":"2024-04-25T17:34:37.298911+0000","last_became_active":"2024-04-25T17:32:32.286608+0000","last_became_peered":"2024-04-25T17:32:32.286608+0000","last_unstale":"2024-04-25T17:34:37.298911+0000","last_undegraded":"2024-04-25T17:34:37.298911+0000","last_fullsized":"2024-04-25T17:34:37.298911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.559724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.999111+0000","last_change":"2024-04-25T17:32:35.068701+0000","last_active":"2024-04-25T17:34:38.999111+0000","last_peered":"2024-04-25T17:34:38.999111+0000","last_clean":"2024-04-25T17:34:38.999111+0000","last_became_active":"2024-04-25T17:32:35.068418+0000","last_became_peered":"2024-04-25T17:32:35.068418+0000","last_unstale":"2024-04-25T17:34:38.999111+0000","last_undegraded":"2024-04-25T17:34:38.999111+0000","last_fullsized":"2024-04-25T17:34:38.999111+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:42:25.431527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605706+0000","last_change":"2024-04-25T17:32:32.042167+0000","last_active":"2024-04-25T17:33:10.605706+0000","last_peered":"2024-04-25T17:33:10.605706+0000","last_clean":"2024-04-25T17:33:10.605706+0000","last_became_active":"2024-04-25T17:32:32.041765+0000","last_became_peered":"2024-04-25T17:32:32.041765+0000","last_unstale":"2024-04-25T17:33:10.605706+0000","last_undegraded":"2024-04-25T17:33:10.605706+0000","last_fullsized":"2024-04-25T17:33:10.605706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:28.901648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":360,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190045+0000","last_change":"2024-04-25T17:31:05.391870+0000","last_active":"2024-04-25T17:33:10.190045+0000","last_peered":"2024-04-25T17:33:10.190045+0000","last_clean":"2024-04-25T17:33:10.190045+0000","last_became_active":"2024-04-25T17:31:05.391426+0000","last_became_peered":"2024-04-25T17:31:05.391426+0000","last_unstale":"2024-04-25T17:33:10.190045+0000","last_undegraded":"2024-04-25T17:33:10.190045+0000","last_fullsized":"2024-04-25T17:33:10.190045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_clean_scrub_stamp":"2024-04-25T17:27:37.393149+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:18.400543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.190518+0000","last_change":"2024-04-25T17:33:10.190518+0000","last_active":"2024-04-25T17:33:10.190518+0000","last_peered":"2024-04-25T17:33:10.190518+0000","last_clean":"2024-04-25T17:33:10.190518+0000","last_became_active":"2024-04-25T17:33:10.190166+0000","last_became_peered":"2024-04-25T17:33:10.190166+0000","last_unstale":"2024-04-25T17:33:10.190518+0000","last_undegraded":"2024-04-25T17:33:10.190518+0000","last_fullsized":"2024-04-25T17:33:10.190518+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:45.707078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.197863+0000","last_change":"2024-04-25T17:33:10.197412+0000","last_active":"2024-04-25T17:33:10.197863+0000","last_peered":"2024-04-25T17:33:10.197863+0000","last_clean":"2024-04-25T17:33:10.197863+0000","last_became_active":"2024-04-25T17:33:10.194388+0000","last_became_peered":"2024-04-25T17:33:10.194388+0000","last_unstale":"2024-04-25T17:33:10.197863+0000","last_undegraded":"2024-04-25T17:33:10.197863+0000","last_fullsized":"2024-04-25T17:33:10.197863+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:35.455072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.195750+0000","last_change":"2024-04-25T17:32:32.285845+0000","last_active":"2024-04-25T17:33:10.195750+0000","last_peered":"2024-04-25T17:33:10.195750+0000","last_clean":"2024-04-25T17:33:10.195750+0000","last_became_active":"2024-04-25T17:32:32.285386+0000","last_became_peered":"2024-04-25T17:32:32.285386+0000","last_unstale":"2024-04-25T17:33:10.195750+0000","last_undegraded":"2024-04-25T17:33:10.195750+0000","last_fullsized":"2024-04-25T17:33:10.195750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:12.115900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187921+0000","last_change":"2024-04-25T17:32:35.065808+0000","last_active":"2024-04-25T17:33:10.187921+0000","last_peered":"2024-04-25T17:33:10.187921+0000","last_clean":"2024-04-25T17:33:10.187921+0000","last_became_active":"2024-04-25T17:32:35.065504+0000","last_became_peered":"2024-04-25T17:32:35.065504+0000","last_unstale":"2024-04-25T17:33:10.187921+0000","last_undegraded":"2024-04-25T17:33:10.187921+0000","last_fullsized":"2024-04-25T17:33:10.187921+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:33.438281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:22.083859+0000","last_change":"2024-04-25T17:32:32.045357+0000","last_active":"2024-04-25T17:33:22.083859+0000","last_peered":"2024-04-25T17:33:22.083859+0000","last_clean":"2024-04-25T17:33:22.083859+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:33:22.083859+0000","last_undegraded":"2024-04-25T17:33:22.083859+0000","last_fullsized":"2024-04-25T17:33:22.083859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:36.113130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.995101+0000","last_change":"2024-04-25T17:32:35.077889+0000","last_active":"2024-04-25T17:34:38.995101+0000","last_peered":"2024-04-25T17:34:38.995101+0000","last_clean":"2024-04-25T17:34:38.995101+0000","last_became_active":"2024-04-25T17:32:35.076470+0000","last_became_peered":"2024-04-25T17:32:35.076470+0000","last_unstale":"2024-04-25T17:34:38.995101+0000","last_undegraded":"2024-04-25T17:34:38.995101+0000","last_fullsized":"2024-04-25T17:34:38.995101+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:45.457146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.605918+0000","last_change":"2024-04-25T17:32:32.406826+0000","last_active":"2024-04-25T17:33:10.605918+0000","last_peered":"2024-04-25T17:33:10.605918+0000","last_clean":"2024-04-25T17:33:10.605918+0000","last_became_active":"2024-04-25T17:32:32.406454+0000","last_became_peered":"2024-04-25T17:32:32.406454+0000","last_unstale":"2024-04-25T17:33:10.605918+0000","last_undegraded":"2024-04-25T17:33:10.605918+0000","last_fullsized":"2024-04-25T17:33:10.605918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:21:11.730942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.998146+0000","last_change":"2024-04-25T17:32:35.075961+0000","last_active":"2024-04-25T17:34:38.998146+0000","last_peered":"2024-04-25T17:34:38.998146+0000","last_clean":"2024-04-25T17:34:38.998146+0000","last_became_active":"2024-04-25T17:32:35.075595+0000","last_became_peered":"2024-04-25T17:32:35.075595+0000","last_unstale":"2024-04-25T17:34:38.998146+0000","last_undegraded":"2024-04-25T17:34:38.998146+0000","last_fullsized":"2024-04-25T17:34:38.998146+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:25.513096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187458+0000","last_change":"2024-04-25T17:32:32.289123+0000","last_active":"2024-04-25T17:33:10.187458+0000","last_peered":"2024-04-25T17:33:10.187458+0000","last_clean":"2024-04-25T17:33:10.187458+0000","last_became_active":"2024-04-25T17:32:32.288783+0000","last_became_peered":"2024-04-25T17:32:32.288783+0000","last_unstale":"2024-04-25T17:33:10.187458+0000","last_undegraded":"2024-04-25T17:33:10.187458+0000","last_fullsized":"2024-04-25T17:33:10.187458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:52.222440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.965579+0000","last_change":"2024-04-25T17:32:35.066364+0000","last_active":"2024-04-25T17:34:38.965579+0000","last_peered":"2024-04-25T17:34:38.965579+0000","last_clean":"2024-04-25T17:34:38.965579+0000","last_became_active":"2024-04-25T17:32:35.065489+0000","last_became_peered":"2024-04-25T17:32:35.065489+0000","last_unstale":"2024-04-25T17:34:38.965579+0000","last_undegraded":"2024-04-25T17:34:38.965579+0000","last_fullsized":"2024-04-25T17:34:38.965579+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:16.817796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.189936+0000","last_change":"2024-04-25T17:32:32.040662+0000","last_active":"2024-04-25T17:33:10.189936+0000","last_peered":"2024-04-25T17:33:10.189936+0000","last_clean":"2024-04-25T17:33:10.189936+0000","last_became_active":"2024-04-25T17:32:32.040334+0000","last_became_peered":"2024-04-25T17:32:32.040334+0000","last_unstale":"2024-04-25T17:33:10.189936+0000","last_undegraded":"2024-04-25T17:33:10.189936+0000","last_fullsized":"2024-04-25T17:33:10.189936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:03.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.999502+0000","last_change":"2024-04-25T17:32:35.079183+0000","last_active":"2024-04-25T17:34:38.999502+0000","last_peered":"2024-04-25T17:34:38.999502+0000","last_clean":"2024-04-25T17:34:38.999502+0000","last_became_active":"2024-04-25T17:32:35.078472+0000","last_became_peered":"2024-04-25T17:32:35.078472+0000","last_unstale":"2024-04-25T17:34:38.999502+0000","last_undegraded":"2024-04-25T17:34:38.999502+0000","last_fullsized":"2024-04-25T17:34:38.999502+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:39:59.120020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.201283+0000","last_change":"2024-04-25T17:33:10.201283+0000","last_active":"2024-04-25T17:33:10.201283+0000","last_peered":"2024-04-25T17:33:10.201283+0000","last_clean":"2024-04-25T17:33:10.201283+0000","last_became_active":"2024-04-25T17:33:10.196192+0000","last_became_peered":"2024-04-25T17:33:10.196192+0000","last_unstale":"2024-04-25T17:33:10.201283+0000","last_undegraded":"2024-04-25T17:33:10.201283+0000","last_fullsized":"2024-04-25T17:33:10.201283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:03.716366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.000372+0000","last_change":"2024-04-25T17:32:35.081593+0000","last_active":"2024-04-25T17:34:39.000372+0000","last_peered":"2024-04-25T17:34:39.000372+0000","last_clean":"2024-04-25T17:34:39.000372+0000","last_became_active":"2024-04-25T17:32:35.081248+0000","last_became_peered":"2024-04-25T17:32:35.081248+0000","last_unstale":"2024-04-25T17:34:39.000372+0000","last_undegraded":"2024-04-25T17:34:39.000372+0000","last_fullsized":"2024-04-25T17:34:39.000372+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:41:01.177899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.198901+0000","last_change":"2024-04-25T17:33:10.198901+0000","last_active":"2024-04-25T17:33:10.198901+0000","last_peered":"2024-04-25T17:33:10.198901+0000","last_clean":"2024-04-25T17:33:10.198901+0000","last_became_active":"2024-04-25T17:33:10.192191+0000","last_became_peered":"2024-04-25T17:33:10.192191+0000","last_unstale":"2024-04-25T17:33:10.198901+0000","last_undegraded":"2024-04-25T17:33:10.198901+0000","last_fullsized":"2024-04-25T17:33:10.198901+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:39.760642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.003272+0000","last_change":"2024-04-25T17:32:35.068592+0000","last_active":"2024-04-25T17:34:39.003272+0000","last_peered":"2024-04-25T17:34:39.003272+0000","last_clean":"2024-04-25T17:34:39.003272+0000","last_became_active":"2024-04-25T17:32:35.068240+0000","last_became_peered":"2024-04-25T17:32:35.068240+0000","last_unstale":"2024-04-25T17:34:39.003272+0000","last_undegraded":"2024-04-25T17:34:39.003272+0000","last_fullsized":"2024-04-25T17:34:39.003272+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:48.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.188180+0000","last_change":"2024-04-25T17:32:32.406702+0000","last_active":"2024-04-25T17:33:10.188180+0000","last_peered":"2024-04-25T17:33:10.188180+0000","last_clean":"2024-04-25T17:33:10.188180+0000","last_became_active":"2024-04-25T17:32:32.406263+0000","last_became_peered":"2024-04-25T17:32:32.406263+0000","last_unstale":"2024-04-25T17:33:10.188180+0000","last_undegraded":"2024-04-25T17:33:10.188180+0000","last_fullsized":"2024-04-25T17:33:10.188180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:27.627002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.000540+0000","last_change":"2024-04-25T17:32:35.073585+0000","last_active":"2024-04-25T17:34:39.000540+0000","last_peered":"2024-04-25T17:34:39.000540+0000","last_clean":"2024-04-25T17:34:39.000540+0000","last_became_active":"2024-04-25T17:32:35.072902+0000","last_became_peered":"2024-04-25T17:32:35.072902+0000","last_unstale":"2024-04-25T17:34:39.000540+0000","last_undegraded":"2024-04-25T17:34:39.000540+0000","last_fullsized":"2024-04-25T17:34:39.000540+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:17.277129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.193299+0000","last_change":"2024-04-25T17:32:32.042546+0000","last_active":"2024-04-25T17:33:10.193299+0000","last_peered":"2024-04-25T17:33:10.193299+0000","last_clean":"2024-04-25T17:33:10.193299+0000","last_became_active":"2024-04-25T17:32:32.042191+0000","last_became_peered":"2024-04-25T17:32:32.042191+0000","last_unstale":"2024-04-25T17:33:10.193299+0000","last_undegraded":"2024-04-25T17:33:10.193299+0000","last_fullsized":"2024-04-25T17:33:10.193299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:58.126260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.993721+0000","last_change":"2024-04-25T17:32:35.077067+0000","last_active":"2024-04-25T17:34:38.993721+0000","last_peered":"2024-04-25T17:34:38.993721+0000","last_clean":"2024-04-25T17:34:38.993721+0000","last_became_active":"2024-04-25T17:32:35.076701+0000","last_became_peered":"2024-04-25T17:32:35.076701+0000","last_unstale":"2024-04-25T17:34:38.993721+0000","last_undegraded":"2024-04-25T17:34:38.993721+0000","last_fullsized":"2024-04-25T17:34:38.993721+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:06.646419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.610032+0000","last_change":"2024-04-25T17:33:10.610032+0000","last_active":"2024-04-25T17:33:10.610032+0000","last_peered":"2024-04-25T17:33:10.610032+0000","last_clean":"2024-04-25T17:33:10.610032+0000","last_became_active":"2024-04-25T17:33:10.607260+0000","last_became_peered":"2024-04-25T17:33:10.607260+0000","last_unstale":"2024-04-25T17:33:10.610032+0000","last_undegraded":"2024-04-25T17:33:10.610032+0000","last_fullsized":"2024-04-25T17:33:10.610032+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:42:41.509257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.001696+0000","last_change":"2024-04-25T17:32:35.071351+0000","last_active":"2024-04-25T17:34:39.001696+0000","last_peered":"2024-04-25T17:34:39.001696+0000","last_clean":"2024-04-25T17:34:39.001696+0000","last_became_active":"2024-04-25T17:32:35.070559+0000","last_became_peered":"2024-04-25T17:32:35.070559+0000","last_unstale":"2024-04-25T17:34:39.001696+0000","last_undegraded":"2024-04-25T17:34:39.001696+0000","last_fullsized":"2024-04-25T17:34:39.001696+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:23.864826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.000743+0000","last_change":"2024-04-25T17:32:35.073326+0000","last_active":"2024-04-25T17:34:39.000743+0000","last_peered":"2024-04-25T17:34:39.000743+0000","last_clean":"2024-04-25T17:34:39.000743+0000","last_became_active":"2024-04-25T17:32:35.072904+0000","last_became_peered":"2024-04-25T17:32:35.072904+0000","last_unstale":"2024-04-25T17:34:39.000743+0000","last_undegraded":"2024-04-25T17:34:39.000743+0000","last_fullsized":"2024-04-25T17:34:39.000743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:50.586127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.001181+0000","last_change":"2024-04-25T17:33:10.195444+0000","last_active":"2024-04-25T17:34:39.001181+0000","last_peered":"2024-04-25T17:34:39.001181+0000","last_clean":"2024-04-25T17:34:39.001181+0000","last_became_active":"2024-04-25T17:33:10.195048+0000","last_became_peered":"2024-04-25T17:33:10.195048+0000","last_unstale":"2024-04-25T17:34:39.001181+0000","last_undegraded":"2024-04-25T17:34:39.001181+0000","last_fullsized":"2024-04-25T17:34:39.001181+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:03.098618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.993532+0000","last_change":"2024-04-25T17:32:35.072053+0000","last_active":"2024-04-25T17:34:38.993532+0000","last_peered":"2024-04-25T17:34:38.993532+0000","last_clean":"2024-04-25T17:34:38.993532+0000","last_became_active":"2024-04-25T17:32:35.071712+0000","last_became_peered":"2024-04-25T17:32:35.071712+0000","last_unstale":"2024-04-25T17:34:38.993532+0000","last_undegraded":"2024-04-25T17:34:38.993532+0000","last_fullsized":"2024-04-25T17:34:38.993532+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:42.928854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.004748+0000","last_change":"2024-04-25T17:32:35.061408+0000","last_active":"2024-04-25T17:34:39.004748+0000","last_peered":"2024-04-25T17:34:39.004748+0000","last_clean":"2024-04-25T17:34:39.004748+0000","last_became_active":"2024-04-25T17:32:35.061105+0000","last_became_peered":"2024-04-25T17:32:35.061105+0000","last_unstale":"2024-04-25T17:34:39.004748+0000","last_undegraded":"2024-04-25T17:34:39.004748+0000","last_fullsized":"2024-04-25T17:34:39.004748+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:48.242157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.687122+0000","last_change":"2024-04-25T17:33:10.687122+0000","last_active":"2024-04-25T17:33:10.687122+0000","last_peered":"2024-04-25T17:33:10.687122+0000","last_clean":"2024-04-25T17:33:10.687122+0000","last_became_active":"2024-04-25T17:33:10.686755+0000","last_became_peered":"2024-04-25T17:33:10.686755+0000","last_unstale":"2024-04-25T17:33:10.687122+0000","last_undegraded":"2024-04-25T17:33:10.687122+0000","last_fullsized":"2024-04-25T17:33:10.687122+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.999655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.996432+0000","last_change":"2024-04-25T17:32:35.077433+0000","last_active":"2024-04-25T17:34:38.996432+0000","last_peered":"2024-04-25T17:34:38.996432+0000","last_clean":"2024-04-25T17:34:38.996432+0000","last_became_active":"2024-04-25T17:32:35.076201+0000","last_became_peered":"2024-04-25T17:32:35.076201+0000","last_unstale":"2024-04-25T17:34:38.996432+0000","last_undegraded":"2024-04-25T17:34:38.996432+0000","last_fullsized":"2024-04-25T17:34:38.996432+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:42.677022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.006228+0000","last_change":"2024-04-25T17:32:35.068017+0000","last_active":"2024-04-25T17:34:39.006228+0000","last_peered":"2024-04-25T17:34:39.006228+0000","last_clean":"2024-04-25T17:34:39.006228+0000","last_became_active":"2024-04-25T17:32:35.067647+0000","last_became_peered":"2024-04-25T17:32:35.067647+0000","last_unstale":"2024-04-25T17:34:39.006228+0000","last_undegraded":"2024-04-25T17:34:39.006228+0000","last_fullsized":"2024-04-25T17:34:39.006228+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:28:11.339402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.977884+0000","last_change":"2024-04-25T17:32:35.075192+0000","last_active":"2024-04-25T17:34:38.977884+0000","last_peered":"2024-04-25T17:34:38.977884+0000","last_clean":"2024-04-25T17:34:38.977884+0000","last_became_active":"2024-04-25T17:32:35.074850+0000","last_became_peered":"2024-04-25T17:32:35.074850+0000","last_unstale":"2024-04-25T17:34:38.977884+0000","last_undegraded":"2024-04-25T17:34:38.977884+0000","last_fullsized":"2024-04-25T17:34:38.977884+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:04:41.343312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.993125+0000","last_change":"2024-04-25T17:32:35.078324+0000","last_active":"2024-04-25T17:34:38.993125+0000","last_peered":"2024-04-25T17:34:38.993125+0000","last_clean":"2024-04-25T17:34:38.993125+0000","last_became_active":"2024-04-25T17:32:35.077958+0000","last_became_peered":"2024-04-25T17:32:35.077958+0000","last_unstale":"2024-04-25T17:34:38.993125+0000","last_undegraded":"2024-04-25T17:34:38.993125+0000","last_fullsized":"2024-04-25T17:34:38.993125+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:10.148060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.990152+0000","last_change":"2024-04-25T17:32:35.078023+0000","last_active":"2024-04-25T17:34:38.990152+0000","last_peered":"2024-04-25T17:34:38.990152+0000","last_clean":"2024-04-25T17:34:38.990152+0000","last_became_active":"2024-04-25T17:32:35.077702+0000","last_became_peered":"2024-04-25T17:32:35.077702+0000","last_unstale":"2024-04-25T17:34:38.990152+0000","last_undegraded":"2024-04-25T17:34:38.990152+0000","last_fullsized":"2024-04-25T17:34:38.990152+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:44:37.933179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:39.000683+0000","last_change":"2024-04-25T17:32:35.078388+0000","last_active":"2024-04-25T17:34:39.000683+0000","last_peered":"2024-04-25T17:34:39.000683+0000","last_clean":"2024-04-25T17:34:39.000683+0000","last_became_active":"2024-04-25T17:32:35.078058+0000","last_became_peered":"2024-04-25T17:32:35.078058+0000","last_unstale":"2024-04-25T17:34:39.000683+0000","last_undegraded":"2024-04-25T17:34:39.000683+0000","last_fullsized":"2024-04-25T17:34:39.000683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:05.699542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:33:10.187895+0000","last_change":"2024-04-25T17:32:35.068532+0000","last_active":"2024-04-25T17:33:10.187895+0000","last_peered":"2024-04-25T17:33:10.187895+0000","last_clean":"2024-04-25T17:33:10.187895+0000","last_became_active":"2024-04-25T17:32:35.068225+0000","last_became_peered":"2024-04-25T17:32:35.068225+0000","last_unstale":"2024-04-25T17:33:10.187895+0000","last_undegraded":"2024-04-25T17:33:10.187895+0000","last_fullsized":"2024-04-25T17:33:10.187895+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:09.911999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T17:34:38.999091+0000","last_change":"2024-04-25T17:32:35.065526+0000","last_active":"2024-04-25T17:34:38.999091+0000","last_peered":"2024-04-25T17:34:38.999091+0000","last_clean":"2024-04-25T17:34:38.999091+0000","last_became_active":"2024-04-25T17:32:35.065146+0000","last_became_peered":"2024-04-25T17:32:35.065146+0000","last_unstale":"2024-04-25T17:34:38.999091+0000","last_undegraded":"2024-04-25T17:34:38.999091+0000","last_fullsized":"2024-04-25T17:34:38.999091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:33.662055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280256,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66726,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":665,"num_read_kb":401,"num_write":172,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":200178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351016,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":482924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1137521,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776181,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":483160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037816,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":483080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299455,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":1075151,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462980,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":482549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":544926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":483820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":546276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":482860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":483518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1064960,"data_stored":1074303,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34007,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T17:34:44.894 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T17:34:44.894 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T17:34:44.894 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T17:34:44.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T17:34:45.138 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:45.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:45 smithi098 ceph-mon[29192]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:34:45.467 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:45 smithi203 ceph-mon[32834]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:34:45.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:45 smithi173 ceph-mon[32856]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:34:46.303 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:46 smithi098 ceph-mon[29192]: pgmap v369: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T17:34:46.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:46 smithi203 ceph-mon[32834]: pgmap v369: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T17:34:46.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:46 smithi173 ceph-mon[32856]: pgmap v369: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T17:34:46.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.880+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/620246962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c4103450 msgr2=0x7fd7c410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:46.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.880+0000 7fd7cb4ee700 1 --2- 172.21.15.98:0/620246962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c4103450 0x7fd7c410f8b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc009990 tx=0x7fd7bc009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:46.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.880+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/620246962 shutdown_connections 2024-04-25T17:34:46.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.880+0000 7fd7cb4ee700 1 --2- 172.21.15.98:0/620246962 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c4103450 0x7fd7c410f8b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:46.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.880+0000 7fd7cb4ee700 1 --2- 172.21.15.98:0/620246962 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7c4108a50 0x7fd7c4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:46.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.880+0000 7fd7cb4ee700 1 --2- 172.21.15.98:0/620246962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c4102a90 0x7fd7c4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:46.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.880+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/620246962 >> 172.21.15.98:0/620246962 conn(0x7fd7c4079ba0 msgr2=0x7fd7c4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:46.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.881+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/620246962 shutdown_connections 2024-04-25T17:34:46.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.881+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/620246962 wait complete. 2024-04-25T17:34:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.881+0000 7fd7cb4ee700 1 Processor -- start 2024-04-25T17:34:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.881+0000 7fd7cb4ee700 1 -- start start 2024-04-25T17:34:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7cb4ee700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7c4102a90 0x7fd7c41990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7cb4ee700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c4103450 0x7fd7c4199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7cb4ee700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c4108a50 0x7fd7c419d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:46.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7cb4ee700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c4112c20 con 0x7fd7c4103450 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7cb4ee700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c4112aa0 con 0x7fd7c4102a90 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7cb4ee700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c4112da0 con 0x7fd7c4108a50 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c8a89700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c4103450 0x7fd7c4199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c8a89700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c4103450 0x7fd7c4199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47156/0 (socket says 172.21.15.98:47156) 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c8a89700 1 -- 172.21.15.98:0/2686162718 learned_addr learned my addr 172.21.15.98:0/2686162718 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c9a8b700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c4108a50 0x7fd7c419d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c928a700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7c4102a90 0x7fd7c41990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c8a89700 1 -- 172.21.15.98:0/2686162718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c4108a50 msgr2=0x7fd7c419d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c8a89700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c4108a50 0x7fd7c419d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c8a89700 1 -- 172.21.15.98:0/2686162718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7c4102a90 msgr2=0x7fd7c41990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c8a89700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7c4102a90 0x7fd7c41990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c8a89700 1 -- 172.21.15.98:0/2686162718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c419e100 con 0x7fd7c4103450 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c928a700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7c4102a90 0x7fd7c41990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:34:46.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c8a89700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c4103450 0x7fd7c4199630 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8007f10 tx=0x7fd7b800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:46.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c27fc700 1 -- 172.21.15.98:0/2686162718 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b800dbd0 con 0x7fd7c4103450 2024-04-25T17:34:46.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/2686162718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c419e3f0 con 0x7fd7c4103450 2024-04-25T17:34:46.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c27fc700 1 -- 172.21.15.98:0/2686162718 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7b800dd70 con 0x7fd7c4103450 2024-04-25T17:34:46.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.882+0000 7fd7c27fc700 1 -- 172.21.15.98:0/2686162718 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8005a10 con 0x7fd7c4103450 2024-04-25T17:34:46.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.883+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/2686162718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c41a5c70 con 0x7fd7c4103450 2024-04-25T17:34:46.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.883+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/2686162718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7c4069730 con 0x7fd7c4103450 2024-04-25T17:34:46.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.885+0000 7fd7c27fc700 1 -- 172.21.15.98:0/2686162718 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd7b801a020 con 0x7fd7c4103450 2024-04-25T17:34:46.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.885+0000 7fd7c27fc700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7ac0a3c40 0x7fd7ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:46.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.885+0000 7fd7c928a700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7ac0a3c40 0x7fd7ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:46.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.886+0000 7fd7c27fc700 1 -- 172.21.15.98:0/2686162718 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fd7b80cfc40 con 0x7fd7c4103450 2024-04-25T17:34:46.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.886+0000 7fd7c928a700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7ac0a3c40 0x7fd7ac0a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4005fd0 tx=0x7fd7b400a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:46.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:46.890+0000 7fd7c27fc700 1 -- 172.21.15.98:0/2686162718 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd7b8010070 con 0x7fd7c4103450 2024-04-25T17:34:47.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.135+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/2686162718 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd7c419a450 con 0x7fd7c4103450 2024-04-25T17:34:47.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.135+0000 7fd7c27fc700 1 -- 172.21.15.98:0/2686162718 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd7b809c5d0 con 0x7fd7c4103450 2024-04-25T17:34:47.137 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:34:47.137 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T17:34:47.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.137+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/2686162718 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7ac0a3c40 msgr2=0x7fd7ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.137+0000 7fd7cb4ee700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7ac0a3c40 0x7fd7ac0a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4005fd0 tx=0x7fd7b400a5e0 comp rx=0 tx=0).stop 2024-04-25T17:34:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.137+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/2686162718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c4103450 msgr2=0x7fd7c4199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.137+0000 7fd7cb4ee700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c4103450 0x7fd7c4199630 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8007f10 tx=0x7fd7b800d3b0 comp rx=0 tx=0).stop 2024-04-25T17:34:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.137+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/2686162718 shutdown_connections 2024-04-25T17:34:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.137+0000 7fd7cb4ee700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fd7ac0a3c40 0x7fd7ac0a6100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.137+0000 7fd7cb4ee700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c4108a50 0x7fd7c419d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.137+0000 7fd7cb4ee700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7c4102a90 0x7fd7c41990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.137+0000 7fd7cb4ee700 1 --2- 172.21.15.98:0/2686162718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7c4103450 0x7fd7c4199630 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.137+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/2686162718 >> 172.21.15.98:0/2686162718 conn(0x7fd7c4079ba0 msgr2=0x7fd7c41017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:47.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.138+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/2686162718 shutdown_connections 2024-04-25T17:34:47.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:47.138+0000 7fd7cb4ee700 1 -- 172.21.15.98:0/2686162718 wait complete. 2024-04-25T17:34:47.840 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T17:34:47.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:34:48.082 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:48.204 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:48 smithi098 ceph-mon[29192]: pgmap v370: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:34:48.204 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:48 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2686162718' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T17:34:48.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:48 smithi203 ceph-mon[32834]: pgmap v370: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:34:48.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:48 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2686162718' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T17:34:48.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:48 smithi173 ceph-mon[32856]: pgmap v370: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:34:48.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:48 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2686162718' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T17:34:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.748+0000 7fab19cd2700 1 -- 172.21.15.98:0/2913368864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab14101970 msgr2=0x7fab14101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.748+0000 7fab19cd2700 1 --2- 172.21.15.98:0/2913368864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab14101970 0x7fab14101df0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7faafc009990 tx=0x7faafc009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.748+0000 7fab19cd2700 1 -- 172.21.15.98:0/2913368864 shutdown_connections 2024-04-25T17:34:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.748+0000 7fab19cd2700 1 --2- 172.21.15.98:0/2913368864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab14102330 0x7fab1410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.748+0000 7fab19cd2700 1 --2- 172.21.15.98:0/2913368864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14107920 0x7fab14107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.748+0000 7fab19cd2700 1 --2- 172.21.15.98:0/2913368864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab14101970 0x7fab14101df0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.748+0000 7fab19cd2700 1 -- 172.21.15.98:0/2913368864 >> 172.21.15.98:0/2913368864 conn(0x7fab14078910 msgr2=0x7fab14078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.748+0000 7fab19cd2700 1 -- 172.21.15.98:0/2913368864 shutdown_connections 2024-04-25T17:34:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.748+0000 7fab19cd2700 1 -- 172.21.15.98:0/2913368864 wait complete. 2024-04-25T17:34:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab19cd2700 1 Processor -- start 2024-04-25T17:34:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab19cd2700 1 -- start start 2024-04-25T17:34:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab19cd2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14101970 0x7fab14199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab19cd2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab14102330 0x7fab14199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab19cd2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab14107920 0x7fab1419d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab19cd2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fab14111b00 con 0x7fab14102330 2024-04-25T17:34:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab19cd2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fab14111980 con 0x7fab14101970 2024-04-25T17:34:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab19cd2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fab14111c80 con 0x7fab14107920 2024-04-25T17:34:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab0affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab14102330 0x7fab14199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab0affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab14102330 0x7fab14199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47172/0 (socket says 172.21.15.98:47172) 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.749+0000 7fab0affd700 1 -- 172.21.15.98:0/1239577879 learned_addr learned my addr 172.21.15.98:0/1239577879 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab137fe700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14101970 0x7fab14199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab13fff700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab14107920 0x7fab1419d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab0affd700 1 -- 172.21.15.98:0/1239577879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab14107920 msgr2=0x7fab1419d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab0affd700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab14107920 0x7fab1419d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab0affd700 1 -- 172.21.15.98:0/1239577879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14101970 msgr2=0x7fab14199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab0affd700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14101970 0x7fab14199000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab0affd700 1 -- 172.21.15.98:0/1239577879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1419e010 con 0x7fab14102330 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab137fe700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14101970 0x7fab14199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab0affd700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab14102330 0x7fab14199540 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faafc0057f0 tx=0x7faafc003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab117fa700 1 -- 172.21.15.98:0/1239577879 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc027070 con 0x7fab14102330 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab19cd2700 1 -- 172.21.15.98:0/1239577879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1419e2a0 con 0x7fab14102330 2024-04-25T17:34:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab117fa700 1 -- 172.21.15.98:0/1239577879 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faafc0041c0 con 0x7fab14102330 2024-04-25T17:34:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab117fa700 1 -- 172.21.15.98:0/1239577879 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc022e60 con 0x7fab14102330 2024-04-25T17:34:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.750+0000 7fab19cd2700 1 -- 172.21.15.98:0/1239577879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab141a5b80 con 0x7fab14102330 2024-04-25T17:34:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.751+0000 7fab19cd2700 1 -- 172.21.15.98:0/1239577879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab14069730 con 0x7fab14102330 2024-04-25T17:34:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.752+0000 7fab117fa700 1 -- 172.21.15.98:0/1239577879 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faafc02a070 con 0x7fab14102330 2024-04-25T17:34:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.753+0000 7fab117fa700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab000a3c40 0x7fab000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.753+0000 7fab137fe700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab000a3c40 0x7fab000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.753+0000 7fab117fa700 1 -- 172.21.15.98:0/1239577879 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7faafc0d1690 con 0x7fab14102330 2024-04-25T17:34:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.753+0000 7fab137fe700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab000a3c40 0x7fab000a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fab04005fd0 tx=0x7fab0400a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:49.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:49.756+0000 7fab117fa700 1 -- 172.21.15.98:0/1239577879 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faafc013070 con 0x7fab14102330 2024-04-25T17:34:50.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.002+0000 7fab19cd2700 1 -- 172.21.15.98:0/1239577879 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab1402df80 con 0x7fab14102330 2024-04-25T17:34:50.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.002+0000 7fab117fa700 1 -- 172.21.15.98:0/1239577879 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11877 (secure 0 0 0) 0x7faafc09e110 con 0x7fab14102330 2024-04-25T17:34:50.004 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:34:50.004 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:32:59.314849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:34:50.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.004+0000 7fab19cd2700 1 -- 172.21.15.98:0/1239577879 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab000a3c40 msgr2=0x7fab000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:50.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.004+0000 7fab19cd2700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab000a3c40 0x7fab000a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fab04005fd0 tx=0x7fab0400a5e0 comp rx=0 tx=0).stop 2024-04-25T17:34:50.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.004+0000 7fab19cd2700 1 -- 172.21.15.98:0/1239577879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab14102330 msgr2=0x7fab14199540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:50.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.004+0000 7fab19cd2700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab14102330 0x7fab14199540 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faafc0057f0 tx=0x7faafc003730 comp rx=0 tx=0).stop 2024-04-25T17:34:50.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.005+0000 7fab19cd2700 1 -- 172.21.15.98:0/1239577879 shutdown_connections 2024-04-25T17:34:50.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.005+0000 7fab19cd2700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fab000a3c40 0x7fab000a6100 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:50.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.005+0000 7fab19cd2700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab14107920 0x7fab1419d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:50.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.005+0000 7fab19cd2700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14101970 0x7fab14199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:50.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.005+0000 7fab19cd2700 1 --2- 172.21.15.98:0/1239577879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab14102330 0x7fab14199540 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:50.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.005+0000 7fab19cd2700 1 -- 172.21.15.98:0/1239577879 >> 172.21.15.98:0/1239577879 conn(0x7fab14078910 msgr2=0x7fab1406b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:50.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.005+0000 7fab19cd2700 1 -- 172.21.15.98:0/1239577879 shutdown_connections 2024-04-25T17:34:50.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:50.005+0000 7fab19cd2700 1 -- 172.21.15.98:0/1239577879 wait complete. 2024-04-25T17:34:50.007 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 25 2024-04-25T17:34:50.325 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:50 smithi173 ceph-mon[32856]: pgmap v371: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-25T17:34:50.325 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:50 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1239577879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:34:50.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:50 smithi098 ceph-mon[29192]: pgmap v371: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-25T17:34:50.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:50 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1239577879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:34:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:50 smithi203 ceph-mon[32834]: pgmap v371: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-25T17:34:50.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:50 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1239577879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:34:50.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:34:50.923 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:52.298 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:52 smithi098 ceph-mon[29192]: pgmap v372: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:34:52.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:52 smithi203 ceph-mon[32834]: pgmap v372: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:34:52.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:52 smithi173 ceph-mon[32856]: pgmap v372: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T17:34:52.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.659+0000 7fba5bd58700 1 -- 172.21.15.98:0/1955104318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba541064e0 msgr2=0x7fba5410e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.659+0000 7fba5bd58700 1 --2- 172.21.15.98:0/1955104318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba541064e0 0x7fba5410e020 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00b0a0 tx=0x7fba4c00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.659+0000 7fba5bd58700 1 -- 172.21.15.98:0/1955104318 shutdown_connections 2024-04-25T17:34:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.659+0000 7fba5bd58700 1 --2- 172.21.15.98:0/1955104318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba54101640 0x7fba54101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.659+0000 7fba5bd58700 1 --2- 172.21.15.98:0/1955104318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba541064e0 0x7fba5410e020 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.659+0000 7fba5bd58700 1 --2- 172.21.15.98:0/1955104318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba54101ff0 0x7fba54105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.659+0000 7fba5bd58700 1 -- 172.21.15.98:0/1955104318 >> 172.21.15.98:0/1955104318 conn(0x7fba54076c60 msgr2=0x7fba54077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.659+0000 7fba5bd58700 1 -- 172.21.15.98:0/1955104318 shutdown_connections 2024-04-25T17:34:52.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.659+0000 7fba5bd58700 1 -- 172.21.15.98:0/1955104318 wait complete. 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.660+0000 7fba5bd58700 1 Processor -- start 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.660+0000 7fba5bd58700 1 -- start start 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5bd58700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba54101640 0x7fba54199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5bd58700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba54101ff0 0x7fba54199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5bd58700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba541064e0 0x7fba5419d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5bd58700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba54110a90 con 0x7fba541064e0 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5bd58700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fba54110910 con 0x7fba54101640 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5bd58700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba54110c10 con 0x7fba54101ff0 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5a2f5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba541064e0 0x7fba5419d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5a2f5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba541064e0 0x7fba5419d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47198/0 (socket says 172.21.15.98:47198) 2024-04-25T17:34:52.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5a2f5700 1 -- 172.21.15.98:0/1703407381 learned_addr learned my addr 172.21.15.98:0/1703407381 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba592f3700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba54101ff0 0x7fba54199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba59af4700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba54101640 0x7fba54199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5a2f5700 1 -- 172.21.15.98:0/1703407381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba54101ff0 msgr2=0x7fba54199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5a2f5700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba54101ff0 0x7fba54199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5a2f5700 1 -- 172.21.15.98:0/1703407381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba54101640 msgr2=0x7fba54199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5a2f5700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba54101640 0x7fba54199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5a2f5700 1 -- 172.21.15.98:0/1703407381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5419e060 con 0x7fba541064e0 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5a2f5700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba541064e0 0x7fba5419d920 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00b830 tx=0x7fba4c004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba4affd700 1 -- 172.21.15.98:0/1703407381 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c021070 con 0x7fba541064e0 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba4affd700 1 -- 172.21.15.98:0/1703407381 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba4c007e30 con 0x7fba541064e0 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba4affd700 1 -- 172.21.15.98:0/1703407381 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c004700 con 0x7fba541064e0 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.661+0000 7fba5bd58700 1 -- 172.21.15.98:0/1703407381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba5419e2f0 con 0x7fba541064e0 2024-04-25T17:34:52.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.662+0000 7fba5bd58700 1 -- 172.21.15.98:0/1703407381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba541a5bd0 con 0x7fba541064e0 2024-04-25T17:34:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.662+0000 7fba5bd58700 1 -- 172.21.15.98:0/1703407381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba54069730 con 0x7fba541064e0 2024-04-25T17:34:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.664+0000 7fba4affd700 1 -- 172.21.15.98:0/1703407381 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fba4c003d70 con 0x7fba541064e0 2024-04-25T17:34:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.664+0000 7fba4affd700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fba400a3c40 0x7fba400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.664+0000 7fba59af4700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fba400a3c40 0x7fba400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.664+0000 7fba4affd700 1 -- 172.21.15.98:0/1703407381 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fba4c00f020 con 0x7fba541064e0 2024-04-25T17:34:52.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.665+0000 7fba59af4700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fba400a3c40 0x7fba400a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fba4400bd70 tx=0x7fba4400b4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:52.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.668+0000 7fba4affd700 1 -- 172.21.15.98:0/1703407381 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba4c02a020 con 0x7fba541064e0 2024-04-25T17:34:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.911+0000 7fba5bd58700 1 -- 172.21.15.98:0/1703407381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba5402df80 con 0x7fba541064e0 2024-04-25T17:34:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.912+0000 7fba4affd700 1 -- 172.21.15.98:0/1703407381 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11877 (secure 0 0 0) 0x7fba4c09e730 con 0x7fba541064e0 2024-04-25T17:34:52.913 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:34:52.913 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:32:59.314849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:34:52.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 -- 172.21.15.98:0/1703407381 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fba400a3c40 msgr2=0x7fba400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:52.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fba400a3c40 0x7fba400a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fba4400bd70 tx=0x7fba4400b4c0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 -- 172.21.15.98:0/1703407381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba541064e0 msgr2=0x7fba5419d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:52.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba541064e0 0x7fba5419d920 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00b830 tx=0x7fba4c004ce0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 -- 172.21.15.98:0/1703407381 shutdown_connections 2024-04-25T17:34:52.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fba400a3c40 0x7fba400a6100 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba541064e0 0x7fba5419d920 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba54101640 0x7fba54199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 --2- 172.21.15.98:0/1703407381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba54101ff0 0x7fba54199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 -- 172.21.15.98:0/1703407381 >> 172.21.15.98:0/1703407381 conn(0x7fba54076c60 msgr2=0x7fba540ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 -- 172.21.15.98:0/1703407381 shutdown_connections 2024-04-25T17:34:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:52.914+0000 7fba5bd58700 1 -- 172.21.15.98:0/1703407381 wait complete. 2024-04-25T17:34:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 25 2024-04-25T17:34:53.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:53 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1703407381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:34:53.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:53 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1703407381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:34:53.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:53 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1703407381' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:34:53.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T17:34:53.839 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:54.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:54 smithi098 ceph-mon[29192]: pgmap v373: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T17:34:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:54 smithi203 ceph-mon[32834]: pgmap v373: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T17:34:54.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:54 smithi173 ceph-mon[32856]: pgmap v373: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T17:34:55.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.544+0000 7f25324c1700 1 -- 172.21.15.98:0/3935292489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f252c101970 msgr2=0x7f252c101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:55.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.544+0000 7f25324c1700 1 --2- 172.21.15.98:0/3935292489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f252c101970 0x7f252c101df0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2520009990 tx=0x7f2520009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:55.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.544+0000 7f25324c1700 1 -- 172.21.15.98:0/3935292489 shutdown_connections 2024-04-25T17:34:55.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.544+0000 7f25324c1700 1 --2- 172.21.15.98:0/3935292489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f252c107920 0x7f252c107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:55.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.544+0000 7f25324c1700 1 --2- 172.21.15.98:0/3935292489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f252c102330 0x7f252c10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:55.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.544+0000 7f25324c1700 1 --2- 172.21.15.98:0/3935292489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f252c101970 0x7f252c101df0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:55.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.544+0000 7f25324c1700 1 -- 172.21.15.98:0/3935292489 >> 172.21.15.98:0/3935292489 conn(0x7f252c0788f0 msgr2=0x7f252c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:55.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.545+0000 7f25324c1700 1 -- 172.21.15.98:0/3935292489 shutdown_connections 2024-04-25T17:34:55.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.545+0000 7f25324c1700 1 -- 172.21.15.98:0/3935292489 wait complete. 2024-04-25T17:34:55.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.545+0000 7f25324c1700 1 Processor -- start 2024-04-25T17:34:55.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.545+0000 7f25324c1700 1 -- start start 2024-04-25T17:34:55.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.545+0000 7f25324c1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f252c101970 0x7f252c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:55.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.545+0000 7f25324c1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f252c102330 0x7f252c19a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:55.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.545+0000 7f25324c1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f252c107920 0x7f252c1a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:55.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.545+0000 7f25324c1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f252c111bc0 con 0x7f252c107920 2024-04-25T17:34:55.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.545+0000 7f25324c1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f252c111a40 con 0x7f252c101970 2024-04-25T17:34:55.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.545+0000 7f25324c1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f252c111d40 con 0x7f252c102330 2024-04-25T17:34:55.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f2530a5e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f252c107920 0x7f252c1a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:55.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f2530a5e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f252c107920 0x7f252c1a5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33336/0 (socket says 172.21.15.98:33336) 2024-04-25T17:34:55.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f2530a5e700 1 -- 172.21.15.98:0/2228436645 learned_addr learned my addr 172.21.15.98:0/2228436645 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f252b7fe700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f252c102330 0x7f252c19a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f252bfff700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f252c101970 0x7f252c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f2530a5e700 1 -- 172.21.15.98:0/2228436645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f252c102330 msgr2=0x7f252c19a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f2530a5e700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f252c102330 0x7f252c19a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f2530a5e700 1 -- 172.21.15.98:0/2228436645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f252c101970 msgr2=0x7f252c19a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:55.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f2530a5e700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f252c101970 0x7f252c19a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f2530a5e700 1 -- 172.21.15.98:0/2228436645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c1a5680 con 0x7f252c107920 2024-04-25T17:34:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f252bfff700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f252c101970 0x7f252c19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:34:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f2530a5e700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f252c107920 0x7f252c1a5140 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f252400b970 tx=0x7f252400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f25297fa700 1 -- 172.21.15.98:0/2228436645 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2524019070 con 0x7f252c107920 2024-04-25T17:34:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f25324c1700 1 -- 172.21.15.98:0/2228436645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f252c1a58b0 con 0x7f252c107920 2024-04-25T17:34:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f25297fa700 1 -- 172.21.15.98:0/2228436645 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2524014440 con 0x7f252c107920 2024-04-25T17:34:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f25297fa700 1 -- 172.21.15.98:0/2228436645 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252400ce60 con 0x7f252c107920 2024-04-25T17:34:55.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.546+0000 7f25324c1700 1 -- 172.21.15.98:0/2228436645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f252c1a5dc0 con 0x7f252c107920 2024-04-25T17:34:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.547+0000 7f25324c1700 1 -- 172.21.15.98:0/2228436645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f252c069730 con 0x7f252c107920 2024-04-25T17:34:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.548+0000 7f25297fa700 1 -- 172.21.15.98:0/2228436645 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f252401c070 con 0x7f252c107920 2024-04-25T17:34:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.549+0000 7f25297fa700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25140a3c40 0x7f25140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.549+0000 7f252bfff700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25140a3c40 0x7f25140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.549+0000 7f25297fa700 1 -- 172.21.15.98:0/2228436645 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f25240cf840 con 0x7f252c107920 2024-04-25T17:34:55.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.549+0000 7f252bfff700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25140a3c40 0x7f25140a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f251c007900 tx=0x7f251c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:55.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.553+0000 7f25297fa700 1 -- 172.21.15.98:0/2228436645 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f252409c230 con 0x7f252c107920 2024-04-25T17:34:55.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.754+0000 7f25324c1700 1 -- 172.21.15.98:0/2228436645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f252c02df80 con 0x7f252c107920 2024-04-25T17:34:55.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.755+0000 7f25297fa700 1 -- 172.21.15.98:0/2228436645 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22686 (secure 0 0 0) 0x7f252409bb30 con 0x7f252c107920 2024-04-25T17:34:55.756 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:34:55.756 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:33:10.182926+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T17:32:30.267079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T17:32:33.956902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:34:55.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.757+0000 7f25324c1700 1 -- 172.21.15.98:0/2228436645 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25140a3c40 msgr2=0x7f25140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.757+0000 7f25324c1700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25140a3c40 0x7f25140a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f251c007900 tx=0x7f251c008040 comp rx=0 tx=0).stop 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.757+0000 7f25324c1700 1 -- 172.21.15.98:0/2228436645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f252c107920 msgr2=0x7f252c1a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.757+0000 7f25324c1700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f252c107920 0x7f252c1a5140 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f252400b970 tx=0x7f252400bd30 comp rx=0 tx=0).stop 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.758+0000 7f25324c1700 1 -- 172.21.15.98:0/2228436645 shutdown_connections 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.758+0000 7f25324c1700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f25140a3c40 0x7f25140a6100 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.758+0000 7f25324c1700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f252c107920 0x7f252c1a5140 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.758+0000 7f25324c1700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f252c101970 0x7f252c19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.758+0000 7f25324c1700 1 --2- 172.21.15.98:0/2228436645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f252c102330 0x7f252c19a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.758+0000 7f25324c1700 1 -- 172.21.15.98:0/2228436645 >> 172.21.15.98:0/2228436645 conn(0x7f252c0788f0 msgr2=0x7f252c0ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.758+0000 7f25324c1700 1 -- 172.21.15.98:0/2228436645 shutdown_connections 2024-04-25T17:34:55.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:55.758+0000 7f25324c1700 1 -- 172.21.15.98:0/2228436645 wait complete. 2024-04-25T17:34:56.426 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:56 smithi098 ceph-mon[29192]: pgmap v374: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T17:34:56.426 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:56 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2228436645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:34:56.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:34:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:56 smithi203 ceph-mon[32834]: pgmap v374: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T17:34:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:56 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2228436645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:34:56.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:56 smithi173 ceph-mon[32856]: pgmap v374: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T17:34:56.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:56 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2228436645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:34:56.692 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:34:58.190 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:58 smithi098 ceph-mon[29192]: pgmap v375: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:58.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.430+0000 7f910ca98700 1 -- 172.21.15.98:0/795287114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9108108a70 msgr2=0x7f9108108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:58.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.430+0000 7f910ca98700 1 --2- 172.21.15.98:0/795287114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9108108a70 0x7f9108108e50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f90fc009990 tx=0x7f90fc009ca0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.431+0000 7f910ca98700 1 -- 172.21.15.98:0/795287114 shutdown_connections 2024-04-25T17:34:58.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.431+0000 7f910ca98700 1 --2- 172.21.15.98:0/795287114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9108103480 0x7f910810f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.431+0000 7f910ca98700 1 --2- 172.21.15.98:0/795287114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9108108a70 0x7f9108108e50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.431+0000 7f910ca98700 1 --2- 172.21.15.98:0/795287114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9108102ac0 0x7f9108102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.431+0000 7f910ca98700 1 -- 172.21.15.98:0/795287114 >> 172.21.15.98:0/795287114 conn(0x7f9108079bb0 msgr2=0x7f9108079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:58.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.431+0000 7f910ca98700 1 -- 172.21.15.98:0/795287114 shutdown_connections 2024-04-25T17:34:58.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.431+0000 7f910ca98700 1 -- 172.21.15.98:0/795287114 wait complete. 2024-04-25T17:34:58.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.431+0000 7f910ca98700 1 Processor -- start 2024-04-25T17:34:58.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.431+0000 7f910ca98700 1 -- start start 2024-04-25T17:34:58.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910ca98700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9108102ac0 0x7f9108074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:58.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910ca98700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9108103480 0x7f9108075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:58.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910ca98700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9108108a70 0x7f91080757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910ca98700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9108112d00 con 0x7f9108102ac0 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910ca98700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9108112b80 con 0x7f9108103480 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910ca98700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9108112e80 con 0x7f9108108a70 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9108102ac0 0x7f9108074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9108102ac0 0x7f9108074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33364/0 (socket says 172.21.15.98:33364) 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910659c700 1 -- 172.21.15.98:0/2697073678 learned_addr learned my addr 172.21.15.98:0/2697073678 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f9106d9d700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9108108a70 0x7f91080757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f9105d9b700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9108103480 0x7f9108075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910659c700 1 -- 172.21.15.98:0/2697073678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9108108a70 msgr2=0x7f91080757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910659c700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9108108a70 0x7f91080757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910659c700 1 -- 172.21.15.98:0/2697073678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9108103480 msgr2=0x7f9108075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910659c700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9108103480 0x7f9108075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910659c700 1 -- 172.21.15.98:0/2697073678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91081a77c0 con 0x7f9108102ac0 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f9106d9d700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9108108a70 0x7f91080757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910659c700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9108102ac0 0x7f9108074d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f90fc000c00 tx=0x7f90fc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:58.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f90f77fe700 1 -- 172.21.15.98:0/2697073678 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90fc027070 con 0x7f9108102ac0 2024-04-25T17:34:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f90f77fe700 1 -- 172.21.15.98:0/2697073678 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90fc0041d0 con 0x7f9108102ac0 2024-04-25T17:34:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910ca98700 1 -- 172.21.15.98:0/2697073678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91081a7a50 con 0x7f9108102ac0 2024-04-25T17:34:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f90f77fe700 1 -- 172.21.15.98:0/2697073678 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90fc00f460 con 0x7f9108102ac0 2024-04-25T17:34:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.432+0000 7f910ca98700 1 -- 172.21.15.98:0/2697073678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91081a7f30 con 0x7f9108102ac0 2024-04-25T17:34:58.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.433+0000 7f910ca98700 1 -- 172.21.15.98:0/2697073678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9108069730 con 0x7f9108102ac0 2024-04-25T17:34:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.434+0000 7f90f77fe700 1 -- 172.21.15.98:0/2697073678 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f90fc02a070 con 0x7f9108102ac0 2024-04-25T17:34:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.435+0000 7f90f77fe700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f90f00a3c40 0x7f90f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:34:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.435+0000 7f9105d9b700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f90f00a3c40 0x7f90f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:34:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.435+0000 7f90f77fe700 1 -- 172.21.15.98:0/2697073678 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f90fc0d13d0 con 0x7f9108102ac0 2024-04-25T17:34:58.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.435+0000 7f9105d9b700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f90f00a3c40 0x7f90f00a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f90f8006fd0 tx=0x7f90f8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:34:58.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.439+0000 7f90f77fe700 1 -- 172.21.15.98:0/2697073678 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90fc05eff0 con 0x7f9108102ac0 2024-04-25T17:34:58.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:58 smithi203 ceph-mon[32834]: pgmap v375: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:58.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:58 smithi173 ceph-mon[32856]: pgmap v375: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:34:58.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.683+0000 7f910ca98700 1 -- 172.21.15.98:0/2697073678 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f910802e0a0 con 0x7f9108102ac0 2024-04-25T17:34:58.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.683+0000 7f90f77fe700 1 -- 172.21.15.98:0/2697073678 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11877 (secure 0 0 0) 0x7f90fc013070 con 0x7f9108102ac0 2024-04-25T17:34:58.684 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:34:58.684 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:32:59.314849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:34:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.685+0000 7f910ca98700 1 -- 172.21.15.98:0/2697073678 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f90f00a3c40 msgr2=0x7f90f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.685+0000 7f910ca98700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f90f00a3c40 0x7f90f00a6100 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f90f8006fd0 tx=0x7f90f8008040 comp rx=0 tx=0).stop 2024-04-25T17:34:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.685+0000 7f910ca98700 1 -- 172.21.15.98:0/2697073678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9108102ac0 msgr2=0x7f9108074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:34:58.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.685+0000 7f910ca98700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9108102ac0 0x7f9108074d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f90fc000c00 tx=0x7f90fc0047b0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.685+0000 7f910ca98700 1 -- 172.21.15.98:0/2697073678 shutdown_connections 2024-04-25T17:34:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.685+0000 7f910ca98700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f90f00a3c40 0x7f90f00a6100 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.685+0000 7f910ca98700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9108108a70 0x7f91080757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.685+0000 7f910ca98700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9108102ac0 0x7f9108074d40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.685+0000 7f910ca98700 1 --2- 172.21.15.98:0/2697073678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9108103480 0x7f9108075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:34:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.685+0000 7f910ca98700 1 -- 172.21.15.98:0/2697073678 >> 172.21.15.98:0/2697073678 conn(0x7f9108079bb0 msgr2=0x7f9108101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:34:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.686+0000 7f910ca98700 1 -- 172.21.15.98:0/2697073678 shutdown_connections 2024-04-25T17:34:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:34:58.686+0000 7f910ca98700 1 -- 172.21.15.98:0/2697073678 wait complete. 2024-04-25T17:34:58.687 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 25 2024-04-25T17:34:59.386 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:34:59 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2697073678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:34:59.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T17:32:37.173463+0000', 'modified': '2024-04-25T17:32:59.314849+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24461, 'mds_1': 24445, 'mds_2': 24440}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24422': {'gid': 24422, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/2662617129', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 2662617129}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 2662617129}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/2803298626', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 2803298626}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 2803298626}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6837/1057173981', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 1057173981}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 1057173981}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.173:6837/1096095017', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 1096095017}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 1096095017}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.203:6839/1340539385', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 1340539385}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 1340539385}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.173:6839/2342732992', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 2342732992}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 2342732992}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24461, 'qdb_cluster': [24445, 24440, 24461]} 2024-04-25T17:34:59.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T17:34:59.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell mds.h status 2024-04-25T17:34:59.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:34:59 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2697073678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:34:59.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:34:59 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2697073678' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:34:59.651 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:00 smithi098 ceph-mon[29192]: pgmap v376: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:00.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:00 smithi203 ceph-mon[32834]: pgmap v376: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:00 smithi173 ceph-mon[32856]: pgmap v376: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:01.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.300+0000 7fb5bcd54700 1 -- 172.21.15.98:0/3340959255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b8103450 msgr2=0x7fb5b810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:01.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.300+0000 7fb5bcd54700 1 --2- 172.21.15.98:0/3340959255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b8103450 0x7fb5b810f8b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000b0a0 tx=0x7fb5b000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.300+0000 7fb5bcd54700 1 -- 172.21.15.98:0/3340959255 shutdown_connections 2024-04-25T17:35:01.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.300+0000 7fb5bcd54700 1 --2- 172.21.15.98:0/3340959255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b8103450 0x7fb5b810f8b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.300+0000 7fb5bcd54700 1 --2- 172.21.15.98:0/3340959255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5b8108a50 0x7fb5b8108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.300+0000 7fb5bcd54700 1 --2- 172.21.15.98:0/3340959255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b8102a90 0x7fb5b8102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.300+0000 7fb5bcd54700 1 -- 172.21.15.98:0/3340959255 >> 172.21.15.98:0/3340959255 conn(0x7fb5b8079ba0 msgr2=0x7fb5b8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:01.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.300+0000 7fb5bcd54700 1 -- 172.21.15.98:0/3340959255 shutdown_connections 2024-04-25T17:35:01.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.300+0000 7fb5bcd54700 1 -- 172.21.15.98:0/3340959255 wait complete. 2024-04-25T17:35:01.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.301+0000 7fb5bcd54700 1 Processor -- start 2024-04-25T17:35:01.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.301+0000 7fb5bcd54700 1 -- start start 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.301+0000 7fb5bcd54700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b8102a90 0x7fb5b8199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.301+0000 7fb5bcd54700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5b8103450 0x7fb5b8199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.301+0000 7fb5bcd54700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b8108a50 0x7fb5b819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.301+0000 7fb5bcd54700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b8112c60 con 0x7fb5b8102a90 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.301+0000 7fb5bcd54700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b8112ae0 con 0x7fb5b8103450 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.301+0000 7fb5bcd54700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b8112de0 con 0x7fb5b8108a50 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b8102a90 0x7fb5b8199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b8102a90 0x7fb5b8199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33376/0 (socket says 172.21.15.98:33376) 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b659c700 1 -- 172.21.15.98:0/1677445105 learned_addr learned my addr 172.21.15.98:0/1677445105 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b5d9b700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5b8103450 0x7fb5b8199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b6d9d700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b8108a50 0x7fb5b819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b659c700 1 -- 172.21.15.98:0/1677445105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b8108a50 msgr2=0x7fb5b819d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:01.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b659c700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b8108a50 0x7fb5b819d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b659c700 1 -- 172.21.15.98:0/1677445105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5b8103450 msgr2=0x7fb5b8199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b659c700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5b8103450 0x7fb5b8199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b659c700 1 -- 172.21.15.98:0/1677445105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b819e110 con 0x7fb5b8102a90 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5b659c700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b8102a90 0x7fb5b8199150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800b530 tx=0x7fb5a800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1677445105 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8015020 con 0x7fb5b8102a90 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1677445105 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a8011ce0 con 0x7fb5b8102a90 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5bcd54700 1 -- 172.21.15.98:0/1677445105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5b819e400 con 0x7fb5b8102a90 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1677445105 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8010d20 con 0x7fb5b8102a90 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.302+0000 7fb5bcd54700 1 -- 172.21.15.98:0/1677445105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5b81a5c80 con 0x7fb5b8102a90 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.304+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1677445105 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb5a8018020 con 0x7fb5b8102a90 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.305+0000 7fb5a77fe700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb5a00a3cc0 0x7fb5a00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.305+0000 7fb5b5d9b700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb5a00a3cc0 0x7fb5a00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.305+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1677445105 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fb5a80cfbf0 con 0x7fb5b8102a90 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.305+0000 7fb5b5d9b700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb5a00a3cc0 0x7fb5a00a6180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac005fd0 tx=0x7fb5ac00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:01.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5c75ddb80 1 Processor -- start 2024-04-25T17:35:01.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5c75ddb80 1 -- start start 2024-04-25T17:35:01.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a565490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5659d0 0x55b81a570340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a570880 0x55b81a572c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b81a4b8c40 con 0x55b81a5650b0 2024-04-25T17:35:01.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b81a45f8e0 con 0x55b81a570880 2024-04-25T17:35:01.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b81a4b4c00 con 0x55b81a5659d0 2024-04-25T17:35:01.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a565490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a565490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33388/0 (socket says 172.21.15.98:33388) 2024-04-25T17:35:01.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5b659c700 1 -- 172.21.15.98:0/425736523 learned_addr learned my addr 172.21.15.98:0/425736523 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:01.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5b659c700 1 -- 172.21.15.98:0/425736523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5659d0 msgr2=0x55b81a570340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:01.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5b659c700 1 --2- 172.21.15.98:0/425736523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5659d0 0x55b81a570340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5b659c700 1 -- 172.21.15.98:0/425736523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a570880 msgr2=0x55b81a572c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:01.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5b659c700 1 --2- 172.21.15.98:0/425736523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a570880 0x55b81a572c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.306+0000 7fb5b659c700 1 -- 172.21.15.98:0/425736523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b81a4236f0 con 0x55b81a5650b0 2024-04-25T17:35:01.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5b659c700 1 --2- 172.21.15.98:0/425736523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a565490 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80a4620 tx=0x7fb5a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a66688d4921469 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:01.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5a4ff9700 1 -- 172.21.15.98:0/425736523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a809bee0 con 0x55b81a5650b0 2024-04-25T17:35:01.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5a4ff9700 1 -- 172.21.15.98:0/425736523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a80a0290 con 0x55b81a5650b0 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5a4ff9700 1 -- 172.21.15.98:0/425736523 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8018510 con 0x55b81a5650b0 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/425736523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 msgr2=0x55b81a565490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/425736523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a565490 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb5a80a4620 tx=0x7fb5a8004750 comp rx=0 tx=0).stop 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/425736523 shutdown_connections 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/425736523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a570880 0x55b81a572c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/425736523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a565490 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/425736523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5659d0 0x55b81a570340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.307+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/425736523 >> 172.21.15.98:0/425736523 conn(0x55b81a492840 msgr2=0x55b81a561580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/425736523 shutdown_connections 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/425736523 wait complete. 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 1 -- start start 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 1 Processor -- start 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5650b0 0x55b81a570070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5659d0 0x55b81a56d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a570880 0x55b81a56d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b81a573370 con 0x55b81a5659d0 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b81a472560 con 0x55b81a570880 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b81a5734f0 con 0x55b81a5650b0 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5c75ddb80 10 client.0 _ll_register_callbacks cb 0x55b81a4518d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5b5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5659d0 0x55b81a56d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5b5d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5659d0 0x55b81a56d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33400/0 (socket says 172.21.15.98:33400) 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.308+0000 7fb5b5d9b700 1 -- 172.21.15.98:0/162253033 learned_addr learned my addr 172.21.15.98:0/162253033 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5b6d9d700 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a570880 0x55b81a56d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5b659c700 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5650b0 0x55b81a570070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5b5d9b700 1 -- 172.21.15.98:0/162253033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5650b0 msgr2=0x55b81a570070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5b5d9b700 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5650b0 0x55b81a570070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5b5d9b700 1 -- 172.21.15.98:0/162253033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a570880 msgr2=0x55b81a56d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5b5d9b700 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a570880 0x55b81a56d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5b5d9b700 1 -- 172.21.15.98:0/162253033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b81a406880 con 0x55b81a5659d0 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5b5d9b700 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5659d0 0x55b81a56d370 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00a7e0 tx=0x7fb5ac010a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d168dffeb2f3625b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb59dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb59dffb700 1 -- 172.21.15.98:0/162253033 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac021070 con 0x55b81a5659d0 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb59dffb700 1 -- 172.21.15.98:0/162253033 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fb5ac01d860 con 0x55b81a5659d0 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/162253033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5a80a0290 con 0x55b81a5659d0 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb59dffb700 1 -- 172.21.15.98:0/162253033 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb5ac02d050 con 0x55b81a5659d0 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5c75ddb80 10 client.14949 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/162253033 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b81a57dd00 con 0x55b81a5659d0 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb59dffb700 1 -- 172.21.15.98:0/162253033 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a80a0290 con 0x55b81a5659d0 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb59dffb700 1 -- 172.21.15.98:0/162253033 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fb5ac00fb40 con 0x55b81a5659d0 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.309+0000 7fb5c75ddb80 10 client.14949 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:01.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.310+0000 7fb5c75ddb80 10 client.14949 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.310+0000 7fb5c75ddb80 20 client.14949 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.310+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55b81a5802c0 0x55b81a581730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.310+0000 7fb5c75ddb80 4 client.14949 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.310+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/162253033 --> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b81a4c6430 con 0x55b81a5802c0 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.310+0000 7fb5b659c700 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55b81a5802c0 0x55b81a581730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.310+0000 7fb59dffb700 10 client.14949 ms_handle_connect on v2:172.21.15.173:6836/1096095017 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.310+0000 7fb5b659c700 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55b81a5802c0 0x55b81a581730 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2014cbe36a58bcd2 server_cookie=e64853e193d86208 in_seq=0 out_seq=0 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.311+0000 7fb59dffb700 1 -- 172.21.15.98:0/162253033 <== mds.2 v2:172.21.15.173:6836/1096095017 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b81a4c6430 con 0x55b81a5802c0 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.311+0000 7fb59dffb700 10 client.14949 handle_command_reply: tid=0 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.311+0000 7fb5c75ddb80 1 client.14949 shutdown 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.311+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/162253033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5659d0 msgr2=0x55b81a56d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.311+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5659d0 0x55b81a56d370 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00a7e0 tx=0x7fb5ac010a40 comp rx=0 tx=0).stop 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.311+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/162253033 shutdown_connections 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.311+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55b81a5802c0 0x55b81a581730 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.311+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5650b0 0x55b81a570070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.311+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a570880 0x55b81a56d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.311+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/162253033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5659d0 0x55b81a56d370 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.312+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/162253033 >> 172.21.15.98:0/162253033 conn(0x55b81a492840 msgr2=0x55b81a5611b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:01.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.312+0000 7fb59dffb700 0 client.14949 ms_handle_reset on v2:172.21.15.173:6836/1096095017 2024-04-25T17:35:01.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.312+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/162253033 shutdown_connections 2024-04-25T17:35:01.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.312+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/162253033 wait complete. 2024-04-25T17:35:01.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.312+0000 7fb5c75ddb80 20 client.14949 trim_cache size 0 max 16384 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5c75ddb80 1 Processor -- start 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5c75ddb80 1 -- start start 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a561060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a5659d0 0x55b81a5615a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a570880 0x55b81a56bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b81a57fa70 con 0x55b81a5650b0 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b81a580140 con 0x55b81a5659d0 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b81a580810 con 0x55b81a570880 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a561060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a561060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33412/0 (socket says 172.21.15.98:33412) 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5b659c700 1 -- 172.21.15.98:0/1023573309 learned_addr learned my addr 172.21.15.98:0/1023573309 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5b5d9b700 1 --2- 172.21.15.98:0/1023573309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a5659d0 0x55b81a5615a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5b6d9d700 1 --2- 172.21.15.98:0/1023573309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a570880 0x55b81a56bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5b659c700 1 -- 172.21.15.98:0/1023573309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a570880 msgr2=0x55b81a56bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5b659c700 1 --2- 172.21.15.98:0/1023573309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a570880 0x55b81a56bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5b659c700 1 -- 172.21.15.98:0/1023573309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a5659d0 msgr2=0x55b81a5615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5b659c700 1 --2- 172.21.15.98:0/1023573309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a5659d0 0x55b81a5615a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.332+0000 7fb5b659c700 1 -- 172.21.15.98:0/1023573309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b81a5602d0 con 0x55b81a5650b0 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb5b659c700 1 --2- 172.21.15.98:0/1023573309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a561060 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800bcf0 tx=0x7fb5a8004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443db686ce3ee2f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb59ffff700 1 -- 172.21.15.98:0/1023573309 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a807f070 con 0x55b81a5650b0 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb59ffff700 1 -- 172.21.15.98:0/1023573309 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a80a0290 con 0x55b81a5650b0 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb59ffff700 1 -- 172.21.15.98:0/1023573309 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a809c140 con 0x55b81a5650b0 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/1023573309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 msgr2=0x55b81a561060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/1023573309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a561060 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800bcf0 tx=0x7fb5a8004860 comp rx=0 tx=0).stop 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/1023573309 shutdown_connections 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/1023573309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a570880 0x55b81a56bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/1023573309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a5650b0 0x55b81a561060 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/1023573309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a5659d0 0x55b81a5615a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/1023573309 >> 172.21.15.98:0/1023573309 conn(0x55b81a492840 msgr2=0x55b81a560700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/1023573309 shutdown_connections 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.333+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/1023573309 wait complete. 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5c75ddb80 1 -- start start 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5c75ddb80 1 Processor -- start 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5650b0 0x55b81a585e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a5659d0 0x55b81a586340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5c75ddb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a570880 0x55b81a5df420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b81a5815b0 con 0x55b81a570880 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b81a580ee0 con 0x55b81a5659d0 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5c75ddb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b81a581c80 con 0x55b81a5650b0 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5c75ddb80 10 client.0 _ll_register_callbacks cb 0x55b81a4518d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:01.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5b6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a570880 0x55b81a5df420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5b6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a570880 0x55b81a5df420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33426/0 (socket says 172.21.15.98:33426) 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5b6d9d700 1 -- 172.21.15.98:0/245483412 learned_addr learned my addr 172.21.15.98:0/245483412 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5b659c700 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5650b0 0x55b81a585e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5b5d9b700 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a5659d0 0x55b81a586340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5b6d9d700 1 -- 172.21.15.98:0/245483412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5650b0 msgr2=0x55b81a585e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5b6d9d700 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5650b0 0x55b81a585e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5b6d9d700 1 -- 172.21.15.98:0/245483412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a5659d0 msgr2=0x55b81a586340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5b6d9d700 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a5659d0 0x55b81a586340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.334+0000 7fb5b6d9d700 1 -- 172.21.15.98:0/245483412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b81a5df9e0 con 0x55b81a570880 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb5b6d9d700 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a570880 0x55b81a5df420 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000b0a0 tx=0x7fb5b0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4ab6a1950e2e46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb59dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb59dffb700 1 -- 172.21.15.98:0/245483412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b00076e0 con 0x55b81a570880 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb59dffb700 1 -- 172.21.15.98:0/245483412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fb5b0007880 con 0x55b81a570880 2024-04-25T17:35:01.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/245483412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5a80a0290 con 0x55b81a570880 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb59dffb700 1 -- 172.21.15.98:0/245483412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb5b0007a80 con 0x55b81a570880 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb5c75ddb80 10 client.14955 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/245483412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b81a5e6d90 con 0x55b81a570880 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb59dffb700 1 -- 172.21.15.98:0/245483412 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5a80a0290 con 0x55b81a570880 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb59dffb700 1 -- 172.21.15.98:0/245483412 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fb5b0011070 con 0x55b81a570880 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb5c75ddb80 10 client.14955 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb5c75ddb80 10 client.14955 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb5c75ddb80 20 client.14955 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55b81a5eb9d0 0x55b81a5ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb5c75ddb80 4 client.14955 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.335+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/245483412 --> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] -- command(tid 0: {"prefix": "status"}) -- 0x55b81a4c04d0 con 0x55b81a5eb9d0 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.336+0000 7fb5b659c700 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55b81a5eb9d0 0x55b81a5ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.336+0000 7fb5b659c700 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55b81a5eb9d0 0x55b81a5ea6e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a1caff1525382e38 server_cookie=93d00418ac3c1bbe in_seq=0 out_seq=0 2024-04-25T17:35:01.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.336+0000 7fb59dffb700 10 client.14955 ms_handle_connect on v2:172.21.15.173:6836/1096095017 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.336+0000 7fb59dffb700 1 -- 172.21.15.98:0/245483412 <== mds.2 v2:172.21.15.173:6836/1096095017 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b81a4c04d0 con 0x55b81a5eb9d0 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.336+0000 7fb59dffb700 10 client.14955 handle_command_reply: tid=0 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.336+0000 7fb5c75ddb80 1 client.14955 shutdown 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.336+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/245483412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a570880 msgr2=0x55b81a5df420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.336+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a570880 0x55b81a5df420 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000b0a0 tx=0x7fb5b0004c80 comp rx=0 tx=0).stop 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/245483412 shutdown_connections 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55b81a5eb9d0 0x55b81a5ea6e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b81a5650b0 0x55b81a585e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b81a570880 0x55b81a5df420 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5c75ddb80 1 --2- 172.21.15.98:0/245483412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b81a5659d0 0x55b81a586340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb59dffb700 0 client.14955 ms_handle_reset on v2:172.21.15.173:6836/1096095017 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/245483412 >> 172.21.15.98:0/245483412 conn(0x55b81a492840 msgr2=0x55b81a56f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/245483412 shutdown_connections 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5c75ddb80 1 -- 172.21.15.98:0/245483412 wait complete. 2024-04-25T17:35:01.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5c75ddb80 20 client.14955 trim_cache size 0 max 16384 2024-04-25T17:35:01.346 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:35:01.346 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:35:01.346 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-04-25T17:35:01.346 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24440, 2024-04-25T17:35:01.346 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T17:35:01.346 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T17:35:01.346 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T17:35:01.346 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 128.21404399100001, 2024-04-25T17:35:01.346 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 25, 2024-04-25T17:35:01.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-04-25T17:35:01.347 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T17:35:01.347 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 176.44943818199999 2024-04-25T17:35:01.347 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:35:01.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 -- 172.21.15.98:0/1677445105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb5a00a3cc0 msgr2=0x7fb5a00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:01.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb5a00a3cc0 0x7fb5a00a6180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac005fd0 tx=0x7fb5ac00a560 comp rx=0 tx=0).stop 2024-04-25T17:35:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 -- 172.21.15.98:0/1677445105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b8102a90 msgr2=0x7fb5b8199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b8102a90 0x7fb5b8199150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800b530 tx=0x7fb5a800b8f0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 -- 172.21.15.98:0/1677445105 shutdown_connections 2024-04-25T17:35:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb5a00a3cc0 0x7fb5a00a6180 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b8108a50 0x7fb5b819d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b8102a90 0x7fb5b8199150 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 --2- 172.21.15.98:0/1677445105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5b8103450 0x7fb5b8199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 -- 172.21.15.98:0/1677445105 >> 172.21.15.98:0/1677445105 conn(0x7fb5b8079ba0 msgr2=0x7fb5b81017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 -- 172.21.15.98:0/1677445105 shutdown_connections 2024-04-25T17:35:01.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:01.337+0000 7fb5bcd54700 1 -- 172.21.15.98:0/1677445105 wait complete. 2024-04-25T17:35:02.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-25T17:35:02.356 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:02.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:02 smithi098 ceph-mon[29192]: pgmap v377: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:02.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:02 smithi203 ceph-mon[32834]: pgmap v377: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:02.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:02 smithi173 ceph-mon[32856]: pgmap v377: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:04.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 -- 172.21.15.98:0/1995210509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e0c1024f0 msgr2=0x7f0e0c102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 --2- 172.21.15.98:0/1995210509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e0c1024f0 0x7f0e0c102970 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0e000099c0 tx=0x7f0e00009cd0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 -- 172.21.15.98:0/1995210509 shutdown_connections 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 --2- 172.21.15.98:0/1995210509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e0c102eb0 0x7f0e0c10e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 --2- 172.21.15.98:0/1995210509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c107c10 0x7f0e0c107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 --2- 172.21.15.98:0/1995210509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e0c1024f0 0x7f0e0c102970 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 -- 172.21.15.98:0/1995210509 >> 172.21.15.98:0/1995210509 conn(0x7f0e0c076c60 msgr2=0x7f0e0c077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 -- 172.21.15.98:0/1995210509 shutdown_connections 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 -- 172.21.15.98:0/1995210509 wait complete. 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 Processor -- start 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.029+0000 7f0e116c6700 1 -- start start 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e116c6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c1024f0 0x7f0e0c198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e116c6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e0c102eb0 0x7f0e0c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e116c6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e0c107c10 0x7f0e0c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e116c6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c111c90 con 0x7f0e0c107c10 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e116c6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c111b10 con 0x7f0e0c1024f0 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e116c6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c111e10 con 0x7f0e0c102eb0 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e0c107c10 0x7f0e0c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e0c107c10 0x7f0e0c19c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38716/0 (socket says 172.21.15.98:38716) 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0b7fe700 1 -- 172.21.15.98:0/2647944580 learned_addr learned my addr 172.21.15.98:0/2647944580 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c1024f0 0x7f0e0c198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e0c102eb0 0x7f0e0c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0b7fe700 1 -- 172.21.15.98:0/2647944580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e0c102eb0 msgr2=0x7f0e0c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e0c102eb0 0x7f0e0c198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0b7fe700 1 -- 172.21.15.98:0/2647944580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c1024f0 msgr2=0x7f0e0c198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c1024f0 0x7f0e0c198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0b7fe700 1 -- 172.21.15.98:0/2647944580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0c19d0e0 con 0x7f0e0c107c10 2024-04-25T17:35:04.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e0c102eb0 0x7f0e0c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e0c107c10 0x7f0e0c19c9a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0e040094f0 tx=0x7f0e0400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0df3fff700 1 -- 172.21.15.98:0/2647944580 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0400ddd0 con 0x7f0e0c107c10 2024-04-25T17:35:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0e116c6700 1 -- 172.21.15.98:0/2647944580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0c19d3d0 con 0x7f0e0c107c10 2024-04-25T17:35:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.030+0000 7f0df3fff700 1 -- 172.21.15.98:0/2647944580 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e04009e20 con 0x7f0e0c107c10 2024-04-25T17:35:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.031+0000 7f0df3fff700 1 -- 172.21.15.98:0/2647944580 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e04015400 con 0x7f0e0c107c10 2024-04-25T17:35:04.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.031+0000 7f0e116c6700 1 -- 172.21.15.98:0/2647944580 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0c1a4c50 con 0x7f0e0c107c10 2024-04-25T17:35:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.032+0000 7f0df3fff700 1 -- 172.21.15.98:0/2647944580 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0e04005020 con 0x7f0e0c107c10 2024-04-25T17:35:04.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.033+0000 7f0df3fff700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0df40a3c40 0x7f0df40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.033+0000 7f0e0affd700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0df40a3c40 0x7f0df40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.033+0000 7f0df3fff700 1 -- 172.21.15.98:0/2647944580 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f0e040cfe20 con 0x7f0e0c107c10 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.034+0000 7f0e0affd700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0df40a3c40 0x7f0df40a6100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc005950 tx=0x7f0dfc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.034+0000 7f0e1bf4fb80 1 Processor -- start 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.034+0000 7f0e1bf4fb80 1 -- start start 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2c7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 0x55ea5f2c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f212ce0 con 0x55ea5f2c3300 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f1b9910 con 0x55ea5f2c29e0 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f20eca0 con 0x55ea5f2c7790 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38726/0 (socket says 172.21.15.98:38726) 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/2939010280 learned_addr learned my addr 172.21.15.98:0/2939010280 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/2939010280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 0x55ea5f2c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0affd700 1 --2- 172.21.15.98:0/2939010280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2c2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/2939010280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 msgr2=0x55ea5f2c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/2939010280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 0x55ea5f2c7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/2939010280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 msgr2=0x55ea5f2c2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/2939010280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2c2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/2939010280 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea5f160880 con 0x55ea5f2c3300 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/2939010280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 0x55ea5f2c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/2939010280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2c7250 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0e000099c0 tx=0x7f0e00005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7f353b3f034605 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0df17fa700 1 -- 172.21.15.98:0/2939010280 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00027070 con 0x55ea5f2c3300 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0df17fa700 1 -- 172.21.15.98:0/2939010280 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e000050d0 con 0x55ea5f2c3300 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.035+0000 7f0df17fa700 1 -- 172.21.15.98:0/2939010280 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0000bac0 con 0x55ea5f2c3300 2024-04-25T17:35:04.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.036+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2939010280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 msgr2=0x55ea5f2c7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.036+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/2939010280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2c7250 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0e000099c0 tx=0x7f0e00005c00 comp rx=0 tx=0).stop 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.036+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2939010280 shutdown_connections 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.036+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/2939010280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 0x55ea5f2c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.036+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/2939010280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.036+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/2939010280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2c7250 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.036+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2939010280 >> 172.21.15.98:0/2939010280 conn(0x55ea5f1ed640 msgr2=0x55ea5f2bb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.036+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2939010280 shutdown_connections 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.036+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2939010280 wait complete. 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.036+0000 7f0e1bf4fb80 1 -- start start 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e1bf4fb80 1 Processor -- start 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2cec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 0x55ea5f2cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f2cf7c0 con 0x55ea5f2c3300 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f1b1b80 con 0x55ea5f2c29e0 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f2cf940 con 0x55ea5f2c7790 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e1bf4fb80 10 client.0 _ll_register_callbacks cb 0x55ea5f1ab900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2cbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38732/0 (socket says 172.21.15.98:38732) 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/1141373673 learned_addr learned my addr 172.21.15.98:0/1141373673 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 0x55ea5f2cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/1141373673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 msgr2=0x55ea5f2cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 0x55ea5f2cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/1141373673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 msgr2=0x55ea5f2cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2cec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/1141373673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea5f1ca220 con 0x55ea5f2c3300 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0affd700 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:35:04.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.037+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2cbf60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000b570 tx=0x7f0e0000b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3814f23dee047cbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0deeff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0deeff5700 1 -- 172.21.15.98:0/1141373673 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00027030 con 0x55ea5f2c3300 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0deeff5700 1 -- 172.21.15.98:0/1141373673 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f0e00003d50 con 0x55ea5f2c3300 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1141373673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e000050d0 con 0x55ea5f2c3300 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0deeff5700 1 -- 172.21.15.98:0/1141373673 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0e0000b730 con 0x55ea5f2c3300 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0e1bf4fb80 10 client.14967 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1141373673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ea5f2c05f0 con 0x55ea5f2c3300 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0deeff5700 1 -- 172.21.15.98:0/1141373673 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e000050d0 con 0x55ea5f2c3300 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0deeff5700 1 -- 172.21.15.98:0/1141373673 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f0e0000fa70 con 0x55ea5f2c3300 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0e1bf4fb80 10 client.14967 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0e1bf4fb80 10 client.14967 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0e1bf4fb80 20 client.14967 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55ea5f2c1a50 0x55ea5f2db340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0e1bf4fb80 4 client.14967 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:35:04.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.038+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1141373673 --> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea5f2204d0 con 0x55ea5f2c1a50 2024-04-25T17:35:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.039+0000 7f0e0affd700 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55ea5f2c1a50 0x55ea5f2db340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.039+0000 7f0deeff5700 10 client.14967 ms_handle_connect on v2:172.21.15.203:6838/1340539385 2024-04-25T17:35:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.039+0000 7f0e0affd700 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55ea5f2c1a50 0x55ea5f2db340 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8b68b12ddef01bd7 server_cookie=2fe57de447c305fb in_seq=0 out_seq=0 2024-04-25T17:35:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0deeff5700 1 -- 172.21.15.98:0/1141373673 <== mds.1 v2:172.21.15.203:6838/1340539385 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea5f2204d0 con 0x55ea5f2c1a50 2024-04-25T17:35:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0deeff5700 10 client.14967 handle_command_reply: tid=0 2024-04-25T17:35:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 client.14967 shutdown 2024-04-25T17:35:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1141373673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 msgr2=0x55ea5f2cbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2cbf60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000b570 tx=0x7f0e0000b5a0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1141373673 shutdown_connections 2024-04-25T17:35:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55ea5f2c1a50 0x55ea5f2db340 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c7790 0x55ea5f2cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/1141373673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c3300 0x55ea5f2cbf60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1141373673 >> 172.21.15.98:0/1141373673 conn(0x55ea5f1ed640 msgr2=0x55ea5f2baa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0deeff5700 0 client.14967 ms_handle_reset on v2:172.21.15.203:6838/1340539385 2024-04-25T17:35:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1141373673 shutdown_connections 2024-04-25T17:35:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1141373673 wait complete. 2024-04-25T17:35:04.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.040+0000 7f0e1bf4fb80 20 client.14967 trim_cache size 0 max 16384 2024-04-25T17:35:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.061+0000 7f0e1bf4fb80 1 Processor -- start 2024-04-25T17:35:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.061+0000 7f0e1bf4fb80 1 -- start start 2024-04-25T17:35:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2bfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c3300 0x55ea5f2e47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f2c1f10 con 0x55ea5f2c7790 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f2c25e0 con 0x55ea5f2c29e0 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f2da420 con 0x55ea5f2c3300 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38748/0 (socket says 172.21.15.98:38748) 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0b7fe700 1 -- 172.21.15.98:0/1380523269 learned_addr learned my addr 172.21.15.98:0/1380523269 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0affd700 1 --2- 172.21.15.98:0/1380523269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2bfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/1380523269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c3300 0x55ea5f2e47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0b7fe700 1 -- 172.21.15.98:0/1380523269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c3300 msgr2=0x55ea5f2e47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/1380523269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c3300 0x55ea5f2e47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0b7fe700 1 -- 172.21.15.98:0/1380523269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 msgr2=0x55ea5f2bfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:04.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/1380523269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2bfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0b7fe700 1 -- 172.21.15.98:0/1380523269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea5f2cf860 con 0x55ea5f2c7790 2024-04-25T17:35:04.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0affd700 1 --2- 172.21.15.98:0/1380523269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2bfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:04.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/1380523269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e4cf0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400d7b0 tx=0x7f0e040cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beffbb5c92cf90fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:04.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.062+0000 7f0df0ff9700 1 -- 172.21.15.98:0/1380523269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0409a1b0 con 0x55ea5f2c7790 2024-04-25T17:35:04.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.063+0000 7f0df0ff9700 1 -- 172.21.15.98:0/1380523269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e04004dd0 con 0x55ea5f2c7790 2024-04-25T17:35:04.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.063+0000 7f0df0ff9700 1 -- 172.21.15.98:0/1380523269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0409c370 con 0x55ea5f2c7790 2024-04-25T17:35:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.063+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1380523269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 msgr2=0x55ea5f2e4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.063+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/1380523269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e4cf0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400d7b0 tx=0x7f0e040cf040 comp rx=0 tx=0).stop 2024-04-25T17:35:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.063+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1380523269 shutdown_connections 2024-04-25T17:35:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.063+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/1380523269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e4cf0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.063+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/1380523269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2bfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.063+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/1380523269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c3300 0x55ea5f2e47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1380523269 >> 172.21.15.98:0/1380523269 conn(0x55ea5f1ed640 msgr2=0x55ea5f2c69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1380523269 shutdown_connections 2024-04-25T17:35:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/1380523269 wait complete. 2024-04-25T17:35:04.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 -- start start 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 Processor -- start 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2e22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c3300 0x55ea5f2e2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f2db1c0 con 0x55ea5f2c7790 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f2daaf0 con 0x55ea5f2c3300 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ea5f2db890 con 0x55ea5f2c29e0 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.064+0000 7f0e1bf4fb80 10 client.0 _ll_register_callbacks cb 0x55ea5f1ab900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c3300 0x55ea5f2e2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c3300 0x55ea5f2e2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:37698/0 (socket says 172.21.15.98:37698) 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/2584860710 learned_addr learned my addr 172.21.15.98:0/2584860710 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0affd700 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2e22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/2584860710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c29e0 msgr2=0x55ea5f2e22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2e22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/2584860710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 msgr2=0x55ea5f2e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0a7fc700 1 -- 172.21.15.98:0/2584860710 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea5f340110 con 0x55ea5f2c3300 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0b7fe700 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e0a7fc700 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c3300 0x55ea5f2e2800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000ff60 tx=0x7f0e00009d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4687d063e318fb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0deeff5700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0deeff5700 1 -- 172.21.15.98:0/2584860710 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00044060 con 0x55ea5f2c3300 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2584860710 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e04004dd0 con 0x55ea5f2c3300 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.065+0000 7f0deeff5700 1 -- 172.21.15.98:0/2584860710 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f0e0002d960 con 0x55ea5f2c3300 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0deeff5700 1 -- 172.21.15.98:0/2584860710 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0e0003c070 con 0x55ea5f2c3300 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0e1bf4fb80 10 client.24674 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2584860710 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ea5f3413f0 con 0x55ea5f2c3300 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0deeff5700 1 -- 172.21.15.98:0/2584860710 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e04004dd0 con 0x55ea5f2c3300 2024-04-25T17:35:04.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0deeff5700 1 -- 172.21.15.98:0/2584860710 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f0e0000f460 con 0x55ea5f2c3300 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0e1bf4fb80 10 client.24674 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0e1bf4fb80 10 client.24674 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0e1bf4fb80 20 client.24674 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55ea5f345f70 0x55ea5f344c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0e1bf4fb80 4 client.24674 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2584860710 --> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] -- command(tid 0: {"prefix": "status"}) -- 0x55ea5f21a570 con 0x55ea5f345f70 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.066+0000 7f0e0affd700 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55ea5f345f70 0x55ea5f344c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0deeff5700 10 client.24674 ms_handle_connect on v2:172.21.15.203:6838/1340539385 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e0affd700 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55ea5f345f70 0x55ea5f344c80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3c93a1b50e6d6067 server_cookie=fd5d42b5f61c40a2 in_seq=0 out_seq=0 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0deeff5700 1 -- 172.21.15.98:0/2584860710 <== mds.1 v2:172.21.15.203:6838/1340539385 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ea5f21a570 con 0x55ea5f345f70 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0deeff5700 10 client.24674 handle_command_reply: tid=0 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 client.24674 shutdown 2024-04-25T17:35:04.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2584860710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c3300 msgr2=0x55ea5f2e2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c3300 0x55ea5f2e2800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000ff60 tx=0x7f0e00009d60 comp rx=0 tx=0).stop 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2584860710 shutdown_connections 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55ea5f345f70 0x55ea5f344c80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ea5f2c29e0 0x55ea5f2e22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ea5f2c7790 0x55ea5f2e2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 --2- 172.21.15.98:0/2584860710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ea5f2c3300 0x55ea5f2e2800 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2584860710 >> 172.21.15.98:0/2584860710 conn(0x55ea5f1ed640 msgr2=0x55ea5f2cd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0deeff5700 0 client.24674 ms_handle_reset on v2:172.21.15.203:6838/1340539385 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2584860710 shutdown_connections 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.067+0000 7f0e1bf4fb80 1 -- 172.21.15.98:0/2584860710 wait complete. 2024-04-25T17:35:04.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e1bf4fb80 20 client.24674 trim_cache size 0 max 16384 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24445, 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 131.95798212599999, 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 25, 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T17:35:04.074 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 176.35082557499999 2024-04-25T17:35:04.075 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:35:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 -- 172.21.15.98:0/2647944580 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0df40a3c40 msgr2=0x7f0df40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0df40a3c40 0x7f0df40a6100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc005950 tx=0x7f0dfc00b410 comp rx=0 tx=0).stop 2024-04-25T17:35:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 -- 172.21.15.98:0/2647944580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e0c107c10 msgr2=0x7f0e0c19c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e0c107c10 0x7f0e0c19c9a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0e040094f0 tx=0x7f0e0400d3b0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 -- 172.21.15.98:0/2647944580 shutdown_connections 2024-04-25T17:35:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0df40a3c40 0x7f0df40a6100 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e0c107c10 0x7f0e0c19c9a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e0c1024f0 0x7f0e0c198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 --2- 172.21.15.98:0/2647944580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e0c102eb0 0x7f0e0c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 -- 172.21.15.98:0/2647944580 >> 172.21.15.98:0/2647944580 conn(0x7f0e0c076c60 msgr2=0x7f0e0c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 -- 172.21.15.98:0/2647944580 shutdown_connections 2024-04-25T17:35:04.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:04.068+0000 7f0e116c6700 1 -- 172.21.15.98:0/2647944580 wait complete. 2024-04-25T17:35:04.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:04 smithi098 ceph-mon[29192]: pgmap v378: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:35:04.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:04 smithi203 ceph-mon[32834]: pgmap v378: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:35:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:04 smithi173 ceph-mon[32856]: pgmap v378: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:35:04.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-25T17:35:05.051 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:06.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:06 smithi098 ceph-mon[29192]: pgmap v379: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:06.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:06 smithi203 ceph-mon[32834]: pgmap v379: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:06 smithi173 ceph-mon[32856]: pgmap v379: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:06.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.814+0000 7f0b95ddf700 1 -- 172.21.15.98:0/2466158265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b9006bb30 msgr2=0x7f0b9010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:06.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.814+0000 7f0b95ddf700 1 --2- 172.21.15.98:0/2466158265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b9006bb30 0x7f0b9010f900 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800b0a0 tx=0x7f0b8800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.814+0000 7f0b95ddf700 1 -- 172.21.15.98:0/2466158265 shutdown_connections 2024-04-25T17:35:06.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.814+0000 7f0b95ddf700 1 --2- 172.21.15.98:0/2466158265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b90108a30 0x7f0b90108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.814+0000 7f0b95ddf700 1 --2- 172.21.15.98:0/2466158265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b9006bb30 0x7f0b9010f900 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.814+0000 7f0b95ddf700 1 --2- 172.21.15.98:0/2466158265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b9006b190 0x7f0b9006b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.814+0000 7f0b95ddf700 1 -- 172.21.15.98:0/2466158265 >> 172.21.15.98:0/2466158265 conn(0x7f0b90078910 msgr2=0x7f0b90078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.814+0000 7f0b95ddf700 1 -- 172.21.15.98:0/2466158265 shutdown_connections 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.814+0000 7f0b95ddf700 1 -- 172.21.15.98:0/2466158265 wait complete. 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.815+0000 7f0b95ddf700 1 Processor -- start 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.815+0000 7f0b95ddf700 1 -- start start 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.815+0000 7f0b95ddf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b9006b190 0x7f0b90199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.815+0000 7f0b95ddf700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b9006bb30 0x7f0b901995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.815+0000 7f0b95ddf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90108a30 0x7f0b9019d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.815+0000 7f0b95ddf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b90112bd0 con 0x7f0b90108a30 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.815+0000 7f0b95ddf700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0b90112a50 con 0x7f0b9006bb30 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.815+0000 7f0b95ddf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b90112d50 con 0x7f0b9006b190 2024-04-25T17:35:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90108a30 0x7f0b9019d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90108a30 0x7f0b9019d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38776/0 (socket says 172.21.15.98:38776) 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8ffff700 1 -- 172.21.15.98:0/4023804055 learned_addr learned my addr 172.21.15.98:0/4023804055 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8effd700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b9006bb30 0x7f0b901995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b9006b190 0x7f0b90199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8ffff700 1 -- 172.21.15.98:0/4023804055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b9006b190 msgr2=0x7f0b90199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8ffff700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b9006b190 0x7f0b90199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8ffff700 1 -- 172.21.15.98:0/4023804055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b9006bb30 msgr2=0x7f0b901995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8ffff700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b9006bb30 0x7f0b901995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8ffff700 1 -- 172.21.15.98:0/4023804055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b9019e040 con 0x7f0b90108a30 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b9006b190 0x7f0b90199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8ffff700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90108a30 0x7f0b9019d900 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0b880062a0 tx=0x7f0b88004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8cff9700 1 -- 172.21.15.98:0/4023804055 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b88021070 con 0x7f0b90108a30 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b95ddf700 1 -- 172.21.15.98:0/4023804055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b9019e2d0 con 0x7f0b90108a30 2024-04-25T17:35:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8cff9700 1 -- 172.21.15.98:0/4023804055 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b88007c80 con 0x7f0b90108a30 2024-04-25T17:35:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.816+0000 7f0b8cff9700 1 -- 172.21.15.98:0/4023804055 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8801d450 con 0x7f0b90108a30 2024-04-25T17:35:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.817+0000 7f0b95ddf700 1 -- 172.21.15.98:0/4023804055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b901a5bb0 con 0x7f0b90108a30 2024-04-25T17:35:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.818+0000 7f0b8cff9700 1 -- 172.21.15.98:0/4023804055 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0b88004020 con 0x7f0b90108a30 2024-04-25T17:35:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.819+0000 7f0b8cff9700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0b780a3c40 0x7f0b780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.819+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0b780a3c40 0x7f0b780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.819+0000 7f0b8cff9700 1 -- 172.21.15.98:0/4023804055 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f0b880d15f0 con 0x7f0b90108a30 2024-04-25T17:35:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.819+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0b780a3c40 0x7f0b780a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0b80005fd0 tx=0x7f0b8000a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0ba0668b80 1 Processor -- start 2024-04-25T17:35:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0ba0668b80 1 -- start start 2024-04-25T17:35:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 0x55900677f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x55900678a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900678a880 0x55900678cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5590066d2c40 con 0x55900677f9d0 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5590066798e0 con 0x55900677f0b0 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5590066cec00 con 0x55900678a880 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0b8effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x55900678a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0b8effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x55900678a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38788/0 (socket says 172.21.15.98:38788) 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0b8effd700 1 -- 172.21.15.98:0/3390684100 learned_addr learned my addr 172.21.15.98:0/3390684100 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.820+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/3390684100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 0x55900677f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.821+0000 7f0b8effd700 1 -- 172.21.15.98:0/3390684100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900678a880 msgr2=0x55900678cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.821+0000 7f0b8effd700 1 --2- 172.21.15.98:0/3390684100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900678a880 0x55900678cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.821+0000 7f0b8effd700 1 -- 172.21.15.98:0/3390684100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 msgr2=0x55900677f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.821+0000 7f0b8effd700 1 --2- 172.21.15.98:0/3390684100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 0x55900677f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.821+0000 7f0b8effd700 1 -- 172.21.15.98:0/3390684100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55900663d6f0 con 0x55900677f9d0 2024-04-25T17:35:06.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.821+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/3390684100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 0x55900677f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:06.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.821+0000 7f0b8effd700 1 --2- 172.21.15.98:0/3390684100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x55900678a340 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400ac80 tx=0x7f0b8400af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280e1e534d99e960 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:06.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.821+0000 7f0b75ffb700 1 -- 172.21.15.98:0/3390684100 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8401d070 con 0x55900677f9d0 2024-04-25T17:35:06.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.821+0000 7f0b75ffb700 1 -- 172.21.15.98:0/3390684100 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b8400fe30 con 0x55900677f9d0 2024-04-25T17:35:06.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.821+0000 7f0b75ffb700 1 -- 172.21.15.98:0/3390684100 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b84010770 con 0x55900677f9d0 2024-04-25T17:35:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 -- 172.21.15.98:0/3390684100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 msgr2=0x55900678a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/3390684100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x55900678a340 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400ac80 tx=0x7f0b8400af90 comp rx=0 tx=0).stop 2024-04-25T17:35:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 -- 172.21.15.98:0/3390684100 shutdown_connections 2024-04-25T17:35:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/3390684100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900678a880 0x55900678cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/3390684100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 0x55900677f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/3390684100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x55900678a340 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 -- 172.21.15.98:0/3390684100 >> 172.21.15.98:0/3390684100 conn(0x5590066ac840 msgr2=0x55900677b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 -- 172.21.15.98:0/3390684100 shutdown_connections 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 -- 172.21.15.98:0/3390684100 wait complete. 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 -- start start 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.822+0000 7f0ba0668b80 1 Processor -- start 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f0b0 0x559006789fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f9d0 0x5590067872b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900678a880 0x5590067877f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55900678d370 con 0x55900677f0b0 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55900668c560 con 0x55900677f9d0 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55900678d4f0 con 0x55900678a880 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0ba0668b80 10 client.0 _ll_register_callbacks cb 0x55900666b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f0b0 0x559006789fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f0b0 0x559006789fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38802/0 (socket says 172.21.15.98:38802) 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8f7fe700 1 -- 172.21.15.98:0/662379155 learned_addr learned my addr 172.21.15.98:0/662379155 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8effd700 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f9d0 0x5590067872b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8ffff700 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900678a880 0x5590067877f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8f7fe700 1 -- 172.21.15.98:0/662379155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900678a880 msgr2=0x5590067877f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900678a880 0x5590067877f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8f7fe700 1 -- 172.21.15.98:0/662379155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f9d0 msgr2=0x5590067872b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f9d0 0x5590067872b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8f7fe700 1 -- 172.21.15.98:0/662379155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559006620880 con 0x55900677f0b0 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f0b0 0x559006789fb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0b80010a30 tx=0x7f0b80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d469721bdbe2ba77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b6effd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b6effd700 1 -- 172.21.15.98:0/662379155 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8000eea0 con 0x55900677f0b0 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0b6effd700 1 -- 172.21.15.98:0/662379155 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f0b8000fa50 con 0x55900677f0b0 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.823+0000 7f0ba0668b80 1 -- 172.21.15.98:0/662379155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b8400fe30 con 0x55900677f0b0 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0b6effd700 1 -- 172.21.15.98:0/662379155 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0b8000fc90 con 0x55900677f0b0 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0ba0668b80 10 client.14985 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0ba0668b80 1 -- 172.21.15.98:0/662379155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559006797c80 con 0x55900677f0b0 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0b6effd700 1 -- 172.21.15.98:0/662379155 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b8400fe30 con 0x55900677f0b0 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0b6effd700 1 -- 172.21.15.98:0/662379155 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f0b80026490 con 0x55900677f0b0 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0ba0668b80 10 client.14985 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0ba0668b80 10 client.14985 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0ba0668b80 20 client.14985 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55900679a2e0 0x55900679b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0ba0668b80 4 client.14985 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0ba0668b80 1 -- 172.21.15.98:0/662379155 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590066e0430 con 0x55900679a2e0 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.824+0000 7f0b8effd700 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55900679a2e0 0x55900679b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.825+0000 7f0b6effd700 10 client.14985 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.825+0000 7f0b8effd700 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55900679a2e0 0x55900679b750 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf98755ce623559a server_cookie=740b909b76591eba in_seq=0 out_seq=0 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.825+0000 7f0b6effd700 1 -- 172.21.15.98:0/662379155 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590066e0430 con 0x55900679a2e0 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.825+0000 7f0b6effd700 10 client.14985 handle_command_reply: tid=0 2024-04-25T17:35:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 client.14985 shutdown 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 -- 172.21.15.98:0/662379155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f0b0 msgr2=0x559006789fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f0b0 0x559006789fb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0b80010a30 tx=0x7f0b80004750 comp rx=0 tx=0).stop 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 -- 172.21.15.98:0/662379155 shutdown_connections 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55900679a2e0 0x55900679b750 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900678a880 0x5590067877f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f0b0 0x559006789fb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0b6effd700 0 client.14985 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/662379155 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f9d0 0x5590067872b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 -- 172.21.15.98:0/662379155 >> 172.21.15.98:0/662379155 conn(0x5590066ac840 msgr2=0x55900677b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 -- 172.21.15.98:0/662379155 shutdown_connections 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 1 -- 172.21.15.98:0/662379155 wait complete. 2024-04-25T17:35:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.826+0000 7f0ba0668b80 20 client.14985 trim_cache size 0 max 16384 2024-04-25T17:35:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.846+0000 7f0ba0668b80 1 Processor -- start 2024-04-25T17:35:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.846+0000 7f0ba0668b80 1 -- start start 2024-04-25T17:35:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.846+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 0x55900677b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.846+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900677f9d0 0x559006787270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.846+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900678a880 0x5590067877b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.846+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559006799a90 con 0x55900678a880 2024-04-25T17:35:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.846+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55900679a160 con 0x55900677f0b0 2024-04-25T17:35:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.846+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55900679a830 con 0x55900677f9d0 2024-04-25T17:35:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b8ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900678a880 0x5590067877b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b8ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900678a880 0x5590067877b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38816/0 (socket says 172.21.15.98:38816) 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b8ffff700 1 -- 172.21.15.98:0/2830036011 learned_addr learned my addr 172.21.15.98:0/2830036011 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/2830036011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 0x55900677b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b8ffff700 1 -- 172.21.15.98:0/2830036011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900677f9d0 msgr2=0x559006787270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b8ffff700 1 --2- 172.21.15.98:0/2830036011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900677f9d0 0x559006787270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b8ffff700 1 -- 172.21.15.98:0/2830036011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 msgr2=0x55900677b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b8ffff700 1 --2- 172.21.15.98:0/2830036011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 0x55900677b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b8ffff700 1 -- 172.21.15.98:0/2830036011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55900677c2a0 con 0x55900678a880 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b8ffff700 1 --2- 172.21.15.98:0/2830036011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900678a880 0x5590067877b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800b830 tx=0x7f0b88009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3128130a3754549c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b757fa700 1 -- 172.21.15.98:0/2830036011 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8802a030 con 0x55900678a880 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b757fa700 1 -- 172.21.15.98:0/2830036011 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b8809e070 con 0x55900678a880 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.847+0000 7f0b757fa700 1 -- 172.21.15.98:0/2830036011 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b880a40b0 con 0x55900678a880 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2830036011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900678a880 msgr2=0x5590067877b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/2830036011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900678a880 0x5590067877b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800b830 tx=0x7f0b88009040 comp rx=0 tx=0).stop 2024-04-25T17:35:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2830036011 shutdown_connections 2024-04-25T17:35:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/2830036011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900678a880 0x5590067877b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/2830036011 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900677f0b0 0x55900677b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/2830036011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900677f9d0 0x559006787270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2830036011 >> 172.21.15.98:0/2830036011 conn(0x5590066ac840 msgr2=0x559006799140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2830036011 shutdown_connections 2024-04-25T17:35:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2830036011 wait complete. 2024-04-25T17:35:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 -- start start 2024-04-25T17:35:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.848+0000 7f0ba0668b80 1 Processor -- start 2024-04-25T17:35:06.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900677f0b0 0x55900678c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x5590067a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0ba0668b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900678a880 0x5590067a5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55900679b5d0 con 0x55900677f9d0 2024-04-25T17:35:06.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55900679af00 con 0x55900678a880 2024-04-25T17:35:06.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0ba0668b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55900679bca0 con 0x55900677f0b0 2024-04-25T17:35:06.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0ba0668b80 10 client.0 _ll_register_callbacks cb 0x55900666b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:06.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x5590067a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x5590067a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38822/0 (socket says 172.21.15.98:38822) 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8effd700 1 -- 172.21.15.98:0/2966898266 learned_addr learned my addr 172.21.15.98:0/2966898266 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900677f0b0 0x55900678c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8ffff700 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900678a880 0x5590067a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8effd700 1 -- 172.21.15.98:0/2966898266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900677f0b0 msgr2=0x55900678c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8effd700 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900677f0b0 0x55900678c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8effd700 1 -- 172.21.15.98:0/2966898266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900678a880 msgr2=0x5590067a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8effd700 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900678a880 0x5590067a5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8effd700 1 -- 172.21.15.98:0/2966898266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559006784460 con 0x55900677f9d0 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b8effd700 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x5590067a4bc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400ac50 tx=0x7f0b8400aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c949ea0f8226f5f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b6effd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b6effd700 1 -- 172.21.15.98:0/2966898266 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8402e030 con 0x55900677f9d0 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b6effd700 1 -- 172.21.15.98:0/2966898266 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f0b8400fc30 con 0x55900677f9d0 2024-04-25T17:35:06.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2966898266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b8809e070 con 0x55900677f9d0 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.849+0000 7f0b6effd700 1 -- 172.21.15.98:0/2966898266 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0b8401d040 con 0x55900677f9d0 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0ba0668b80 10 client.14991 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2966898266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559006805d80 con 0x55900677f9d0 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0b6effd700 1 -- 172.21.15.98:0/2966898266 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b8809e070 con 0x55900677f9d0 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0b6effd700 1 -- 172.21.15.98:0/2966898266 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f0b840053e0 con 0x55900677f9d0 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0ba0668b80 10 client.14991 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0ba0668b80 10 client.14991 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0ba0668b80 20 client.14991 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55900680a670 0x559006809380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0ba0668b80 4 client.14991 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2966898266 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "status"}) -- 0x5590066da4d0 con 0x55900680a670 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.850+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55900680a670 0x559006809380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0b6effd700 10 client.14991 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:35:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0b8f7fe700 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55900680a670 0x559006809380 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34d9b4beae6cb5b9 server_cookie=db98ef9b3cd54731 in_seq=0 out_seq=0 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0b6effd700 1 -- 172.21.15.98:0/2966898266 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5590066da4d0 con 0x55900680a670 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0b6effd700 10 client.14991 handle_command_reply: tid=0 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 client.14991 shutdown 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2966898266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 msgr2=0x5590067a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x5590067a4bc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400ac50 tx=0x7f0b8400aba0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2966898266 shutdown_connections 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55900680a670 0x559006809380 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55900677f0b0 0x55900678c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0b6effd700 0 client.14991 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55900678a880 0x5590067a5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 --2- 172.21.15.98:0/2966898266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55900677f9d0 0x5590067a4bc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2966898266 >> 172.21.15.98:0/2966898266 conn(0x5590066ac840 msgr2=0x559006799140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2966898266 shutdown_connections 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 1 -- 172.21.15.98:0/2966898266 wait complete. 2024-04-25T17:35:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.851+0000 7f0ba0668b80 20 client.14991 trim_cache size 0 max 16384 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 148.78172648399999, 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 25, 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T17:35:06.858 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 167.50918662399999 2024-04-25T17:35:06.859 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:35:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 -- 172.21.15.98:0/4023804055 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0b780a3c40 msgr2=0x7f0b780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0b780a3c40 0x7f0b780a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0b80005fd0 tx=0x7f0b8000a5e0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 -- 172.21.15.98:0/4023804055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90108a30 msgr2=0x7f0b9019d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90108a30 0x7f0b9019d900 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0b880062a0 tx=0x7f0b88004e20 comp rx=0 tx=0).stop 2024-04-25T17:35:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 -- 172.21.15.98:0/4023804055 shutdown_connections 2024-04-25T17:35:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0b780a3c40 0x7f0b780a6100 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b9006b190 0x7f0b90199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b90108a30 0x7f0b9019d900 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 --2- 172.21.15.98:0/4023804055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b9006bb30 0x7f0b901995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 -- 172.21.15.98:0/4023804055 >> 172.21.15.98:0/4023804055 conn(0x7f0b90078910 msgr2=0x7f0b900ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 -- 172.21.15.98:0/4023804055 shutdown_connections 2024-04-25T17:35:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:06.852+0000 7f0b95ddf700 1 -- 172.21.15.98:0/4023804055 wait complete. 2024-04-25T17:35:07.800 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T17:35:07.800 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T17:35:07.810 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:35:07.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T17:35:08.052 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:08.448 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:08 smithi098 ceph-mon[29192]: pgmap v380: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:08.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:08 smithi203 ceph-mon[32834]: pgmap v380: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:08 smithi173 ceph-mon[32856]: pgmap v380: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:09.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.879+0000 7f40752c4700 1 -- 172.21.15.98:0/1655264063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4070101990 msgr2=0x7f4070101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:09.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.879+0000 7f40752c4700 1 --2- 172.21.15.98:0/1655264063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4070101990 0x7f4070101db0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f40600099c0 tx=0x7f4060009cd0 comp rx=0 tx=0).stop 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.879+0000 7f40752c4700 1 -- 172.21.15.98:0/1655264063 shutdown_connections 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.879+0000 7f40752c4700 1 --2- 172.21.15.98:0/1655264063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4070103530 0x7f4070109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.879+0000 7f40752c4700 1 --2- 172.21.15.98:0/1655264063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4070101990 0x7f4070101db0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.879+0000 7f40752c4700 1 --2- 172.21.15.98:0/1655264063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4070102b90 0x7f4070102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.879+0000 7f40752c4700 1 -- 172.21.15.98:0/1655264063 >> 172.21.15.98:0/1655264063 conn(0x7f40700fcf30 msgr2=0x7f40700ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.879+0000 7f40752c4700 1 -- 172.21.15.98:0/1655264063 shutdown_connections 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.879+0000 7f40752c4700 1 -- 172.21.15.98:0/1655264063 wait complete. 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f40752c4700 1 Processor -- start 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f40752c4700 1 -- start start 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f40752c4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4070101990 0x7f407019fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f40752c4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4070102b90 0x7f407019ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f40752c4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4070103530 0x7f40701a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f40752c4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f407010d500 con 0x7f4070103530 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f40752c4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f407010d380 con 0x7f4070101990 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f40752c4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f407010d680 con 0x7f4070102b90 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f406f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4070103530 0x7f40701a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f406f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4070103530 0x7f40701a68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38846/0 (socket says 172.21.15.98:38846) 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f406f7fe700 1 -- 172.21.15.98:0/1853167088 learned_addr learned my addr 172.21.15.98:0/1853167088 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.880+0000 7f406e7fc700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4070102b90 0x7f407019ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f406effd700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4070101990 0x7f407019fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f406f7fe700 1 -- 172.21.15.98:0/1853167088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4070102b90 msgr2=0x7f407019ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f406f7fe700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4070102b90 0x7f407019ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f406f7fe700 1 -- 172.21.15.98:0/1853167088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4070101990 msgr2=0x7f407019fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f406f7fe700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4070101990 0x7f407019fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f406f7fe700 1 -- 172.21.15.98:0/1853167088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40701a6fa0 con 0x7f4070103530 2024-04-25T17:35:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f406f7fe700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4070103530 0x7f40701a68d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f406800b970 tx=0x7f406800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f4067fff700 1 -- 172.21.15.98:0/1853167088 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068019070 con 0x7f4070103530 2024-04-25T17:35:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f40752c4700 1 -- 172.21.15.98:0/1853167088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40701a7290 con 0x7f4070103530 2024-04-25T17:35:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f4067fff700 1 -- 172.21.15.98:0/1853167088 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f406800cde0 con 0x7f4070103530 2024-04-25T17:35:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f4067fff700 1 -- 172.21.15.98:0/1853167088 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4068007370 con 0x7f4070103530 2024-04-25T17:35:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.881+0000 7f40752c4700 1 -- 172.21.15.98:0/1853167088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40701a7800 con 0x7f4070103530 2024-04-25T17:35:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.882+0000 7f40752c4700 1 -- 172.21.15.98:0/1853167088 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4070069730 con 0x7f4070103530 2024-04-25T17:35:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.883+0000 7f4067fff700 1 -- 172.21.15.98:0/1853167088 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f406801c070 con 0x7f4070103530 2024-04-25T17:35:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.884+0000 7f4067fff700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f405c0a3c40 0x7f405c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.884+0000 7f406effd700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f405c0a3c40 0x7f405c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.884+0000 7f4067fff700 1 -- 172.21.15.98:0/1853167088 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f40680d0000 con 0x7f4070103530 2024-04-25T17:35:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.884+0000 7f406effd700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f405c0a3c40 0x7f405c0a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f406000b580 tx=0x7f4060005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:09.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:09.888+0000 7f4067fff700 1 -- 172.21.15.98:0/1853167088 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4068023070 con 0x7f4070103530 2024-04-25T17:35:10.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.107+0000 7f40752c4700 1 -- 172.21.15.98:0/1853167088 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f40700ff150 con 0x7f405c0a3c40 2024-04-25T17:35:10.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.108+0000 7f4067fff700 1 -- 172.21.15.98:0/1853167088 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f40700ff150 con 0x7f405c0a3c40 2024-04-25T17:35:10.109 INFO:teuthology.orchestra.run.smithi098.stdout:Backend: cephadm 2024-04-25T17:35:10.109 INFO:teuthology.orchestra.run.smithi098.stdout:Available: Yes 2024-04-25T17:35:10.109 INFO:teuthology.orchestra.run.smithi098.stdout:Paused: No 2024-04-25T17:35:10.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.110+0000 7f40752c4700 1 -- 172.21.15.98:0/1853167088 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f405c0a3c40 msgr2=0x7f405c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.110+0000 7f40752c4700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f405c0a3c40 0x7f405c0a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f406000b580 tx=0x7f4060005fb0 comp rx=0 tx=0).stop 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.110+0000 7f40752c4700 1 -- 172.21.15.98:0/1853167088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4070103530 msgr2=0x7f40701a68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.110+0000 7f40752c4700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4070103530 0x7f40701a68d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f406800b970 tx=0x7f406800bc80 comp rx=0 tx=0).stop 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.110+0000 7f40752c4700 1 -- 172.21.15.98:0/1853167088 shutdown_connections 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.110+0000 7f40752c4700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f405c0a3c40 0x7f405c0a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.110+0000 7f40752c4700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4070103530 0x7f40701a68d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.110+0000 7f40752c4700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4070101990 0x7f407019fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.110+0000 7f40752c4700 1 --2- 172.21.15.98:0/1853167088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4070102b90 0x7f407019ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.110+0000 7f40752c4700 1 -- 172.21.15.98:0/1853167088 >> 172.21.15.98:0/1853167088 conn(0x7f40700fcf30 msgr2=0x7f40700fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.111+0000 7f40752c4700 1 -- 172.21.15.98:0/1853167088 shutdown_connections 2024-04-25T17:35:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:10.111+0000 7f40752c4700 1 -- 172.21.15.98:0/1853167088 wait complete. 2024-04-25T17:35:10.149 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:10 smithi098 ceph-mon[29192]: pgmap v381: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:10 smithi203 ceph-mon[32834]: pgmap v381: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:10.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:10 smithi173 ceph-mon[32856]: pgmap v381: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:10.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T17:35:11.154 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:11.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:11 smithi098 ceph-mon[29192]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:11.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:11 smithi203 ceph-mon[32834]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:11.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:11 smithi173 ceph-mon[32856]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:12.367 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:12 smithi098 ceph-mon[29192]: pgmap v382: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:12.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:12 smithi203 ceph-mon[32834]: pgmap v382: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:12.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:12 smithi173 ceph-mon[32856]: pgmap v382: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.930+0000 7f21b84bd700 1 -- 172.21.15.98:0/3887913655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b0102aa0 msgr2=0x7f21b0102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.930+0000 7f21b84bd700 1 --2- 172.21.15.98:0/3887913655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b0102aa0 0x7f21b0102ec0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f21a800b0d0 tx=0x7f21a800b3e0 comp rx=0 tx=0).stop 2024-04-25T17:35:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.930+0000 7f21b84bd700 1 -- 172.21.15.98:0/3887913655 shutdown_connections 2024-04-25T17:35:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.930+0000 7f21b84bd700 1 --2- 172.21.15.98:0/3887913655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b0104640 0x7f21b010aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.930+0000 7f21b84bd700 1 --2- 172.21.15.98:0/3887913655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b0102aa0 0x7f21b0102ec0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.930+0000 7f21b84bd700 1 --2- 172.21.15.98:0/3887913655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0103ca0 0x7f21b0104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.930+0000 7f21b84bd700 1 -- 172.21.15.98:0/3887913655 >> 172.21.15.98:0/3887913655 conn(0x7f21b00fe040 msgr2=0x7f21b0100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.930+0000 7f21b84bd700 1 -- 172.21.15.98:0/3887913655 shutdown_connections 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.930+0000 7f21b84bd700 1 -- 172.21.15.98:0/3887913655 wait complete. 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b84bd700 1 Processor -- start 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b84bd700 1 -- start start 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b84bd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0102aa0 0x7f21b00767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b84bd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b0103ca0 0x7f21b0074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b84bd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b0104640 0x7f21b0075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b84bd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21b010e510 con 0x7f21b0103ca0 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b84bd700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f21b010e390 con 0x7f21b0104640 2024-04-25T17:35:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b84bd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21b010e690 con 0x7f21b0102aa0 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b5a58700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b0103ca0 0x7f21b0074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b5a58700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b0103ca0 0x7f21b0074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38864/0 (socket says 172.21.15.98:38864) 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b5a58700 1 -- 172.21.15.98:0/3592950318 learned_addr learned my addr 172.21.15.98:0/3592950318 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b6a5a700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b0104640 0x7f21b0075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b6259700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0102aa0 0x7f21b00767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b5a58700 1 -- 172.21.15.98:0/3592950318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0102aa0 msgr2=0x7f21b00767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b5a58700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0102aa0 0x7f21b00767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b5a58700 1 -- 172.21.15.98:0/3592950318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b0104640 msgr2=0x7f21b0075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b5a58700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b0104640 0x7f21b0075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.931+0000 7f21b5a58700 1 -- 172.21.15.98:0/3592950318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b0075c20 con 0x7f21b0103ca0 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.932+0000 7f21b5a58700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b0103ca0 0x7f21b0074e30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f21a000c9b0 tx=0x7f21a000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.932+0000 7f21a77fe700 1 -- 172.21.15.98:0/3592950318 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0007b00 con 0x7f21b0103ca0 2024-04-25T17:35:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.932+0000 7f21b84bd700 1 -- 172.21.15.98:0/3592950318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b01a2c80 con 0x7f21b0103ca0 2024-04-25T17:35:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.932+0000 7f21a77fe700 1 -- 172.21.15.98:0/3592950318 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21a0007ca0 con 0x7f21b0103ca0 2024-04-25T17:35:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.932+0000 7f21a77fe700 1 -- 172.21.15.98:0/3592950318 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0018730 con 0x7f21b0103ca0 2024-04-25T17:35:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.932+0000 7f21b84bd700 1 -- 172.21.15.98:0/3592950318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b01a3190 con 0x7f21b0103ca0 2024-04-25T17:35:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.933+0000 7f21b84bd700 1 -- 172.21.15.98:0/3592950318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21b0069730 con 0x7f21b0103ca0 2024-04-25T17:35:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.934+0000 7f21a77fe700 1 -- 172.21.15.98:0/3592950318 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f21a000f450 con 0x7f21b0103ca0 2024-04-25T17:35:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.934+0000 7f21a77fe700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f219c0a3c40 0x7f219c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.934+0000 7f21b6259700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f219c0a3c40 0x7f219c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.934+0000 7f21a77fe700 1 -- 172.21.15.98:0/3592950318 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f21a00d0450 con 0x7f21b0103ca0 2024-04-25T17:35:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.935+0000 7f21b6259700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f219c0a3c40 0x7f219c0a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f21a800bf60 tx=0x7f21a800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:12.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:12.939+0000 7f21a77fe700 1 -- 172.21.15.98:0/3592950318 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21a0014030 con 0x7f21b0103ca0 2024-04-25T17:35:13.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.148+0000 7f21b84bd700 1 -- 172.21.15.98:0/3592950318 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f21b0062e80 con 0x7f219c0a3c40 2024-04-25T17:35:13.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.161+0000 7f21a77fe700 1 -- 172.21.15.98:0/3592950318 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f21b0062e80 con 0x7f219c0a3c40 2024-04-25T17:35:13.162 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T17:35:13.162 INFO:teuthology.orchestra.run.smithi098.stdout:mds.a smithi098 running (2m) 32s ago 2m 14.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 f94962645788 2024-04-25T17:35:13.162 INFO:teuthology.orchestra.run.smithi098.stdout:mds.b smithi173 running (2m) 39s ago 2m 25.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 d433fc0d5710 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mds.c smithi203 running (3m) 39s ago 3m 20.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 9af6cac92cbc 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mds.d smithi098 running (3m) 32s ago 3m 14.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 beecde30ebb8 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mds.e smithi173 running (3m) 39s ago 3m 21.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 bad144b5616e 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mds.f smithi203 running (3m) 39s ago 3m 19.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 2eb811a66a3f 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mds.g smithi098 running (3m) 32s ago 3m 14.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 b13d255fce75 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mds.h smithi173 running (3m) 39s ago 3m 20.7M - 19.0.0-3156-gd69f13d8 da4b004c09e8 ab34c40f9d58 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mds.i smithi203 running (3m) 39s ago 3m 20.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 8e4ed0f945d2 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mds.j smithi098 running (3m) 32s ago 3m 16.6M - 19.0.0-3156-gd69f13d8 da4b004c09e8 7541741ae1be 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mds.k smithi173 running (3m) 39s ago 3m 15.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 48a48d1de11c 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mds.l smithi203 running (3m) 39s ago 3m 15.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 e1e5757a8f29 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi098 *:9283,8765 running (10m) 32s ago 10m 514M - 19.0.0-3156-gd69f13d8 da4b004c09e8 d234cf7edff5 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi173 *:8443,8765 running (8m) 39s ago 8m 455M - 19.0.0-3156-gd69f13d8 da4b004c09e8 4799370d0d30 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.z smithi203 *:8443,8765 running (8m) 39s ago 8m 455M - 19.0.0-3156-gd69f13d8 da4b004c09e8 00d2f9f98858 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (10m) 32s ago 10m 54.3M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 2fd9d01e54ef 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi173 running (9m) 39s ago 9m 38.5M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 a6b8778b700b 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi203 running (9m) 39s ago 9m 42.7M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 6c75708555c3 2024-04-25T17:35:13.163 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (8m) 32s ago 8m 45.1M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 f9817654b41b 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi173 running (8m) 39s ago 8m 49.2M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 9c8e8c3a8c16 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi203 running (7m) 39s ago 7m 46.5M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 b787bc3f334d 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (7m) 32s ago 7m 43.6M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 751eb5674678 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi173 running (6m) 39s ago 6m 56.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 93d596e474a4 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi203 running (6m) 39s ago 6m 57.5M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 b266277954d2 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi098 running (6m) 32s ago 6m 41.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 0de12907b006 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi173 running (5m) 39s ago 5m 57.6M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 94152295af34 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 smithi203 running (5m) 39s ago 5m 43.2M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 241567a7fa06 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 smithi098 running (4m) 32s ago 4m 54.6M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 7625593bf82e 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 smithi173 running (4m) 39s ago 4m 45.1M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 d6eb83e1cc04 2024-04-25T17:35:13.164 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 smithi203 running (3m) 39s ago 3m 41.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 6b92e27d9b89 2024-04-25T17:35:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 -- 172.21.15.98:0/3592950318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f219c0a3c40 msgr2=0x7f219c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f219c0a3c40 0x7f219c0a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f21a800bf60 tx=0x7f21a800b040 comp rx=0 tx=0).stop 2024-04-25T17:35:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 -- 172.21.15.98:0/3592950318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b0103ca0 msgr2=0x7f21b0074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b0103ca0 0x7f21b0074e30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f21a000c9b0 tx=0x7f21a000cd70 comp rx=0 tx=0).stop 2024-04-25T17:35:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 -- 172.21.15.98:0/3592950318 shutdown_connections 2024-04-25T17:35:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f219c0a3c40 0x7f219c0a6100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21b0102aa0 0x7f21b00767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b0104640 0x7f21b0075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 --2- 172.21.15.98:0/3592950318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21b0103ca0 0x7f21b0074e30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:13.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 -- 172.21.15.98:0/3592950318 >> 172.21.15.98:0/3592950318 conn(0x7f21b00fe040 msgr2=0x7f21b0106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:13.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 -- 172.21.15.98:0/3592950318 shutdown_connections 2024-04-25T17:35:13.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:13.164+0000 7f21b84bd700 1 -- 172.21.15.98:0/3592950318 wait complete. 2024-04-25T17:35:14.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T17:35:14.354 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:14.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:14 smithi098 ceph-mon[29192]: pgmap v383: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:35:14.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:14 smithi098 ceph-mon[29192]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:14.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:14 smithi203 ceph-mon[32834]: pgmap v383: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:35:14.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:14 smithi203 ceph-mon[32834]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:14.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:14 smithi173 ceph-mon[32856]: pgmap v383: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:35:14.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:14 smithi173 ceph-mon[32856]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.991+0000 7ff813d43700 1 -- 172.21.15.98:0/101452849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c102a90 msgr2=0x7ff80c102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:15.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.991+0000 7ff813d43700 1 --2- 172.21.15.98:0/101452849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c102a90 0x7ff80c102eb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff8080099c0 tx=0x7ff808009cd0 comp rx=0 tx=0).stop 2024-04-25T17:35:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.991+0000 7ff813d43700 1 -- 172.21.15.98:0/101452849 shutdown_connections 2024-04-25T17:35:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.991+0000 7ff813d43700 1 --2- 172.21.15.98:0/101452849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c104630 0x7ff80c10aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.991+0000 7ff813d43700 1 --2- 172.21.15.98:0/101452849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c102a90 0x7ff80c102eb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.991+0000 7ff813d43700 1 --2- 172.21.15.98:0/101452849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff80c103c90 0x7ff80c1040f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.991+0000 7ff813d43700 1 -- 172.21.15.98:0/101452849 >> 172.21.15.98:0/101452849 conn(0x7ff80c0fe030 msgr2=0x7ff80c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.991+0000 7ff813d43700 1 -- 172.21.15.98:0/101452849 shutdown_connections 2024-04-25T17:35:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.991+0000 7ff813d43700 1 -- 172.21.15.98:0/101452849 wait complete. 2024-04-25T17:35:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff813d43700 1 Processor -- start 2024-04-25T17:35:15.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff813d43700 1 -- start start 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff813d43700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff80c102a90 0x7ff80c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff813d43700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c103c90 0x7ff80c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff813d43700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c104630 0x7ff80c1a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff813d43700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c10e520 con 0x7ff80c103c90 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff813d43700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c10e3a0 con 0x7ff80c102a90 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff813d43700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c10e6a0 con 0x7ff80c104630 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff8112de700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c103c90 0x7ff80c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff8112de700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c103c90 0x7ff80c1991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39468/0 (socket says 172.21.15.98:39468) 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.992+0000 7ff8112de700 1 -- 172.21.15.98:0/2458951263 learned_addr learned my addr 172.21.15.98:0/2458951263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff8112de700 1 -- 172.21.15.98:0/2458951263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c104630 msgr2=0x7ff80c1a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff8112de700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c104630 0x7ff80c1a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff8112de700 1 -- 172.21.15.98:0/2458951263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff80c102a90 msgr2=0x7ff80c198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff811adf700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff80c102a90 0x7ff80c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff8112de700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff80c102a90 0x7ff80c198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff8112de700 1 -- 172.21.15.98:0/2458951263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c1a0860 con 0x7ff80c103c90 2024-04-25T17:35:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff8112de700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c103c90 0x7ff80c1991a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00e9a0 tx=0x7ff7fc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff802ffd700 1 -- 172.21.15.98:0/2458951263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc00cbd0 con 0x7ff80c103c90 2024-04-25T17:35:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff802ffd700 1 -- 172.21.15.98:0/2458951263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7fc00cd70 con 0x7ff80c103c90 2024-04-25T17:35:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff802ffd700 1 -- 172.21.15.98:0/2458951263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc018680 con 0x7ff80c103c90 2024-04-25T17:35:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff813d43700 1 -- 172.21.15.98:0/2458951263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80c1a0b50 con 0x7ff80c103c90 2024-04-25T17:35:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.993+0000 7ff813d43700 1 -- 172.21.15.98:0/2458951263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80c1a1090 con 0x7ff80c103c90 2024-04-25T17:35:15.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.994+0000 7ff813d43700 1 -- 172.21.15.98:0/2458951263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff80c069770 con 0x7ff80c103c90 2024-04-25T17:35:15.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.995+0000 7ff802ffd700 1 -- 172.21.15.98:0/2458951263 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff7fc010970 con 0x7ff80c103c90 2024-04-25T17:35:15.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.996+0000 7ff802ffd700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7f80a3c40 0x7ff7f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:15.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.996+0000 7ff811adf700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7f80a3c40 0x7ff7f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:15.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.996+0000 7ff802ffd700 1 -- 172.21.15.98:0/2458951263 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7ff7fc0d0bc0 con 0x7ff80c103c90 2024-04-25T17:35:15.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:15.996+0000 7ff811adf700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7f80a3c40 0x7ff7f80a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff808000c00 tx=0x7ff808005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:16.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.000+0000 7ff802ffd700 1 -- 172.21.15.98:0/2458951263 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7fc09d5b0 con 0x7ff80c103c90 2024-04-25T17:35:16.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.207+0000 7ff813d43700 1 -- 172.21.15.98:0/2458951263 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ff80c062e80 con 0x7ff7f80a3c40 2024-04-25T17:35:16.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.214+0000 7ff802ffd700 1 -- 172.21.15.98:0/2458951263 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ff80c062e80 con 0x7ff7f80a3c40 2024-04-25T17:35:16.214 INFO:teuthology.orchestra.run.smithi098.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T17:35:16.214 INFO:teuthology.orchestra.run.smithi098.stdout:mds.all 12/12 42s ago 3m smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-25T17:35:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:mgr 3/3 42s ago 8m smithi098=x;smithi173=y;smithi203=z;count:3 2024-04-25T17:35:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:mon 3/3 42s ago 9m smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c;count:3 2024-04-25T17:35:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:osd 12 42s ago - 2024-04-25T17:35:16.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 -- 172.21.15.98:0/2458951263 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7f80a3c40 msgr2=0x7ff7f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7f80a3c40 0x7ff7f80a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff808000c00 tx=0x7ff808005fd0 comp rx=0 tx=0).stop 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 -- 172.21.15.98:0/2458951263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c103c90 msgr2=0x7ff80c1991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c103c90 0x7ff80c1991a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00e9a0 tx=0x7ff7fc00ed60 comp rx=0 tx=0).stop 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 -- 172.21.15.98:0/2458951263 shutdown_connections 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff7f80a3c40 0x7ff7f80a6100 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff80c104630 0x7ff80c1a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff80c102a90 0x7ff80c198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 --2- 172.21.15.98:0/2458951263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff80c103c90 0x7ff80c1991a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 -- 172.21.15.98:0/2458951263 >> 172.21.15.98:0/2458951263 conn(0x7ff80c0fe030 msgr2=0x7ff80c1070b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 -- 172.21.15.98:0/2458951263 shutdown_connections 2024-04-25T17:35:16.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:16.216+0000 7ff813d43700 1 -- 172.21.15.98:0/2458951263 wait complete. 2024-04-25T17:35:16.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:16 smithi098 ceph-mon[29192]: pgmap v384: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:16 smithi203 ceph-mon[32834]: pgmap v384: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:16.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:16 smithi173 ceph-mon[32856]: pgmap v384: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:16.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T17:35:17.210 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:18.448 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:18 smithi098 ceph-mon[29192]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:18.448 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:18 smithi098 ceph-mon[29192]: pgmap v385: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:18.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:18 smithi203 ceph-mon[32834]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:18.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:18 smithi203 ceph-mon[32834]: pgmap v385: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:18.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:18 smithi173 ceph-mon[32856]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:18.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:18 smithi173 ceph-mon[32856]: pgmap v385: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.006+0000 7f156e8a5700 1 -- 172.21.15.98:0/2101438303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1568102ac0 msgr2=0x7f1568102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.006+0000 7f156e8a5700 1 --2- 172.21.15.98:0/2101438303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1568102ac0 0x7f1568102ee0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f155c0099c0 tx=0x7f155c009cd0 comp rx=0 tx=0).stop 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.006+0000 7f156e8a5700 1 -- 172.21.15.98:0/2101438303 shutdown_connections 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.006+0000 7f156e8a5700 1 --2- 172.21.15.98:0/2101438303 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1568104660 0x7f156810af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.006+0000 7f156e8a5700 1 --2- 172.21.15.98:0/2101438303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1568102ac0 0x7f1568102ee0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.006+0000 7f156e8a5700 1 --2- 172.21.15.98:0/2101438303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1568103cc0 0x7f1568104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.006+0000 7f156e8a5700 1 -- 172.21.15.98:0/2101438303 >> 172.21.15.98:0/2101438303 conn(0x7f15680fe060 msgr2=0x7f15681004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.006+0000 7f156e8a5700 1 -- 172.21.15.98:0/2101438303 shutdown_connections 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.006+0000 7f156e8a5700 1 -- 172.21.15.98:0/2101438303 wait complete. 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f156e8a5700 1 Processor -- start 2024-04-25T17:35:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f156e8a5700 1 -- start start 2024-04-25T17:35:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f156e8a5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1568102ac0 0x7f1568198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f156e8a5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1568103cc0 0x7f1568199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f156e8a5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1568104660 0x7f15681a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f156e8a5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f156810e630 con 0x7f1568102ac0 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f156e8a5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f156810e4b0 con 0x7f1568104660 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f156e8a5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f156810e7b0 con 0x7f1568103cc0 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f1567fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1568102ac0 0x7f1568198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f1567fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1568102ac0 0x7f1568198ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39486/0 (socket says 172.21.15.98:39486) 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.007+0000 7f1567fff700 1 -- 172.21.15.98:0/4195656114 learned_addr learned my addr 172.21.15.98:0/4195656114 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f156ce42700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1568104660 0x7f15681a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f15677fe700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1568103cc0 0x7f1568199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f1567fff700 1 -- 172.21.15.98:0/4195656114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1568103cc0 msgr2=0x7f1568199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f1567fff700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1568103cc0 0x7f1568199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f1567fff700 1 -- 172.21.15.98:0/4195656114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1568104660 msgr2=0x7f15681a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f1567fff700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1568104660 0x7f15681a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f1567fff700 1 -- 172.21.15.98:0/4195656114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15681a0930 con 0x7f1568102ac0 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f156ce42700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1568104660 0x7f15681a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f1567fff700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1568102ac0 0x7f1568198ce0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f155c009f50 tx=0x7f155c003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f15657fa700 1 -- 172.21.15.98:0/4195656114 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c027070 con 0x7f1568102ac0 2024-04-25T17:35:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f15657fa700 1 -- 172.21.15.98:0/4195656114 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f155c022470 con 0x7f1568102ac0 2024-04-25T17:35:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f156e8a5700 1 -- 172.21.15.98:0/4195656114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15681a0bc0 con 0x7f1568102ac0 2024-04-25T17:35:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f15657fa700 1 -- 172.21.15.98:0/4195656114 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c00f6c0 con 0x7f1568102ac0 2024-04-25T17:35:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.008+0000 7f156e8a5700 1 -- 172.21.15.98:0/4195656114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15681a10a0 con 0x7f1568102ac0 2024-04-25T17:35:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.009+0000 7f156e8a5700 1 -- 172.21.15.98:0/4195656114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1568069730 con 0x7f1568102ac0 2024-04-25T17:35:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.010+0000 7f15657fa700 1 -- 172.21.15.98:0/4195656114 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f155c02a070 con 0x7f1568102ac0 2024-04-25T17:35:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.011+0000 7f15657fa700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15500a3c40 0x7f15500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.011+0000 7f15677fe700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15500a3c40 0x7f15500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.011+0000 7f15657fa700 1 -- 172.21.15.98:0/4195656114 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f155c0d1460 con 0x7f1568102ac0 2024-04-25T17:35:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.011+0000 7f15677fe700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15500a3c40 0x7f15500a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1558006fd0 tx=0x7f1558008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.016+0000 7f15657fa700 1 -- 172.21.15.98:0/4195656114 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f155c013070 con 0x7f1568102ac0 2024-04-25T17:35:19.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.220+0000 7f156e8a5700 1 -- 172.21.15.98:0/4195656114 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f1568062e80 con 0x7f15500a3c40 2024-04-25T17:35:19.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.222+0000 7f15657fa700 1 -- 172.21.15.98:0/4195656114 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f1568062e80 con 0x7f15500a3c40 2024-04-25T17:35:19.222 INFO:teuthology.orchestra.run.smithi098.stdout:HOST ADDR LABELS STATUS 2024-04-25T17:35:19.223 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 172.21.15.98 2024-04-25T17:35:19.223 INFO:teuthology.orchestra.run.smithi098.stdout:smithi173 172.21.15.173 2024-04-25T17:35:19.223 INFO:teuthology.orchestra.run.smithi098.stdout:smithi203 172.21.15.203 2024-04-25T17:35:19.223 INFO:teuthology.orchestra.run.smithi098.stdout:3 hosts in cluster 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 -- 172.21.15.98:0/4195656114 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15500a3c40 msgr2=0x7f15500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15500a3c40 0x7f15500a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1558006fd0 tx=0x7f1558008040 comp rx=0 tx=0).stop 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 -- 172.21.15.98:0/4195656114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1568102ac0 msgr2=0x7f1568198ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1568102ac0 0x7f1568198ce0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f155c009f50 tx=0x7f155c003a10 comp rx=0 tx=0).stop 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 -- 172.21.15.98:0/4195656114 shutdown_connections 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f15500a3c40 0x7f15500a6100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1568104660 0x7f15681a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1568102ac0 0x7f1568198ce0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 --2- 172.21.15.98:0/4195656114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1568103cc0 0x7f1568199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 -- 172.21.15.98:0/4195656114 >> 172.21.15.98:0/4195656114 conn(0x7f15680fe060 msgr2=0x7f1568107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 -- 172.21.15.98:0/4195656114 shutdown_connections 2024-04-25T17:35:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:19.224+0000 7f156e8a5700 1 -- 172.21.15.98:0/4195656114 wait complete. 2024-04-25T17:35:19.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T17:35:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:20.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:20 smithi098 ceph-mon[29192]: pgmap v386: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:20.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:20 smithi203 ceph-mon[32834]: pgmap v386: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:20.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:20 smithi173 ceph-mon[32856]: pgmap v386: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:21.432 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:21 smithi098 ceph-mon[29192]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:21.469 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:21 smithi203 ceph-mon[32834]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:21.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:21 smithi173 ceph-mon[32856]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.995+0000 7fb0c12ee700 1 -- 172.21.15.98:0/3596333997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc103540 msgr2=0x7fb0bc109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.995+0000 7fb0c12ee700 1 --2- 172.21.15.98:0/3596333997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc103540 0x7fb0bc109df0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb0b400b0d0 tx=0x7fb0b400b3e0 comp rx=0 tx=0).stop 2024-04-25T17:35:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.995+0000 7fb0c12ee700 1 -- 172.21.15.98:0/3596333997 shutdown_connections 2024-04-25T17:35:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.995+0000 7fb0c12ee700 1 --2- 172.21.15.98:0/3596333997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc103540 0x7fb0bc109df0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.995+0000 7fb0c12ee700 1 --2- 172.21.15.98:0/3596333997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0bc1019a0 0x7fb0bc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.995+0000 7fb0c12ee700 1 --2- 172.21.15.98:0/3596333997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc102ba0 0x7fb0bc103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.995+0000 7fb0c12ee700 1 -- 172.21.15.98:0/3596333997 >> 172.21.15.98:0/3596333997 conn(0x7fb0bc0fcf40 msgr2=0x7fb0bc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.995+0000 7fb0c12ee700 1 -- 172.21.15.98:0/3596333997 shutdown_connections 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.995+0000 7fb0c12ee700 1 -- 172.21.15.98:0/3596333997 wait complete. 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0c12ee700 1 Processor -- start 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0c12ee700 1 -- start start 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0c12ee700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc1019a0 0x7fb0bc06b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0c12ee700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0bc102ba0 0x7fb0bc19d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0c12ee700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc103540 0x7fb0bc19ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0c12ee700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc10d490 con 0x7fb0bc1019a0 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0c12ee700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc10d310 con 0x7fb0bc102ba0 2024-04-25T17:35:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0c12ee700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc10d610 con 0x7fb0bc103540 2024-04-25T17:35:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0baffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc1019a0 0x7fb0bc06b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0baffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc1019a0 0x7fb0bc06b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39498/0 (socket says 172.21.15.98:39498) 2024-04-25T17:35:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0baffd700 1 -- 172.21.15.98:0/4178277655 learned_addr learned my addr 172.21.15.98:0/4178277655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.996+0000 7fb0ba7fc700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0bc102ba0 0x7fb0bc19d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0bb7fe700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc103540 0x7fb0bc19ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0baffd700 1 -- 172.21.15.98:0/4178277655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc103540 msgr2=0x7fb0bc19ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0baffd700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc103540 0x7fb0bc19ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0baffd700 1 -- 172.21.15.98:0/4178277655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0bc102ba0 msgr2=0x7fb0bc19d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0baffd700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0bc102ba0 0x7fb0bc19d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0baffd700 1 -- 172.21.15.98:0/4178277655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0bc1a0870 con 0x7fb0bc1019a0 2024-04-25T17:35:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0bb7fe700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc103540 0x7fb0bc19ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:35:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0ba7fc700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0bc102ba0 0x7fb0bc19d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0baffd700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc1019a0 0x7fb0bc06b280 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00d980 tx=0x7fb0ac00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0a3fff700 1 -- 172.21.15.98:0/4178277655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac017030 con 0x7fb0bc1019a0 2024-04-25T17:35:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0c12ee700 1 -- 172.21.15.98:0/4178277655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0bc1a0b60 con 0x7fb0bc1019a0 2024-04-25T17:35:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0a3fff700 1 -- 172.21.15.98:0/4178277655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0ac0044d0 con 0x7fb0bc1019a0 2024-04-25T17:35:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0a3fff700 1 -- 172.21.15.98:0/4178277655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac003aa0 con 0x7fb0bc1019a0 2024-04-25T17:35:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.997+0000 7fb0c12ee700 1 -- 172.21.15.98:0/4178277655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0bc1a10f0 con 0x7fb0bc1019a0 2024-04-25T17:35:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.998+0000 7fb0c12ee700 1 -- 172.21.15.98:0/4178277655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0bc069730 con 0x7fb0bc1019a0 2024-04-25T17:35:22.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:21.999+0000 7fb0a3fff700 1 -- 172.21.15.98:0/4178277655 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb0ac01a070 con 0x7fb0bc1019a0 2024-04-25T17:35:22.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.000+0000 7fb0a3fff700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb0a40a3c40 0x7fb0a40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:22.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.000+0000 7fb0ba7fc700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb0a40a3c40 0x7fb0a40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:22.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.000+0000 7fb0a3fff700 1 -- 172.21.15.98:0/4178277655 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fb0ac0cf5d0 con 0x7fb0bc1019a0 2024-04-25T17:35:22.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.000+0000 7fb0ba7fc700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb0a40a3c40 0x7fb0a40a6100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc06b800 tx=0x7fb0b0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:22.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.004+0000 7fb0a3fff700 1 -- 172.21.15.98:0/4178277655 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb0ac09bfc0 con 0x7fb0bc1019a0 2024-04-25T17:35:22.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.213+0000 7fb0c12ee700 1 -- 172.21.15.98:0/4178277655 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fb0bc062e80 con 0x7fb0a40a3c40 2024-04-25T17:35:22.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.215+0000 7fb0a3fff700 1 -- 172.21.15.98:0/4178277655 <== mgr.14152 v2:172.21.15.98:6800/4145487482 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fb0bc062e80 con 0x7fb0a40a3c40 2024-04-25T17:35:22.216 INFO:teuthology.orchestra.run.smithi098.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T17:35:22.216 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-25T17:35:22.216 INFO:teuthology.orchestra.run.smithi098.stdout:smithi173 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204019S400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-25T17:35:22.216 INFO:teuthology.orchestra.run.smithi098.stdout:smithi203 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T17:35:22.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 -- 172.21.15.98:0/4178277655 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb0a40a3c40 msgr2=0x7fb0a40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:22.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb0a40a3c40 0x7fb0a40a6100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc06b800 tx=0x7fb0b0008040 comp rx=0 tx=0).stop 2024-04-25T17:35:22.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 -- 172.21.15.98:0/4178277655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc1019a0 msgr2=0x7fb0bc06b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:22.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc1019a0 0x7fb0bc06b280 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00d980 tx=0x7fb0ac00dd40 comp rx=0 tx=0).stop 2024-04-25T17:35:22.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 -- 172.21.15.98:0/4178277655 shutdown_connections 2024-04-25T17:35:22.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb0a40a3c40 0x7fb0a40a6100 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:22.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc103540 0x7fb0bc19ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:22.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0bc1019a0 0x7fb0bc06b280 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:22.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 --2- 172.21.15.98:0/4178277655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0bc102ba0 0x7fb0bc19d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:22.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 -- 172.21.15.98:0/4178277655 >> 172.21.15.98:0/4178277655 conn(0x7fb0bc0fcf40 msgr2=0x7fb0bc105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:22.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 -- 172.21.15.98:0/4178277655 shutdown_connections 2024-04-25T17:35:22.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:22.217+0000 7fb0c12ee700 1 -- 172.21.15.98:0/4178277655 wait complete. 2024-04-25T17:35:22.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:22 smithi098 ceph-mon[29192]: pgmap v387: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:22.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:22 smithi203 ceph-mon[32834]: pgmap v387: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:22.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:22 smithi173 ceph-mon[32856]: pgmap v387: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:23.037 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T17:35:23.046 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:35:23.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T17:35:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:24.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:24 smithi098 ceph-mon[29192]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:24.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:24 smithi098 ceph-mon[29192]: pgmap v388: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:35:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:24 smithi203 ceph-mon[32834]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:24 smithi203 ceph-mon[32834]: pgmap v388: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:35:24.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:24 smithi173 ceph-mon[32856]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:24.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:24 smithi173 ceph-mon[32856]: pgmap v388: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-25T17:35:25.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.073+0000 7fbba1ce8700 1 -- 172.21.15.98:0/593821961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9c102aa0 msgr2=0x7fbb9c102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:25.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.073+0000 7fbba1ce8700 1 --2- 172.21.15.98:0/593821961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9c102aa0 0x7fbb9c102ec0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00b0d0 tx=0x7fbb8c00b3e0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.073+0000 7fbba1ce8700 1 -- 172.21.15.98:0/593821961 shutdown_connections 2024-04-25T17:35:25.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.073+0000 7fbba1ce8700 1 --2- 172.21.15.98:0/593821961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb9c104640 0x7fbb9c10aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.073+0000 7fbba1ce8700 1 --2- 172.21.15.98:0/593821961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9c102aa0 0x7fbb9c102ec0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.073+0000 7fbba1ce8700 1 --2- 172.21.15.98:0/593821961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9c103ca0 0x7fbb9c104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.073+0000 7fbba1ce8700 1 -- 172.21.15.98:0/593821961 >> 172.21.15.98:0/593821961 conn(0x7fbb9c0fe040 msgr2=0x7fbb9c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:25.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.073+0000 7fbba1ce8700 1 -- 172.21.15.98:0/593821961 shutdown_connections 2024-04-25T17:35:25.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.073+0000 7fbba1ce8700 1 -- 172.21.15.98:0/593821961 wait complete. 2024-04-25T17:35:25.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.073+0000 7fbba1ce8700 1 Processor -- start 2024-04-25T17:35:25.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbba1ce8700 1 -- start start 2024-04-25T17:35:25.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbba1ce8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9c102aa0 0x7fbb9c0767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:25.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbba1ce8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb9c103ca0 0x7fbb9c074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:25.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbba1ce8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9c104640 0x7fbb9c075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:25.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbba1ce8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c10e530 con 0x7fbb9c102aa0 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbba1ce8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c10e3b0 con 0x7fbb9c103ca0 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbba1ce8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c10e6b0 con 0x7fbb9c104640 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9c102aa0 0x7fbb9c0767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9c102aa0 0x7fbb9c0767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60994/0 (socket says 172.21.15.98:60994) 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9b7fe700 1 -- 172.21.15.98:0/3178369145 learned_addr learned my addr 172.21.15.98:0/3178369145 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9affd700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb9c103ca0 0x7fbb9c074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9bfff700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9c104640 0x7fbb9c075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9b7fe700 1 -- 172.21.15.98:0/3178369145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9c104640 msgr2=0x7fbb9c075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9b7fe700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9c104640 0x7fbb9c075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9b7fe700 1 -- 172.21.15.98:0/3178369145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb9c103ca0 msgr2=0x7fbb9c074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9b7fe700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb9c103ca0 0x7fbb9c074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9b7fe700 1 -- 172.21.15.98:0/3178369145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9c075c20 con 0x7fbb9c102aa0 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.074+0000 7fbb9b7fe700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9c102aa0 0x7fbb9c0767e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009040 tx=0x7fbb8c004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:25.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.075+0000 7fbb98ff9700 1 -- 172.21.15.98:0/3178369145 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c01f030 con 0x7fbb9c102aa0 2024-04-25T17:35:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.075+0000 7fbba1ce8700 1 -- 172.21.15.98:0/3178369145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9c1a2d00 con 0x7fbb9c102aa0 2024-04-25T17:35:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.075+0000 7fbb98ff9700 1 -- 172.21.15.98:0/3178369145 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb8c003a00 con 0x7fbb9c102aa0 2024-04-25T17:35:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.075+0000 7fbb98ff9700 1 -- 172.21.15.98:0/3178369145 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c004630 con 0x7fbb9c102aa0 2024-04-25T17:35:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.075+0000 7fbba1ce8700 1 -- 172.21.15.98:0/3178369145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb9c1a3150 con 0x7fbb9c102aa0 2024-04-25T17:35:25.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.075+0000 7fbba1ce8700 1 -- 172.21.15.98:0/3178369145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb9c069730 con 0x7fbb9c102aa0 2024-04-25T17:35:25.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.077+0000 7fbb98ff9700 1 -- 172.21.15.98:0/3178369145 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbb8c003bb0 con 0x7fbb9c102aa0 2024-04-25T17:35:25.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.077+0000 7fbb98ff9700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fbb840a3c40 0x7fbb840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:25.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.077+0000 7fbb9affd700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fbb840a3c40 0x7fbb840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:25.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.078+0000 7fbb98ff9700 1 -- 172.21.15.98:0/3178369145 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fbb8c00f020 con 0x7fbb9c102aa0 2024-04-25T17:35:25.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.078+0000 7fbb9affd700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fbb840a3c40 0x7fbb840a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbb90004630 tx=0x7fbb9000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:25.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.081+0000 7fbb98ff9700 1 -- 172.21.15.98:0/3178369145 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb8c029020 con 0x7fbb9c102aa0 2024-04-25T17:35:25.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.327+0000 7fbba1ce8700 1 -- 172.21.15.98:0/3178369145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fbb9c02e000 con 0x7fbb9c102aa0 2024-04-25T17:35:25.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.327+0000 7fbb98ff9700 1 -- 172.21.15.98:0/3178369145 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3054 (secure 0 0 0) 0x7fbb8c09e530 con 0x7fbb9c102aa0 2024-04-25T17:35:25.328 INFO:teuthology.orchestra.run.smithi098.stdout:e25 2024-04-25T17:35:25.328 INFO:teuthology.orchestra.run.smithi098.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T17:35:25.328 INFO:teuthology.orchestra.run.smithi098.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T17:35:25.328 INFO:teuthology.orchestra.run.smithi098.stdout:legacy client fscid: 1 2024-04-25T17:35:25.328 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:35:25.328 INFO:teuthology.orchestra.run.smithi098.stdout:Filesystem 'cephfs' (1) 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:fs_name cephfs 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 25 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-04-25T17:32:37.173463+0000 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-04-25T17:32:59.314849+0000 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:tableserver 0 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:root 0 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:session_timeout 300 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:session_autoclose 300 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:max_file_size 1099511627776 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:max_xattr_size 65536 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:required_client_features {} 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure 0 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure_osd_epoch 0 2024-04-25T17:35:25.329 INFO:teuthology.orchestra.run.smithi098.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:max_mds 3 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:in 0,1,2 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:up {0=24461,1=24445,2=24440} 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:failed 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:damaged 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:stopped 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:data_pools [3] 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:metadata_pool 2 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:inline_data disabled 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:balancer 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:bal_rank_mask -1 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:standby_count_wanted 0 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:qdb_cluster leader: 24461 members: 24445,24440,24461 2024-04-25T17:35:25.330 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b{0:24461} state up:active seq 6 addr [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c{0:24430} state up:standby-replay seq 1 addr [v2:172.21.15.203:6836/1057173981,v1:172.21.15.203:6837/1057173981] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.i{1:24445} state up:active seq 13 addr [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f{1:24424} state up:standby-replay seq 1 addr [v2:172.21.15.203:6834/2803298626,v1:172.21.15.203:6835/2803298626] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.h{2:24440} state up:active seq 14 addr [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e{2:24422} state up:standby-replay seq 1 addr [v2:172.21.15.173:6834/2662617129,v1:172.21.15.173:6835/2662617129] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:Standby daemons: 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.j{-1:14520} state up:standby seq 1 addr [v2:172.21.15.98:6834/682650272,v1:172.21.15.98:6835/682650272] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d{-1:14526} state up:standby seq 1 addr [v2:172.21.15.98:6836/3048794856,v1:172.21.15.98:6837/3048794856] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.g{-1:14577} state up:standby seq 1 addr [v2:172.21.15.98:6838/672802539,v1:172.21.15.98:6839/672802539] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a{-1:14589} state up:standby seq 1 addr [v2:172.21.15.98:6840/121413521,v1:172.21.15.98:6841/121413521] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.331 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.l{-1:24397} state up:standby seq 1 addr [v2:172.21.15.203:6832/2408762956,v1:172.21.15.203:6833/2408762956] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.332 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.k{-1:24404} state up:standby seq 1 addr [v2:172.21.15.173:6832/2248833890,v1:172.21.15.173:6833/2248833890] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 -- 172.21.15.98:0/3178369145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fbb840a3c40 msgr2=0x7fbb840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fbb840a3c40 0x7fbb840a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbb90004630 tx=0x7fbb9000b3c0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 -- 172.21.15.98:0/3178369145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9c102aa0 msgr2=0x7fbb9c0767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9c102aa0 0x7fbb9c0767e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009040 tx=0x7fbb8c004d50 comp rx=0 tx=0).stop 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 -- 172.21.15.98:0/3178369145 shutdown_connections 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fbb840a3c40 0x7fbb840a6100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb9c104640 0x7fbb9c075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb9c102aa0 0x7fbb9c0767e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 --2- 172.21.15.98:0/3178369145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb9c103ca0 0x7fbb9c074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 -- 172.21.15.98:0/3178369145 >> 172.21.15.98:0/3178369145 conn(0x7fbb9c0fe040 msgr2=0x7fbb9c107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 -- 172.21.15.98:0/3178369145 shutdown_connections 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:25.329+0000 7fbba1ce8700 1 -- 172.21.15.98:0/3178369145 wait complete. 2024-04-25T17:35:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 25 2024-04-25T17:35:26.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79754014-0328-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T17:35:26.431 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:26.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:26 smithi098 ceph-mon[29192]: pgmap v389: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:26.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:26 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3178369145' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T17:35:26.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:26 smithi203 ceph-mon[32834]: pgmap v389: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:26.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:26 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3178369145' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T17:35:26.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:26 smithi173 ceph-mon[32856]: pgmap v389: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:26.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:26 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3178369145' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T17:35:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.248+0000 7feb545ef700 1 -- 172.21.15.98:0/3880017691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb4c102ac0 msgr2=0x7feb4c102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.248+0000 7feb545ef700 1 --2- 172.21.15.98:0/3880017691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb4c102ac0 0x7feb4c102ee0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7feb3c0099c0 tx=0x7feb3c009cd0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.248+0000 7feb545ef700 1 -- 172.21.15.98:0/3880017691 shutdown_connections 2024-04-25T17:35:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.248+0000 7feb545ef700 1 --2- 172.21.15.98:0/3880017691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb4c104660 0x7feb4c10af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.248+0000 7feb545ef700 1 --2- 172.21.15.98:0/3880017691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb4c102ac0 0x7feb4c102ee0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.248+0000 7feb545ef700 1 --2- 172.21.15.98:0/3880017691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb4c103cc0 0x7feb4c104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.248+0000 7feb545ef700 1 -- 172.21.15.98:0/3880017691 >> 172.21.15.98:0/3880017691 conn(0x7feb4c0fe060 msgr2=0x7feb4c1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:28.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.248+0000 7feb545ef700 1 -- 172.21.15.98:0/3880017691 shutdown_connections 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.248+0000 7feb545ef700 1 -- 172.21.15.98:0/3880017691 wait complete. 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb545ef700 1 Processor -- start 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb545ef700 1 -- start start 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb545ef700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb4c102ac0 0x7feb4c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb545ef700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb4c103cc0 0x7feb4c199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb545ef700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb4c104660 0x7feb4c1a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb545ef700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c10e530 con 0x7feb4c103cc0 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb545ef700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c10e3b0 con 0x7feb4c104660 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb545ef700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c10e6b0 con 0x7feb4c102ac0 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb51b8a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb4c103cc0 0x7feb4c199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb51b8a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb4c103cc0 0x7feb4c199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32786/0 (socket says 172.21.15.98:32786) 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb51b8a700 1 -- 172.21.15.98:0/1320794731 learned_addr learned my addr 172.21.15.98:0/1320794731 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:28.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb52b8c700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb4c104660 0x7feb4c1a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb51b8a700 1 -- 172.21.15.98:0/1320794731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb4c102ac0 msgr2=0x7feb4c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb51b8a700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb4c102ac0 0x7feb4c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb51b8a700 1 -- 172.21.15.98:0/1320794731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb4c104660 msgr2=0x7feb4c1a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb51b8a700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb4c104660 0x7feb4c1a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.249+0000 7feb51b8a700 1 -- 172.21.15.98:0/1320794731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c1a08a0 con 0x7feb4c103cc0 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.250+0000 7feb51b8a700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb4c103cc0 0x7feb4c199190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feb4800d950 tx=0x7feb4800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.250+0000 7feb437fe700 1 -- 172.21.15.98:0/1320794731 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb48016070 con 0x7feb4c103cc0 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.250+0000 7feb437fe700 1 -- 172.21.15.98:0/1320794731 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb48010460 con 0x7feb4c103cc0 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.250+0000 7feb437fe700 1 -- 172.21.15.98:0/1320794731 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb4800f620 con 0x7feb4c103cc0 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.250+0000 7feb545ef700 1 -- 172.21.15.98:0/1320794731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4c1a0b90 con 0x7feb4c103cc0 2024-04-25T17:35:28.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.250+0000 7feb545ef700 1 -- 172.21.15.98:0/1320794731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4c1a1120 con 0x7feb4c103cc0 2024-04-25T17:35:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.251+0000 7feb545ef700 1 -- 172.21.15.98:0/1320794731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb4c069730 con 0x7feb4c103cc0 2024-04-25T17:35:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.252+0000 7feb437fe700 1 -- 172.21.15.98:0/1320794731 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7feb48019070 con 0x7feb4c103cc0 2024-04-25T17:35:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.252+0000 7feb437fe700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feb380a3cc0 0x7feb380a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.252+0000 7feb5238b700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feb380a3cc0 0x7feb380a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.253+0000 7feb437fe700 1 -- 172.21.15.98:0/1320794731 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7feb480cffb0 con 0x7feb4c103cc0 2024-04-25T17:35:28.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.253+0000 7feb5238b700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feb380a3cc0 0x7feb380a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feb3c003790 tx=0x7feb3c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.257+0000 7feb437fe700 1 -- 172.21.15.98:0/1320794731 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb4809c9a0 con 0x7feb4c103cc0 2024-04-25T17:35:28.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:28 smithi098 ceph-mon[29192]: pgmap v390: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.457+0000 7feb545ef700 1 -- 172.21.15.98:0/1320794731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7feb4c063650 con 0x7feb4c103cc0 2024-04-25T17:35:28.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.457+0000 7feb437fe700 1 -- 172.21.15.98:0/1320794731 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) ==== 56+0+5293 (secure 0 0 0) 0x7feb4809c2a0 con 0x7feb4c103cc0 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 91 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-04-25T17:24:14.957466+0000 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-04-25T17:33:10.182926+0000 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:crush_version 26 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:full_ratio 0.95 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:backfillfull_ratio 0.9 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:nearfull_ratio 0.85 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:require_min_compat_client luminous 2024-04-25T17:35:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:min_compat_client luminous 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:require_osd_release squid 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:stretch_mode_enabled false 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 89 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:max_osd 12 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 up in weight 1 up_from 8 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6802/4172359021,v1:172.21.15.98:6803/4172359021] [v2:172.21.15.98:6804/4172359021,v1:172.21.15.98:6805/4172359021] exists,up 91254e6f-c19b-4a74-8b41-718ef0f5eb4d 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/3686689866,v1:172.21.15.173:6801/3686689866] [v2:172.21.15.173:6802/3686689866,v1:172.21.15.173:6803/3686689866] exists,up 1450194d-3e02-4d6a-ac19-364cb679d83b 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6800/2010273295,v1:172.21.15.203:6801/2010273295] [v2:172.21.15.203:6802/2010273295,v1:172.21.15.203:6803/2010273295] exists,up bf18400e-90d7-46f8-af4a-1abca0ed8066 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 up in weight 1 up_from 26 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6810/2541442286,v1:172.21.15.98:6811/2541442286] [v2:172.21.15.98:6812/2541442286,v1:172.21.15.98:6813/2541442286] exists,up e2b72e19-266c-42bf-9cce-045bba17c014 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 up in weight 1 up_from 34 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/2870689522,v1:172.21.15.173:6809/2870689522] [v2:172.21.15.173:6810/2870689522,v1:172.21.15.173:6811/2870689522] exists,up 2773c8f9-a178-40dc-9da0-51a8ac5d163b 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 up in weight 1 up_from 39 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6808/2904529843,v1:172.21.15.203:6809/2904529843] [v2:172.21.15.203:6810/2904529843,v1:172.21.15.203:6811/2904529843] exists,up 21751edb-9f9a-4bce-a13a-2e728141a500 2024-04-25T17:35:28.460 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 up in weight 1 up_from 47 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6818/3355912265,v1:172.21.15.98:6819/3355912265] [v2:172.21.15.98:6820/3355912265,v1:172.21.15.98:6821/3355912265] exists,up 73964ed9-657d-4097-b943-83b96cc1d04e 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 up in weight 1 up_from 52 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/3192706308,v1:172.21.15.173:6817/3192706308] [v2:172.21.15.173:6818/3192706308,v1:172.21.15.173:6819/3192706308] exists,up 6e248182-1de5-462a-a6ab-ec33e5ef7e7c 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 up in weight 1 up_from 60 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6816/2633569866,v1:172.21.15.203:6817/2633569866] [v2:172.21.15.203:6818/2633569866,v1:172.21.15.203:6819/2633569866] exists,up c486a634-e021-4c49-974f-56246017c27a 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 up in weight 1 up_from 68 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6826/3104319246,v1:172.21.15.98:6827/3104319246] [v2:172.21.15.98:6828/3104319246,v1:172.21.15.98:6829/3104319246] exists,up 650eada2-90dc-441b-a6e3-05fde6577647 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 up in weight 1 up_from 73 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/3555470519,v1:172.21.15.173:6825/3555470519] [v2:172.21.15.173:6826/3555470519,v1:172.21.15.173:6827/3555470519] exists,up cc1b690a-95fa-4a33-a0d4-185446d78a09 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 up in weight 1 up_from 81 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6824/2267615274,v1:172.21.15.203:6825/2267615274] [v2:172.21.15.203:6826/2267615274,v1:172.21.15.203:6827/2267615274] exists,up 44f36bfb-b366-4c58-a814-c4ddda0f32f5 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.0 [8,1] 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.a [2,11] 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.b [8,11] 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.e [2,4] 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.1 [4,9] 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.10 [5,9] 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.21 [4,8] 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.24 [4,9] 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/3170662380 expires 2024-04-26T17:25:08.883124+0000 2024-04-25T17:35:28.461 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/3170662380 expires 2024-04-26T17:25:08.883124+0000 2024-04-25T17:35:28.462 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/1795364490 expires 2024-04-26T17:25:08.883124+0000 2024-04-25T17:35:28.462 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/773164127 expires 2024-04-26T17:25:08.883124+0000 2024-04-25T17:35:28.462 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3412860846 expires 2024-04-26T17:24:41.622373+0000 2024-04-25T17:35:28.462 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/3033733589 expires 2024-04-26T17:24:41.622373+0000 2024-04-25T17:35:28.462 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3143366689 expires 2024-04-26T17:24:41.622373+0000 2024-04-25T17:35:28.462 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3075118794 expires 2024-04-26T17:24:41.622373+0000 2024-04-25T17:35:28.462 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/3033733589 expires 2024-04-26T17:24:41.622373+0000 2024-04-25T17:35:28.462 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/672531975 expires 2024-04-26T17:25:08.883124+0000 2024-04-25T17:35:28.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 -- 172.21.15.98:0/1320794731 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feb380a3cc0 msgr2=0x7feb380a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:28.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feb380a3cc0 0x7feb380a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feb3c003790 tx=0x7feb3c000bc0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 -- 172.21.15.98:0/1320794731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb4c103cc0 msgr2=0x7feb4c199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:28.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb4c103cc0 0x7feb4c199190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feb4800d950 tx=0x7feb4800dd10 comp rx=0 tx=0).stop 2024-04-25T17:35:28.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 -- 172.21.15.98:0/1320794731 shutdown_connections 2024-04-25T17:35:28.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feb380a3cc0 0x7feb380a6180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb4c102ac0 0x7feb4c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb4c104660 0x7feb4c1a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 --2- 172.21.15.98:0/1320794731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb4c103cc0 0x7feb4c199190 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:28.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 -- 172.21.15.98:0/1320794731 >> 172.21.15.98:0/1320794731 conn(0x7feb4c0fe060 msgr2=0x7feb4c1070e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:28.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 -- 172.21.15.98:0/1320794731 shutdown_connections 2024-04-25T17:35:28.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:28.460+0000 7feb545ef700 1 -- 172.21.15.98:0/1320794731 wait complete. 2024-04-25T17:35:28.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:28 smithi203 ceph-mon[32834]: pgmap v390: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:28.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:28 smithi173 ceph-mon[32856]: pgmap v390: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:29.124 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T17:35:29.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:35:29.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:29 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1320794731' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T17:35:29.377 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:29.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:29 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1320794731' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T17:35:29.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:29 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1320794731' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T17:35:30.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:30 smithi098 ceph-mon[29192]: pgmap v391: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:30.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:30 smithi203 ceph-mon[32834]: pgmap v391: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:30.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:30 smithi173 ceph-mon[32856]: pgmap v391: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T17:35:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.163+0000 7fe937739700 1 -- 172.21.15.98:0/3117823253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe930102300 msgr2=0x7fe93010e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.163+0000 7fe937739700 1 --2- 172.21.15.98:0/3117823253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe930102300 0x7fe93010e7f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe928009990 tx=0x7fe928009ca0 comp rx=0 tx=0).stop 2024-04-25T17:35:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.163+0000 7fe937739700 1 -- 172.21.15.98:0/3117823253 shutdown_connections 2024-04-25T17:35:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.163+0000 7fe937739700 1 --2- 172.21.15.98:0/3117823253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe930107990 0x7fe930107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.163+0000 7fe937739700 1 --2- 172.21.15.98:0/3117823253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe930102300 0x7fe93010e7f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.163+0000 7fe937739700 1 --2- 172.21.15.98:0/3117823253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe930101940 0x7fe930101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.163+0000 7fe937739700 1 -- 172.21.15.98:0/3117823253 >> 172.21.15.98:0/3117823253 conn(0x7fe9300788f0 msgr2=0x7fe930078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.163+0000 7fe937739700 1 -- 172.21.15.98:0/3117823253 shutdown_connections 2024-04-25T17:35:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.163+0000 7fe937739700 1 -- 172.21.15.98:0/3117823253 wait complete. 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe937739700 1 Processor -- start 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe937739700 1 -- start start 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe937739700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe930101940 0x7fe930198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe937739700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe930102300 0x7fe930199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe937739700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe930107990 0x7fe93019d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe937739700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe930111ae0 con 0x7fe930101940 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe937739700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe930111960 con 0x7fe930107990 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe937739700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe930111c60 con 0x7fe930102300 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe9354d5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe930101940 0x7fe930198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe9354d5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe930101940 0x7fe930198fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32814/0 (socket says 172.21.15.98:32814) 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe9354d5700 1 -- 172.21.15.98:0/732279785 learned_addr learned my addr 172.21.15.98:0/732279785 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe934cd4700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe930102300 0x7fe930199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.164+0000 7fe935cd6700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe930107990 0x7fe93019d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe9354d5700 1 -- 172.21.15.98:0/732279785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe930102300 msgr2=0x7fe930199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe9354d5700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe930102300 0x7fe930199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe9354d5700 1 -- 172.21.15.98:0/732279785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe930107990 msgr2=0x7fe93019d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe9354d5700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe930107990 0x7fe93019d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe9354d5700 1 -- 172.21.15.98:0/732279785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93019dff0 con 0x7fe930101940 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe9354d5700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe930101940 0x7fe930198fe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe920007e20 tx=0x7fe92000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe92e7fc700 1 -- 172.21.15.98:0/732279785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe920016020 con 0x7fe930101940 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe937739700 1 -- 172.21.15.98:0/732279785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe93019e2e0 con 0x7fe930101940 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe92e7fc700 1 -- 172.21.15.98:0/732279785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe92000ac90 con 0x7fe930101940 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe937739700 1 -- 172.21.15.98:0/732279785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9301a5b60 con 0x7fe930101940 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.165+0000 7fe92e7fc700 1 -- 172.21.15.98:0/732279785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9200115e0 con 0x7fe930101940 2024-04-25T17:35:31.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.166+0000 7fe937739700 1 -- 172.21.15.98:0/732279785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe930069730 con 0x7fe930101940 2024-04-25T17:35:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.167+0000 7fe92e7fc700 1 -- 172.21.15.98:0/732279785 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe920019020 con 0x7fe930101940 2024-04-25T17:35:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.168+0000 7fe92e7fc700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe9180a3c40 0x7fe9180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.168+0000 7fe934cd4700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe9180a3c40 0x7fe9180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.168+0000 7fe92e7fc700 1 -- 172.21.15.98:0/732279785 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fe9200cfd80 con 0x7fe930101940 2024-04-25T17:35:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.168+0000 7fe934cd4700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe9180a3c40 0x7fe9180a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe930106c00 tx=0x7fe92400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:31.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.172+0000 7fe92e7fc700 1 -- 172.21.15.98:0/732279785 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe920020070 con 0x7fe930101940 2024-04-25T17:35:31.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.416+0000 7fe937739700 1 -- 172.21.15.98:0/732279785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe93019a340 con 0x7fe930101940 2024-04-25T17:35:31.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.416+0000 7fe92e7fc700 1 -- 172.21.15.98:0/732279785 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11877 (secure 0 0 0) 0x7fe92009c770 con 0x7fe930101940 2024-04-25T17:35:31.417 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:35:31.417 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:32:59.314849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:35:31.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.418+0000 7fe937739700 1 -- 172.21.15.98:0/732279785 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe9180a3c40 msgr2=0x7fe9180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:31.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.418+0000 7fe937739700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe9180a3c40 0x7fe9180a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe930106c00 tx=0x7fe92400b410 comp rx=0 tx=0).stop 2024-04-25T17:35:31.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.418+0000 7fe937739700 1 -- 172.21.15.98:0/732279785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe930101940 msgr2=0x7fe930198fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:31.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.418+0000 7fe937739700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe930101940 0x7fe930198fe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe920007e20 tx=0x7fe92000a340 comp rx=0 tx=0).stop 2024-04-25T17:35:31.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.419+0000 7fe937739700 1 -- 172.21.15.98:0/732279785 shutdown_connections 2024-04-25T17:35:31.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.419+0000 7fe937739700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fe9180a3c40 0x7fe9180a6100 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:31.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.419+0000 7fe937739700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe930107990 0x7fe93019d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:31.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.419+0000 7fe937739700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe930101940 0x7fe930198fe0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:31.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.419+0000 7fe937739700 1 --2- 172.21.15.98:0/732279785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe930102300 0x7fe930199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:31.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.419+0000 7fe937739700 1 -- 172.21.15.98:0/732279785 >> 172.21.15.98:0/732279785 conn(0x7fe9300788f0 msgr2=0x7fe930105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:31.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.419+0000 7fe937739700 1 -- 172.21.15.98:0/732279785 shutdown_connections 2024-04-25T17:35:31.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:31.419+0000 7fe937739700 1 -- 172.21.15.98:0/732279785 wait complete. 2024-04-25T17:35:31.420 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 25 2024-04-25T17:35:32.059 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:35:32.298 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:32.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:32 smithi098 ceph-mon[29192]: pgmap v392: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:32.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:32 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/732279785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:35:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:32 smithi203 ceph-mon[32834]: pgmap v392: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:32.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:32 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/732279785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:35:32.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:32 smithi173 ceph-mon[32856]: pgmap v392: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T17:35:32.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:32 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/732279785' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:35:34.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.110+0000 7f2a183fa700 1 -- 172.21.15.98:0/1523749256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a10106930 msgr2=0x7f2a10106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:34.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.110+0000 7f2a183fa700 1 --2- 172.21.15.98:0/1523749256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a10106930 0x7f2a10106d10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c009990 tx=0x7f2a0c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:35:34.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 -- 172.21.15.98:0/1523749256 shutdown_connections 2024-04-25T17:35:34.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 --2- 172.21.15.98:0/1523749256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a10102650 0x7f2a1010e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:34.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 --2- 172.21.15.98:0/1523749256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a10106930 0x7f2a10106d10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:34.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 --2- 172.21.15.98:0/1523749256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a10101c90 0x7f2a10102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:34.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 -- 172.21.15.98:0/1523749256 >> 172.21.15.98:0/1523749256 conn(0x7f2a100777b0 msgr2=0x7f2a10077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:34.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 -- 172.21.15.98:0/1523749256 shutdown_connections 2024-04-25T17:35:34.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 -- 172.21.15.98:0/1523749256 wait complete. 2024-04-25T17:35:34.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 Processor -- start 2024-04-25T17:35:34.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 -- start start 2024-04-25T17:35:34.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a10101c90 0x7f2a10199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:34.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a10102650 0x7f2a1019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a10106930 0x7f2a101a5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2a10111b80 con 0x7f2a10106930 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2a10111a00 con 0x7f2a10102650 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.111+0000 7f2a183fa700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a10111d00 con 0x7f2a10101c90 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a16196700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a10101c90 0x7f2a10199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a16997700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a10106930 0x7f2a101a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a16196700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a10101c90 0x7f2a10199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:51690/0 (socket says 172.21.15.98:51690) 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a16196700 1 -- 172.21.15.98:0/456691661 learned_addr learned my addr 172.21.15.98:0/456691661 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a15995700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a10102650 0x7f2a1019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a16997700 1 -- 172.21.15.98:0/456691661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a10101c90 msgr2=0x7f2a10199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:34.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a16997700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a10101c90 0x7f2a10199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a16997700 1 -- 172.21.15.98:0/456691661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a10102650 msgr2=0x7f2a1019a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a16997700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a10102650 0x7f2a1019a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a16997700 1 -- 172.21.15.98:0/456691661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a101a5560 con 0x7f2a10106930 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a15995700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a10102650 0x7f2a1019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a16997700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a10106930 0x7f2a101a5020 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800ab40 tx=0x7f2a0800ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a077fe700 1 -- 172.21.15.98:0/456691661 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08017020 con 0x7f2a10106930 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a077fe700 1 -- 172.21.15.98:0/456691661 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a08004bb0 con 0x7f2a10106930 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a077fe700 1 -- 172.21.15.98:0/456691661 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08005280 con 0x7f2a10106930 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a183fa700 1 -- 172.21.15.98:0/456691661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a101a5790 con 0x7f2a10106930 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.112+0000 7f2a183fa700 1 -- 172.21.15.98:0/456691661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a101a5c50 con 0x7f2a10106930 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.113+0000 7f2a183fa700 1 -- 172.21.15.98:0/456691661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a10109f00 con 0x7f2a10106930 2024-04-25T17:35:34.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.114+0000 7f2a077fe700 1 -- 172.21.15.98:0/456691661 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2a0800c640 con 0x7f2a10106930 2024-04-25T17:35:34.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.115+0000 7f2a077fe700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f29fc0a3c40 0x7f29fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:34.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.115+0000 7f2a16196700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f29fc0a3c40 0x7f29fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:34.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.115+0000 7f2a077fe700 1 -- 172.21.15.98:0/456691661 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f2a080cf990 con 0x7f2a10106930 2024-04-25T17:35:34.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.115+0000 7f2a16196700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f29fc0a3c40 0x7f29fc0a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c003800 tx=0x7f2a0c009630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:34.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.119+0000 7f2a077fe700 1 -- 172.21.15.98:0/456691661 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2a08021020 con 0x7f2a10106930 2024-04-25T17:35:34.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.360+0000 7f2a183fa700 1 -- 172.21.15.98:0/456691661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a101030b0 con 0x7f2a10106930 2024-04-25T17:35:34.361 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:34 smithi098 ceph-mon[29192]: pgmap v393: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T17:35:34.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.361+0000 7f2a077fe700 1 -- 172.21.15.98:0/456691661 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11877 (secure 0 0 0) 0x7f2a0809c410 con 0x7f2a10106930 2024-04-25T17:35:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:35:34.362 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:32:59.314849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:35:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 -- 172.21.15.98:0/456691661 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f29fc0a3c40 msgr2=0x7f29fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f29fc0a3c40 0x7f29fc0a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c003800 tx=0x7f2a0c009630 comp rx=0 tx=0).stop 2024-04-25T17:35:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 -- 172.21.15.98:0/456691661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a10106930 msgr2=0x7f2a101a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a10106930 0x7f2a101a5020 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800ab40 tx=0x7f2a0800ae50 comp rx=0 tx=0).stop 2024-04-25T17:35:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 -- 172.21.15.98:0/456691661 shutdown_connections 2024-04-25T17:35:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f29fc0a3c40 0x7f29fc0a6100 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a10101c90 0x7f2a10199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2a10106930 0x7f2a101a5020 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:34.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 --2- 172.21.15.98:0/456691661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2a10102650 0x7f2a1019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:34.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 -- 172.21.15.98:0/456691661 >> 172.21.15.98:0/456691661 conn(0x7f2a100777b0 msgr2=0x7f2a100ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:34.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 -- 172.21.15.98:0/456691661 shutdown_connections 2024-04-25T17:35:34.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:34.364+0000 7f2a183fa700 1 -- 172.21.15.98:0/456691661 wait complete. 2024-04-25T17:35:34.366 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 25 2024-04-25T17:35:34.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:34 smithi203 ceph-mon[32834]: pgmap v393: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T17:35:34.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:34 smithi173 ceph-mon[32856]: pgmap v393: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T17:35:35.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T17:35:35.413 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:35.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:35 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/456691661' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:35:35.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:35 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/456691661' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:35:35.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:35 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/456691661' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:35:36.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:36 smithi098 ceph-mon[29192]: pgmap v394: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:35:36.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:36 smithi203 ceph-mon[32834]: pgmap v394: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:35:36.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:36 smithi173 ceph-mon[32856]: pgmap v394: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-25T17:35:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.256+0000 7f9a94339700 1 -- 172.21.15.98:0/401968548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a8c06b170 msgr2=0x7f9a8c06b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.256+0000 7f9a94339700 1 --2- 172.21.15.98:0/401968548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a8c06b170 0x7f9a8c06b5d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009990 tx=0x7f9a7c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.256+0000 7f9a94339700 1 -- 172.21.15.98:0/401968548 shutdown_connections 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.256+0000 7f9a94339700 1 --2- 172.21.15.98:0/401968548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a8c06bb10 0x7f9a8c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.256+0000 7f9a94339700 1 --2- 172.21.15.98:0/401968548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a8c108a30 0x7f9a8c108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.256+0000 7f9a94339700 1 --2- 172.21.15.98:0/401968548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a8c06b170 0x7f9a8c06b5d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.256+0000 7f9a94339700 1 -- 172.21.15.98:0/401968548 >> 172.21.15.98:0/401968548 conn(0x7f9a8c0788f0 msgr2=0x7f9a8c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.256+0000 7f9a94339700 1 -- 172.21.15.98:0/401968548 shutdown_connections 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.256+0000 7f9a94339700 1 -- 172.21.15.98:0/401968548 wait complete. 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a94339700 1 Processor -- start 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a94339700 1 -- start start 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a94339700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a8c06b170 0x7f9a8c101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a94339700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a8c06bb10 0x7f9a8c101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a94339700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a8c108a30 0x7f9a8c1024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a94339700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c112cb0 con 0x7f9a8c06b170 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a94339700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c112b30 con 0x7f9a8c06bb10 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a94339700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c112e30 con 0x7f9a8c108a30 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a920d5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a8c06b170 0x7f9a8c101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a920d5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a8c06b170 0x7f9a8c101a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55190/0 (socket says 172.21.15.98:55190) 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a920d5700 1 -- 172.21.15.98:0/4260825162 learned_addr learned my addr 172.21.15.98:0/4260825162 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a928d6700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a8c108a30 0x7f9a8c1024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a918d4700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a8c06bb10 0x7f9a8c101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a920d5700 1 -- 172.21.15.98:0/4260825162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a8c108a30 msgr2=0x7f9a8c1024c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a920d5700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a8c108a30 0x7f9a8c1024c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a920d5700 1 -- 172.21.15.98:0/4260825162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a8c06bb10 msgr2=0x7f9a8c101f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a920d5700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a8c06bb10 0x7f9a8c101f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a920d5700 1 -- 172.21.15.98:0/4260825162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c1a7800 con 0x7f9a8c06b170 2024-04-25T17:35:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.257+0000 7f9a920d5700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a8c06b170 0x7f9a8c101a40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800b530 tx=0x7f9a8800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.258+0000 7f9a837fe700 1 -- 172.21.15.98:0/4260825162 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88015020 con 0x7f9a8c06b170 2024-04-25T17:35:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.258+0000 7f9a94339700 1 -- 172.21.15.98:0/4260825162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8c1a7af0 con 0x7f9a8c06b170 2024-04-25T17:35:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.258+0000 7f9a837fe700 1 -- 172.21.15.98:0/4260825162 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a88011ce0 con 0x7f9a8c06b170 2024-04-25T17:35:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.258+0000 7f9a837fe700 1 -- 172.21.15.98:0/4260825162 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88010d20 con 0x7f9a8c06b170 2024-04-25T17:35:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.258+0000 7f9a94339700 1 -- 172.21.15.98:0/4260825162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8c1a8080 con 0x7f9a8c06b170 2024-04-25T17:35:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.259+0000 7f9a94339700 1 -- 172.21.15.98:0/4260825162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a8c069730 con 0x7f9a8c06b170 2024-04-25T17:35:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.260+0000 7f9a837fe700 1 -- 172.21.15.98:0/4260825162 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9a88018020 con 0x7f9a8c06b170 2024-04-25T17:35:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.260+0000 7f9a837fe700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9a780a3c40 0x7f9a780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.260+0000 7f9a918d4700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9a780a3c40 0x7f9a780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.260+0000 7f9a837fe700 1 -- 172.21.15.98:0/4260825162 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f9a880d01a0 con 0x7f9a8c06b170 2024-04-25T17:35:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.261+0000 7f9a918d4700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9a780a3c40 0x7f9a780a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c006010 tx=0x7f9a7c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:37.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.264+0000 7f9a837fe700 1 -- 172.21.15.98:0/4260825162 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a8801f020 con 0x7f9a8c06b170 2024-04-25T17:35:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.461+0000 7f9a94339700 1 -- 172.21.15.98:0/4260825162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a8c02df80 con 0x7f9a8c06b170 2024-04-25T17:35:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.463+0000 7f9a837fe700 1 -- 172.21.15.98:0/4260825162 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+22686 (secure 0 0 0) 0x7f9a8809cb90 con 0x7f9a8c06b170 2024-04-25T17:35:37.464 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:35:37.464 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":91,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:33:10.182926+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T17:32:30.267079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T17:32:33.956902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 -- 172.21.15.98:0/4260825162 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9a780a3c40 msgr2=0x7f9a780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9a780a3c40 0x7f9a780a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c006010 tx=0x7f9a7c01a040 comp rx=0 tx=0).stop 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 -- 172.21.15.98:0/4260825162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a8c06b170 msgr2=0x7f9a8c101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a8c06b170 0x7f9a8c101a40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800b530 tx=0x7f9a8800b8f0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 -- 172.21.15.98:0/4260825162 shutdown_connections 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f9a780a3c40 0x7f9a780a6100 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a8c108a30 0x7f9a8c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a8c06b170 0x7f9a8c101a40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 --2- 172.21.15.98:0/4260825162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a8c06bb10 0x7f9a8c101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 -- 172.21.15.98:0/4260825162 >> 172.21.15.98:0/4260825162 conn(0x7f9a8c0788f0 msgr2=0x7f9a8c0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 -- 172.21.15.98:0/4260825162 shutdown_connections 2024-04-25T17:35:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:37.465+0000 7f9a94339700 1 -- 172.21.15.98:0/4260825162 wait complete. 2024-04-25T17:35:38.142 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T17:32:37.173463+0000', 'modified': '2024-04-25T17:32:59.314849+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24461, 'mds_1': 24445, 'mds_2': 24440}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24422': {'gid': 24422, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/2662617129', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 2662617129}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 2662617129}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/2803298626', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 2803298626}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 2803298626}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6837/1057173981', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 1057173981}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 1057173981}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.173:6837/1096095017', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 1096095017}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 1096095017}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.203:6839/1340539385', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 1340539385}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 1340539385}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.173:6839/2342732992', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 2342732992}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 2342732992}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24461, 'qdb_cluster': [24445, 24440, 24461]} 2024-04-25T17:35:38.142 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T17:35:38.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell mds.h status 2024-04-25T17:35:38.384 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:38.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:38 smithi098 ceph-mon[29192]: pgmap v395: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:38 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4260825162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:35:38.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:38 smithi203 ceph-mon[32834]: pgmap v395: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:38.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:38 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/4260825162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:35:38.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:38 smithi173 ceph-mon[32856]: pgmap v395: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:38.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:38 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/4260825162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:35:40.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.034+0000 7fa660178700 1 -- 172.21.15.98:0/243169834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa658102300 msgr2=0x7fa65810e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:40.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.034+0000 7fa660178700 1 --2- 172.21.15.98:0/243169834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa658102300 0x7fa65810e7f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa65000b0a0 tx=0x7fa65000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.034+0000 7fa660178700 1 -- 172.21.15.98:0/243169834 shutdown_connections 2024-04-25T17:35:40.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.034+0000 7fa660178700 1 --2- 172.21.15.98:0/243169834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa658102300 0x7fa65810e7f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.034+0000 7fa660178700 1 --2- 172.21.15.98:0/243169834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa658107990 0x7fa658107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.034+0000 7fa660178700 1 --2- 172.21.15.98:0/243169834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa658101940 0x7fa658101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.034+0000 7fa660178700 1 -- 172.21.15.98:0/243169834 >> 172.21.15.98:0/243169834 conn(0x7fa6580788f0 msgr2=0x7fa658078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.034+0000 7fa660178700 1 -- 172.21.15.98:0/243169834 shutdown_connections 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.034+0000 7fa660178700 1 -- 172.21.15.98:0/243169834 wait complete. 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa660178700 1 Processor -- start 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa660178700 1 -- start start 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa660178700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa658101940 0x7fa6581990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa660178700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa658102300 0x7fa658199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa660178700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa658107990 0x7fa65819d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa660178700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa658111b50 con 0x7fa658102300 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa660178700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581119d0 con 0x7fa658101940 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa660178700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa658111cd0 con 0x7fa658107990 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65d713700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa658102300 0x7fa658199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65d713700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa658102300 0x7fa658199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55206/0 (socket says 172.21.15.98:55206) 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65d713700 1 -- 172.21.15.98:0/761247318 learned_addr learned my addr 172.21.15.98:0/761247318 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65e715700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa658107990 0x7fa65819d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65df14700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa658101940 0x7fa6581990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65d713700 1 -- 172.21.15.98:0/761247318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa658107990 msgr2=0x7fa65819d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65d713700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa658107990 0x7fa65819d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65d713700 1 -- 172.21.15.98:0/761247318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa658101940 msgr2=0x7fa6581990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65d713700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa658101940 0x7fa6581990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65d713700 1 -- 172.21.15.98:0/761247318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65819e0e0 con 0x7fa658102300 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.035+0000 7fa65e715700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa658107990 0x7fa65819d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.036+0000 7fa65d713700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa658102300 0x7fa658199610 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa64800b530 tx=0x7fa64800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.036+0000 7fa64effd700 1 -- 172.21.15.98:0/761247318 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648015020 con 0x7fa658102300 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.036+0000 7fa660178700 1 -- 172.21.15.98:0/761247318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65819e3d0 con 0x7fa658102300 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.036+0000 7fa64effd700 1 -- 172.21.15.98:0/761247318 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa648011ce0 con 0x7fa658102300 2024-04-25T17:35:40.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.036+0000 7fa64effd700 1 -- 172.21.15.98:0/761247318 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648010d20 con 0x7fa658102300 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.036+0000 7fa660178700 1 -- 172.21.15.98:0/761247318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6581a5c50 con 0x7fa658102300 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.038+0000 7fa64effd700 1 -- 172.21.15.98:0/761247318 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa648018020 con 0x7fa658102300 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.039+0000 7fa64effd700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa6440a3c40 0x7fa6440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.039+0000 7fa65df14700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa6440a3c40 0x7fa6440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.039+0000 7fa64effd700 1 -- 172.21.15.98:0/761247318 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fa6480cfc90 con 0x7fa658102300 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.039+0000 7fa65df14700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa6440a3c40 0x7fa6440a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa654005950 tx=0x7fa6540058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa66aa01b80 1 Processor -- start 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa66aa01b80 1 -- start start 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6ae0b0 0x55c38b6ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6b9880 0x55c38b6bbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa66aa01b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c38b601c40 con 0x55c38b6ae9d0 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa66aa01b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c38b5a88e0 con 0x55c38b6b9880 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa66aa01b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c38b5fdc00 con 0x55c38b6ae0b0 2024-04-25T17:35:40.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65d713700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65d713700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55208/0 (socket says 172.21.15.98:55208) 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65d713700 1 -- 172.21.15.98:0/367854255 learned_addr learned my addr 172.21.15.98:0/367854255 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65d713700 1 -- 172.21.15.98:0/367854255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6ae0b0 msgr2=0x55c38b6ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65e715700 1 --2- 172.21.15.98:0/367854255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6b9880 0x55c38b6bbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65df14700 1 --2- 172.21.15.98:0/367854255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6ae0b0 0x55c38b6ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65d713700 1 --2- 172.21.15.98:0/367854255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6ae0b0 0x55c38b6ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65d713700 1 -- 172.21.15.98:0/367854255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6b9880 msgr2=0x55c38b6bbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65d713700 1 --2- 172.21.15.98:0/367854255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6b9880 0x55c38b6bbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65d713700 1 -- 172.21.15.98:0/367854255 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c38b56c6f0 con 0x55c38b6ae9d0 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.040+0000 7fa65d713700 1 --2- 172.21.15.98:0/367854255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b9340 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa6480988f0 tx=0x7fa648004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c5285adca5a52a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.041+0000 7fa643fff700 1 -- 172.21.15.98:0/367854255 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64809bf20 con 0x55c38b6ae9d0 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.041+0000 7fa643fff700 1 -- 172.21.15.98:0/367854255 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6480a02d0 con 0x55c38b6ae9d0 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.041+0000 7fa643fff700 1 -- 172.21.15.98:0/367854255 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480b54a0 con 0x55c38b6ae9d0 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.041+0000 7fa66aa01b80 1 -- 172.21.15.98:0/367854255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 msgr2=0x55c38b6b9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:40.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.041+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/367854255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b9340 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa6480988f0 tx=0x7fa648004750 comp rx=0 tx=0).stop 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.041+0000 7fa66aa01b80 1 -- 172.21.15.98:0/367854255 shutdown_connections 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.041+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/367854255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6ae0b0 0x55c38b6ae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.041+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/367854255 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6b9880 0x55c38b6bbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.041+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/367854255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b9340 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.041+0000 7fa66aa01b80 1 -- 172.21.15.98:0/367854255 >> 172.21.15.98:0/367854255 conn(0x55c38b5db840 msgr2=0x55c38b6aa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.042+0000 7fa66aa01b80 1 -- 172.21.15.98:0/367854255 shutdown_connections 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.042+0000 7fa66aa01b80 1 -- 172.21.15.98:0/367854255 wait complete. 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.042+0000 7fa66aa01b80 1 -- start start 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.042+0000 7fa66aa01b80 1 Processor -- start 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.042+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.042+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.042+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 0x55c38b6b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.042+0000 7fa66aa01b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c38b6bc370 con 0x55c38b6ae9d0 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.042+0000 7fa66aa01b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c38b5bb560 con 0x55c38b6ae0b0 2024-04-25T17:35:40.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.042+0000 7fa66aa01b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c38b6bc4f0 con 0x55c38b6b9880 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa66aa01b80 10 client.0 _ll_register_callbacks cb 0x55c38b59a8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65d713700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65d713700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55214/0 (socket says 172.21.15.98:55214) 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65d713700 1 -- 172.21.15.98:0/722535868 learned_addr learned my addr 172.21.15.98:0/722535868 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65df14700 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65e715700 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 0x55c38b6b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65d713700 1 -- 172.21.15.98:0/722535868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 msgr2=0x55c38b6b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65d713700 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 0x55c38b6b6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65d713700 1 -- 172.21.15.98:0/722535868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae0b0 msgr2=0x55c38b6b8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65d713700 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65d713700 1 -- 172.21.15.98:0/722535868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c38b54f880 con 0x55c38b6ae9d0 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65e715700 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 0x55c38b6b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65df14700 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa65d713700 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b62c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa64800bcf0 tx=0x7fa64809bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f5e3595a6bee6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa6417fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:35:40.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa6417fa700 1 -- 172.21.15.98:0/722535868 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480b5450 con 0x55c38b6ae9d0 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa6417fa700 1 -- 172.21.15.98:0/722535868 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fa64809c180 con 0x55c38b6ae9d0 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa66aa01b80 1 -- 172.21.15.98:0/722535868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6480a02d0 con 0x55c38b6ae9d0 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa6417fa700 1 -- 172.21.15.98:0/722535868 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa64809c380 con 0x55c38b6ae9d0 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa66aa01b80 10 client.15063 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa66aa01b80 1 -- 172.21.15.98:0/722535868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c38b6bc370 con 0x55c38b6ae9d0 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.043+0000 7fa6417fa700 1 -- 172.21.15.98:0/722535868 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6480a02d0 con 0x55c38b6ae9d0 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.044+0000 7fa6417fa700 1 -- 172.21.15.98:0/722535868 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fa64809f030 con 0x55c38b6ae9d0 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.044+0000 7fa66aa01b80 10 client.15063 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.044+0000 7fa66aa01b80 10 client.15063 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.044+0000 7fa66aa01b80 20 client.15063 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.044+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55c38b6c8ce0 0x55c38b6ca150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.044+0000 7fa66aa01b80 4 client.15063 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.044+0000 7fa66aa01b80 1 -- 172.21.15.98:0/722535868 --> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c38b60f430 con 0x55c38b6c8ce0 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.044+0000 7fa65df14700 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55c38b6c8ce0 0x55c38b6ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.044+0000 7fa6417fa700 10 client.15063 ms_handle_connect on v2:172.21.15.173:6836/1096095017 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.044+0000 7fa65df14700 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55c38b6c8ce0 0x55c38b6ca150 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9e5b31e6f0e80fc0 server_cookie=2dca69d050a8fa0a in_seq=0 out_seq=0 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa6417fa700 1 -- 172.21.15.98:0/722535868 <== mds.2 v2:172.21.15.173:6836/1096095017 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c38b60f430 con 0x55c38b6c8ce0 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa6417fa700 10 client.15063 handle_command_reply: tid=0 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 client.15063 shutdown 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 -- 172.21.15.98:0/722535868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 msgr2=0x55c38b6b62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b62c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa64800bcf0 tx=0x7fa64809bf20 comp rx=0 tx=0).stop 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 -- 172.21.15.98:0/722535868 shutdown_connections 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55c38b6c8ce0 0x55c38b6ca150 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 0x55c38b6b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa6417fa700 0 client.15063 ms_handle_reset on v2:172.21.15.173:6836/1096095017 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/722535868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b62c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 -- 172.21.15.98:0/722535868 >> 172.21.15.98:0/722535868 conn(0x55c38b5db840 msgr2=0x55c38b6aa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:40.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 -- 172.21.15.98:0/722535868 shutdown_connections 2024-04-25T17:35:40.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.045+0000 7fa66aa01b80 1 -- 172.21.15.98:0/722535868 wait complete. 2024-04-25T17:35:40.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.046+0000 7fa66aa01b80 20 client.15063 trim_cache size 0 max 16384 2024-04-25T17:35:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.065+0000 7fa66aa01b80 1 Processor -- start 2024-04-25T17:35:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.065+0000 7fa66aa01b80 1 -- start start 2024-04-25T17:35:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6b9880 0x55c38b6d35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa66aa01b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c38b6c8490 con 0x55c38b6ae9d0 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa66aa01b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c38b6c8b60 con 0x55c38b6b9880 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa66aa01b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c38b6c9230 con 0x55c38b6ae0b0 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa65d713700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa65d713700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55230/0 (socket says 172.21.15.98:55230) 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa65d713700 1 -- 172.21.15.98:0/1280314935 learned_addr learned my addr 172.21.15.98:0/1280314935 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa65e715700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6b9880 0x55c38b6d35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa65df14700 1 --2- 172.21.15.98:0/1280314935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa65d713700 1 -- 172.21.15.98:0/1280314935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6ae0b0 msgr2=0x55c38b6b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa65d713700 1 --2- 172.21.15.98:0/1280314935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa65d713700 1 -- 172.21.15.98:0/1280314935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6b9880 msgr2=0x55c38b6d35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:40.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa65d713700 1 --2- 172.21.15.98:0/1280314935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6b9880 0x55c38b6d35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.066+0000 7fa65d713700 1 -- 172.21.15.98:0/1280314935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c38b6a92d0 con 0x55c38b6ae9d0 2024-04-25T17:35:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.067+0000 7fa65d713700 1 --2- 172.21.15.98:0/1280314935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b1800 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa6480a4d50 tx=0x7fa648004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1585785f858f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.067+0000 7fa6437fe700 1 -- 172.21.15.98:0/1280314935 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480b9070 con 0x55c38b6ae9d0 2024-04-25T17:35:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.067+0000 7fa6437fe700 1 -- 172.21.15.98:0/1280314935 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa648065070 con 0x55c38b6ae9d0 2024-04-25T17:35:40.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.067+0000 7fa6437fe700 1 -- 172.21.15.98:0/1280314935 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64809b370 con 0x55c38b6ae9d0 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1280314935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 msgr2=0x55c38b6b1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/1280314935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b1800 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa6480a4d50 tx=0x7fa648004750 comp rx=0 tx=0).stop 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1280314935 shutdown_connections 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/1280314935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/1280314935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6b9880 0x55c38b6d35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/1280314935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b1800 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1280314935 >> 172.21.15.98:0/1280314935 conn(0x55c38b5db840 msgr2=0x55c38b6aa060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1280314935 shutdown_connections 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1280314935 wait complete. 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 -- start start 2024-04-25T17:35:40.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.068+0000 7fa66aa01b80 1 Processor -- start 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa66aa01b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 0x55c38b6b6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa66aa01b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c38b6c9fd0 con 0x55c38b6ae0b0 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa65df14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa65df14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55236/0 (socket says 172.21.15.98:55236) 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa65df14700 1 -- 172.21.15.98:0/1513915969 learned_addr learned my addr 172.21.15.98:0/1513915969 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1513915969 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c38b6c9900 con 0x55c38b6ae9d0 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1513915969 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c38b6ca6a0 con 0x55c38b6b9880 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa65d713700 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa66aa01b80 10 client.0 _ll_register_callbacks cb 0x55c38b59a8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa65e715700 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 0x55c38b6b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa65df14700 1 -- 172.21.15.98:0/1513915969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 msgr2=0x55c38b6b6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:40.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa65df14700 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 0x55c38b6b6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa65df14700 1 -- 172.21.15.98:0/1513915969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae9d0 msgr2=0x55c38b6b5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa65df14700 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.069+0000 7fa65df14700 1 -- 172.21.15.98:0/1513915969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c38b6cf9c0 con 0x55c38b6ae0b0 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.070+0000 7fa65df14700 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b59b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55c38b6b7480 tx=0x7fa6540047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c228f3c314969704 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.070+0000 7fa6417fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.070+0000 7fa6417fa700 1 -- 172.21.15.98:0/1513915969 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654031670 con 0x55c38b6ae0b0 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.070+0000 7fa6417fa700 1 -- 172.21.15.98:0/1513915969 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fa654011440 con 0x55c38b6ae0b0 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.070+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1513915969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa648065070 con 0x55c38b6ae0b0 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.070+0000 7fa6417fa700 1 -- 172.21.15.98:0/1513915969 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa654031810 con 0x55c38b6ae0b0 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.070+0000 7fa66aa01b80 10 client.15069 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.070+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1513915969 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c38b734690 con 0x55c38b6ae0b0 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.071+0000 7fa6417fa700 1 -- 172.21.15.98:0/1513915969 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa648065070 con 0x55c38b6ae0b0 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.071+0000 7fa6417fa700 1 -- 172.21.15.98:0/1513915969 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fa654011c90 con 0x55c38b6ae0b0 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.071+0000 7fa66aa01b80 10 client.15069 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:40.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.071+0000 7fa66aa01b80 10 client.15069 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.071+0000 7fa66aa01b80 20 client.15069 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.071+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55c38b7392b0 0x55c38b737fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.071+0000 7fa66aa01b80 4 client.15069 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.071+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1513915969 --> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] -- command(tid 0: {"prefix": "status"}) -- 0x55c38b6094d0 con 0x55c38b7392b0 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.071+0000 7fa65d713700 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55c38b7392b0 0x55c38b737fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa65d713700 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55c38b7392b0 0x55c38b737fc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e862264d661fdef server_cookie=2708792a9e74711f in_seq=0 out_seq=0 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa6417fa700 10 client.15069 ms_handle_connect on v2:172.21.15.173:6836/1096095017 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa6417fa700 1 -- 172.21.15.98:0/1513915969 <== mds.2 v2:172.21.15.173:6836/1096095017 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55c38b6094d0 con 0x55c38b7392b0 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa6417fa700 10 client.15069 handle_command_reply: tid=0 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 client.15069 shutdown 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1513915969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae0b0 msgr2=0x55c38b6b59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b59b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55c38b6b7480 tx=0x7fa6540047c0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1513915969 shutdown_connections 2024-04-25T17:35:40.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x55c38b7392b0 0x55c38b737fc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c38b6b9880 0x55c38b6b6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c38b6ae0b0 0x55c38b6b59b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 --2- 172.21.15.98:0/1513915969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c38b6ae9d0 0x55c38b6b5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa6417fa700 0 client.15069 ms_handle_reset on v2:172.21.15.173:6836/1096095017 2024-04-25T17:35:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1513915969 >> 172.21.15.98:0/1513915969 conn(0x55c38b5db840 msgr2=0x55c38b6b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1513915969 shutdown_connections 2024-04-25T17:35:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 1 -- 172.21.15.98:0/1513915969 wait complete. 2024-04-25T17:35:40.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.072+0000 7fa66aa01b80 20 client.15069 trim_cache size 0 max 16384 2024-04-25T17:35:40.077 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:35:40.077 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24440, 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 166.94949940199999, 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 25, 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 215.18489574 2024-04-25T17:35:40.078 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:35:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 -- 172.21.15.98:0/761247318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa6440a3c40 msgr2=0x7fa6440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa6440a3c40 0x7fa6440a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa654005950 tx=0x7fa6540058e0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 -- 172.21.15.98:0/761247318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa658102300 msgr2=0x7fa658199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa658102300 0x7fa658199610 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa64800b530 tx=0x7fa64800b8f0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 -- 172.21.15.98:0/761247318 shutdown_connections 2024-04-25T17:35:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fa6440a3c40 0x7fa6440a6100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa658107990 0x7fa65819d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa658101940 0x7fa6581990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 --2- 172.21.15.98:0/761247318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa658102300 0x7fa658199610 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 -- 172.21.15.98:0/761247318 >> 172.21.15.98:0/761247318 conn(0x7fa6580788f0 msgr2=0x7fa65806aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 -- 172.21.15.98:0/761247318 shutdown_connections 2024-04-25T17:35:40.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:40.073+0000 7fa660178700 1 -- 172.21.15.98:0/761247318 wait complete. 2024-04-25T17:35:40.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:40 smithi098 ceph-mon[29192]: pgmap v396: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-25T17:35:40.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:40 smithi203 ceph-mon[32834]: pgmap v396: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-25T17:35:40.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:40 smithi173 ceph-mon[32856]: pgmap v396: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-25T17:35:40.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-25T17:35:41.038 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:42.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:42 smithi098 ceph-mon[29192]: pgmap v397: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:42.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:42 smithi203 ceph-mon[32834]: pgmap v397: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:42.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:42 smithi173 ceph-mon[32856]: pgmap v397: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-25T17:35:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.823+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/3960656463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d88108ab0 msgr2=0x7f8d88108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.823+0000 7f8d8d5d2700 1 --2- 172.21.15.98:0/3960656463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d88108ab0 0x7f8d88108e90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009990 tx=0x7f8d78009ca0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.823+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/3960656463 shutdown_connections 2024-04-25T17:35:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.823+0000 7f8d8d5d2700 1 --2- 172.21.15.98:0/3960656463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d88103480 0x7f8d8810f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.823+0000 7f8d8d5d2700 1 --2- 172.21.15.98:0/3960656463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d88108ab0 0x7f8d88108e90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.823+0000 7f8d8d5d2700 1 --2- 172.21.15.98:0/3960656463 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d88102ac0 0x7f8d88102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.823+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/3960656463 >> 172.21.15.98:0/3960656463 conn(0x7f8d88079bf0 msgr2=0x7f8d8807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.823+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/3960656463 shutdown_connections 2024-04-25T17:35:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.823+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/3960656463 wait complete. 2024-04-25T17:35:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d8d5d2700 1 Processor -- start 2024-04-25T17:35:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d8d5d2700 1 -- start start 2024-04-25T17:35:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d8d5d2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d88102ac0 0x7f8d88074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d8d5d2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d88103480 0x7f8d88075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d8d5d2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d88108ab0 0x7f8d880757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d8d5d2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8d88112ce0 con 0x7f8d88102ac0 2024-04-25T17:35:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d8d5d2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8d88112b60 con 0x7f8d88108ab0 2024-04-25T17:35:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d8d5d2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8d88112e60 con 0x7f8d88103480 2024-04-25T17:35:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d86ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d88102ac0 0x7f8d88074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d86ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d88102ac0 0x7f8d88074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55244/0 (socket says 172.21.15.98:55244) 2024-04-25T17:35:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d867fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d88103480 0x7f8d88075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d86ffd700 1 -- 172.21.15.98:0/2781686609 learned_addr learned my addr 172.21.15.98:0/2781686609 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.824+0000 7f8d877fe700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d88108ab0 0x7f8d880757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d877fe700 1 -- 172.21.15.98:0/2781686609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d88103480 msgr2=0x7f8d88075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d877fe700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d88103480 0x7f8d88075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d877fe700 1 -- 172.21.15.98:0/2781686609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d88102ac0 msgr2=0x7f8d88074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d877fe700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d88102ac0 0x7f8d88074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d877fe700 1 -- 172.21.15.98:0/2781686609 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d881a7700 con 0x7f8d88108ab0 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d86ffd700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d88102ac0 0x7f8d88074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d877fe700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d88108ab0 0x7f8d880757c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d80007f10 tx=0x7f8d8000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d6ffff700 1 -- 172.21.15.98:0/2781686609 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80004db0 con 0x7f8d88108ab0 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d6ffff700 1 -- 172.21.15.98:0/2781686609 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d80003a00 con 0x7f8d88108ab0 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d6ffff700 1 -- 172.21.15.98:0/2781686609 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80004580 con 0x7f8d88108ab0 2024-04-25T17:35:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/2781686609 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d881a79f0 con 0x7f8d88108ab0 2024-04-25T17:35:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.825+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/2781686609 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d881a7f30 con 0x7f8d88108ab0 2024-04-25T17:35:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.827+0000 7f8d6ffff700 1 -- 172.21.15.98:0/2781686609 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8d8001c020 con 0x7f8d88108ab0 2024-04-25T17:35:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.828+0000 7f8d6ffff700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8d700a3cc0 0x7f8d700a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.828+0000 7f8d6ffff700 1 -- 172.21.15.98:0/2781686609 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f8d800cfa20 con 0x7f8d88108ab0 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.828+0000 7f8d86ffd700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8d700a3cc0 0x7f8d700a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.828+0000 7f8d86ffd700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8d700a3cc0 0x7f8d700a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8d78006010 tx=0x7f8d7801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d97e5bb80 1 Processor -- start 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d97e5bb80 1 -- start start 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59c7d0 0x55eacc59cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59d0f0 0x55eacc5a1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 0x55eacc5a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eacc4eece0 con 0x55eacc5a1580 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eacc495910 con 0x55eacc59d0f0 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eacc4eaca0 con 0x55eacc59c7d0 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 0x55eacc5a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d867fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59d0f0 0x55eacc5a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d86ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59c7d0 0x55eacc59cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d867fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59d0f0 0x55eacc5a1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:35406/0 (socket says 172.21.15.98:35406) 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d86ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59c7d0 0x55eacc59cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:51774/0 (socket says 172.21.15.98:51774) 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.829+0000 7f8d86ffd700 1 -- 172.21.15.98:0/1433867342 learned_addr learned my addr 172.21.15.98:0/1433867342 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.830+0000 7f8d877fe700 1 -- 172.21.15.98:0/1433867342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59c7d0 msgr2=0x55eacc59cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.830+0000 7f8d877fe700 1 --2- 172.21.15.98:0/1433867342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59c7d0 0x55eacc59cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.830+0000 7f8d877fe700 1 -- 172.21.15.98:0/1433867342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59d0f0 msgr2=0x55eacc5a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.830+0000 7f8d877fe700 1 --2- 172.21.15.98:0/1433867342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59d0f0 0x55eacc5a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.830+0000 7f8d877fe700 1 -- 172.21.15.98:0/1433867342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eacc43c880 con 0x55eacc5a1580 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.830+0000 7f8d877fe700 1 --2- 172.21.15.98:0/1433867342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 0x55eacc5a19e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000d700 tx=0x7f8d800980c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bf432408a82fbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.830+0000 7f8d6d7fa700 1 -- 172.21.15.98:0/1433867342 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80019040 con 0x55eacc5a1580 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.830+0000 7f8d6d7fa700 1 -- 172.21.15.98:0/1433867342 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d8009c410 con 0x55eacc5a1580 2024-04-25T17:35:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.830+0000 7f8d6d7fa700 1 -- 172.21.15.98:0/1433867342 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80003bb0 con 0x55eacc5a1580 2024-04-25T17:35:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1433867342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 msgr2=0x55eacc5a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/1433867342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 0x55eacc5a19e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000d700 tx=0x7f8d800980c0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1433867342 shutdown_connections 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/1433867342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59c7d0 0x55eacc59cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/1433867342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 0x55eacc5a19e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/1433867342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59d0f0 0x55eacc5a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1433867342 >> 172.21.15.98:0/1433867342 conn(0x55eacc4c9640 msgr2=0x55eacc597820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1433867342 shutdown_connections 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1433867342 wait complete. 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 -- start start 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 Processor -- start 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 0x55eacc5a8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59d0f0 0x55eacc5a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 0x55eacc5a6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eacc5a95b0 con 0x55eacc5a1580 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.831+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eacc48db80 con 0x55eacc59c7d0 2024-04-25T17:35:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eacc5a9730 con 0x55eacc59d0f0 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d97e5bb80 10 client.0 _ll_register_callbacks cb 0x55eacc487900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d86ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 0x55eacc5a8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d867fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59d0f0 0x55eacc5a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d86ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 0x55eacc5a8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:35408/0 (socket says 172.21.15.98:35408) 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d86ffd700 1 -- 172.21.15.98:0/817746057 learned_addr learned my addr 172.21.15.98:0/817746057 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d877fe700 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 0x55eacc5a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d86ffd700 1 -- 172.21.15.98:0/817746057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59d0f0 msgr2=0x55eacc5a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d86ffd700 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59d0f0 0x55eacc5a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d86ffd700 1 -- 172.21.15.98:0/817746057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 msgr2=0x55eacc5a6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d86ffd700 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 0x55eacc5a6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d86ffd700 1 -- 172.21.15.98:0/817746057 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eacc4a6220 con 0x55eacc59c7d0 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d86ffd700 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 0x55eacc5a8ae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d78004730 tx=0x7f8d78009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44de13ec0c64d375 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d6aff5700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d6aff5700 1 -- 172.21.15.98:0/817746057 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7800f660 con 0x55eacc59c7d0 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/817746057 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d8009c410 con 0x55eacc59c7d0 2024-04-25T17:35:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d6aff5700 1 -- 172.21.15.98:0/817746057 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f8d7800f800 con 0x55eacc59c7d0 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d6aff5700 1 -- 172.21.15.98:0/817746057 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8d7800fa00 con 0x55eacc59c7d0 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.832+0000 7f8d97e5bb80 10 client.24752 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.833+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/817746057 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eacc5b3eb0 con 0x55eacc59c7d0 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.833+0000 7f8d6aff5700 1 -- 172.21.15.98:0/817746057 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d8009c410 con 0x55eacc59c7d0 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.833+0000 7f8d6aff5700 1 -- 172.21.15.98:0/817746057 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f8d7803a070 con 0x55eacc59c7d0 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.833+0000 7f8d97e5bb80 10 client.24752 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.833+0000 7f8d97e5bb80 10 client.24752 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.833+0000 7f8d97e5bb80 20 client.24752 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.833+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55eacc5b6570 0x55eacc5b79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.833+0000 7f8d97e5bb80 4 client.24752 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.833+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/817746057 --> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eacc4fc4d0 con 0x55eacc5b6570 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.833+0000 7f8d867fc700 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55eacc5b6570 0x55eacc5b79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.834+0000 7f8d6aff5700 10 client.24752 ms_handle_connect on v2:172.21.15.203:6838/1340539385 2024-04-25T17:35:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.834+0000 7f8d867fc700 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55eacc5b6570 0x55eacc5b79e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a415c6e913176fda server_cookie=305d8e745af18016 in_seq=0 out_seq=0 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.834+0000 7f8d6aff5700 1 -- 172.21.15.98:0/817746057 <== mds.1 v2:172.21.15.203:6838/1340539385 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eacc4fc4d0 con 0x55eacc5b6570 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.834+0000 7f8d6aff5700 10 client.24752 handle_command_reply: tid=0 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.834+0000 7f8d97e5bb80 1 client.24752 shutdown 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/817746057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 msgr2=0x55eacc5a8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 0x55eacc5a8ae0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d78004730 tx=0x7f8d78009ca0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/817746057 shutdown_connections 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55eacc5b6570 0x55eacc5b79e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc5a1580 0x55eacc5a6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 0x55eacc5a8ae0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d6aff5700 0 client.24752 ms_handle_reset on v2:172.21.15.203:6838/1340539385 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/817746057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59d0f0 0x55eacc5a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/817746057 >> 172.21.15.98:0/817746057 conn(0x55eacc4c9640 msgr2=0x55eacc596a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/817746057 shutdown_connections 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/817746057 wait complete. 2024-04-25T17:35:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.835+0000 7f8d97e5bb80 20 client.24752 trim_cache size 0 max 16384 2024-04-25T17:35:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d97e5bb80 1 Processor -- start 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d97e5bb80 1 -- start start 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 0x55eacc59bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59d0f0 0x55eacc5a8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc5a1580 0x55eacc5c0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eacc5b5d20 con 0x55eacc59d0f0 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eacc5b63f0 con 0x55eacc59c7d0 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eacc5b6ac0 con 0x55eacc5a1580 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d867fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59d0f0 0x55eacc5a8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d867fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59d0f0 0x55eacc5a8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55266/0 (socket says 172.21.15.98:55266) 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d867fc700 1 -- 172.21.15.98:0/783642325 learned_addr learned my addr 172.21.15.98:0/783642325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d86ffd700 1 --2- 172.21.15.98:0/783642325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 0x55eacc59bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d877fe700 1 --2- 172.21.15.98:0/783642325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc5a1580 0x55eacc5c0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d867fc700 1 -- 172.21.15.98:0/783642325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc5a1580 msgr2=0x55eacc5c0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d867fc700 1 --2- 172.21.15.98:0/783642325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc5a1580 0x55eacc5c0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d867fc700 1 -- 172.21.15.98:0/783642325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 msgr2=0x55eacc59bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d867fc700 1 --2- 172.21.15.98:0/783642325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 0x55eacc59bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d867fc700 1 -- 172.21.15.98:0/783642325 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eacc5a6d10 con 0x55eacc59d0f0 2024-04-25T17:35:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.862+0000 7f8d877fe700 1 --2- 172.21.15.98:0/783642325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc5a1580 0x55eacc5c0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.863+0000 7f8d867fc700 1 --2- 172.21.15.98:0/783642325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59d0f0 0x55eacc5a8620 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c004790 tx=0x7f8d7c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e4c825a68dc5eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.863+0000 7f8d6cff9700 1 -- 172.21.15.98:0/783642325 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c003a60 con 0x55eacc59d0f0 2024-04-25T17:35:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.863+0000 7f8d6cff9700 1 -- 172.21.15.98:0/783642325 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d7c003c00 con 0x55eacc59d0f0 2024-04-25T17:35:42.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.863+0000 7f8d6cff9700 1 -- 172.21.15.98:0/783642325 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c010690 con 0x55eacc59d0f0 2024-04-25T17:35:42.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/783642325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59d0f0 msgr2=0x55eacc5a8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/783642325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59d0f0 0x55eacc5a8620 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c004790 tx=0x7f8d7c009260 comp rx=0 tx=0).stop 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/783642325 shutdown_connections 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/783642325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc5a1580 0x55eacc5c0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/783642325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc59c7d0 0x55eacc59bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/783642325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59d0f0 0x55eacc5a8620 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/783642325 >> 172.21.15.98:0/783642325 conn(0x55eacc4c9640 msgr2=0x55eacc5a0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/783642325 shutdown_connections 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/783642325 wait complete. 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 -- start start 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.864+0000 7f8d97e5bb80 1 Processor -- start 2024-04-25T17:35:42.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59c7d0 0x55eacc5c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59d0f0 0x55eacc5c1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d97e5bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc5a1580 0x55eacc5bb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eacc5b7860 con 0x55eacc59c7d0 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eacc5b7190 con 0x55eacc5a1580 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d97e5bb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eacc5b7f30 con 0x55eacc59d0f0 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d97e5bb80 10 client.0 _ll_register_callbacks cb 0x55eacc487900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d86ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59c7d0 0x55eacc5c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d867fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59d0f0 0x55eacc5c1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d86ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59c7d0 0x55eacc5c0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55274/0 (socket says 172.21.15.98:55274) 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d86ffd700 1 -- 172.21.15.98:0/1432032394 learned_addr learned my addr 172.21.15.98:0/1432032394 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d877fe700 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc5a1580 0x55eacc5bb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d86ffd700 1 -- 172.21.15.98:0/1432032394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59d0f0 msgr2=0x55eacc5c1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d86ffd700 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59d0f0 0x55eacc5c1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d86ffd700 1 -- 172.21.15.98:0/1432032394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc5a1580 msgr2=0x55eacc5bb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d86ffd700 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc5a1580 0x55eacc5bb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d86ffd700 1 -- 172.21.15.98:0/1432032394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eacc5bb980 con 0x55eacc59c7d0 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d877fe700 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc5a1580 0x55eacc5bb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d86ffd700 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59c7d0 0x55eacc5c0d50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8d78005930 tx=0x7f8d7800fe50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8e513328b23afa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d6aff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.865+0000 7f8d6aff5700 1 -- 172.21.15.98:0/1432032394 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78009510 con 0x55eacc59c7d0 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d6aff5700 1 -- 172.21.15.98:0/1432032394 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f8d78031430 con 0x55eacc59c7d0 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1432032394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d7c003c00 con 0x55eacc59c7d0 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d6aff5700 1 -- 172.21.15.98:0/1432032394 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8d7804c050 con 0x55eacc59c7d0 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d97e5bb80 10 client.15087 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:42.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1432032394 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eacc61e260 con 0x55eacc59c7d0 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d6aff5700 1 -- 172.21.15.98:0/1432032394 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d7c003c00 con 0x55eacc59c7d0 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d6aff5700 1 -- 172.21.15.98:0/1432032394 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f8d7803a040 con 0x55eacc59c7d0 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d97e5bb80 10 client.15087 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d97e5bb80 10 client.15087 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d97e5bb80 20 client.15087 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55eacc622e20 0x55eacc621b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d97e5bb80 4 client.15087 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1432032394 --> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] -- command(tid 0: {"prefix": "status"}) -- 0x55eacc4f6570 con 0x55eacc622e20 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.866+0000 7f8d867fc700 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55eacc622e20 0x55eacc621b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d6aff5700 10 client.15087 ms_handle_connect on v2:172.21.15.203:6838/1340539385 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d867fc700 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55eacc622e20 0x55eacc621b30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5fe115b0e52b411c server_cookie=e58ec57bea574773 in_seq=0 out_seq=0 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d6aff5700 1 -- 172.21.15.98:0/1432032394 <== mds.1 v2:172.21.15.203:6838/1340539385 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55eacc4f6570 con 0x55eacc622e20 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d6aff5700 10 client.15087 handle_command_reply: tid=0 2024-04-25T17:35:42.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 client.15087 shutdown 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1432032394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59c7d0 msgr2=0x55eacc5c0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59c7d0 0x55eacc5c0d50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8d78005930 tx=0x7f8d7800fe50 comp rx=0 tx=0).stop 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1432032394 shutdown_connections 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55eacc622e20 0x55eacc621b30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d6aff5700 0 client.15087 ms_handle_reset on v2:172.21.15.203:6838/1340539385 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eacc5a1580 0x55eacc5bb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eacc59c7d0 0x55eacc5c0d50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 --2- 172.21.15.98:0/1432032394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eacc59d0f0 0x55eacc5c1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1432032394 >> 172.21.15.98:0/1432032394 conn(0x55eacc4c9640 msgr2=0x55eacc5b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1432032394 shutdown_connections 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.867+0000 7f8d97e5bb80 1 -- 172.21.15.98:0/1432032394 wait complete. 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d97e5bb80 20 client.15087 trim_cache size 0 max 16384 2024-04-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24445, 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 170.75786250499999, 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 25, 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 215.150707418 2024-04-25T17:35:42.875 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:35:42.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/2781686609 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8d700a3cc0 msgr2=0x7f8d700a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:42.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8d700a3cc0 0x7f8d700a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8d78006010 tx=0x7f8d7801a040 comp rx=0 tx=0).stop 2024-04-25T17:35:42.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/2781686609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d88108ab0 msgr2=0x7f8d880757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:42.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d88108ab0 0x7f8d880757c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d80007f10 tx=0x7f8d8000d3b0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/2781686609 shutdown_connections 2024-04-25T17:35:42.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f8d700a3cc0 0x7f8d700a6180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d88108ab0 0x7f8d880757c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d88102ac0 0x7f8d88074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 --2- 172.21.15.98:0/2781686609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d88103480 0x7f8d88075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:42.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/2781686609 >> 172.21.15.98:0/2781686609 conn(0x7f8d88079bf0 msgr2=0x7f8d88100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:42.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/2781686609 shutdown_connections 2024-04-25T17:35:42.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:42.868+0000 7f8d8d5d2700 1 -- 172.21.15.98:0/2781686609 wait complete. 2024-04-25T17:35:43.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:43.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:43 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:43.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:43 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:43.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:43 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:43.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-25T17:35:43.922 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:44.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:44 smithi098 ceph-mon[29192]: pgmap v398: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T17:35:44.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:44.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:44.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:44 smithi098 ceph-mon[29192]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:44.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:44 smithi203 ceph-mon[32834]: pgmap v398: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T17:35:44.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:44.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:44.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:44 smithi203 ceph-mon[32834]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:44.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:44 smithi173 ceph-mon[32856]: pgmap v398: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-25T17:35:44.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:44.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:44.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:44 smithi173 ceph-mon[32856]: from='mgr.14152 172.21.15.98:0/2879628905' entity='mgr.x' 2024-04-25T17:35:45.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.772+0000 7efefb3fa700 1 -- 172.21.15.98:0/1750399067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efef4105480 msgr2=0x7efef4105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:45.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.772+0000 7efefb3fa700 1 --2- 172.21.15.98:0/1750399067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efef4105480 0x7efef4105910 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7efeec00b0a0 tx=0x7efeec00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.772+0000 7efefb3fa700 1 -- 172.21.15.98:0/1750399067 shutdown_connections 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.772+0000 7efefb3fa700 1 --2- 172.21.15.98:0/1750399067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efef4105480 0x7efef4105910 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.772+0000 7efefb3fa700 1 --2- 172.21.15.98:0/1750399067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efef4107970 0x7efef406b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.772+0000 7efefb3fa700 1 --2- 172.21.15.98:0/1750399067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef406b740 0x7efef406bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.772+0000 7efefb3fa700 1 -- 172.21.15.98:0/1750399067 >> 172.21.15.98:0/1750399067 conn(0x7efef40788f0 msgr2=0x7efef4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.773+0000 7efefb3fa700 1 -- 172.21.15.98:0/1750399067 shutdown_connections 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.773+0000 7efefb3fa700 1 -- 172.21.15.98:0/1750399067 wait complete. 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.773+0000 7efefb3fa700 1 Processor -- start 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.773+0000 7efefb3fa700 1 -- start start 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.773+0000 7efefb3fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efef406b740 0x7efef41a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.773+0000 7efefb3fa700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef4105480 0x7efef41a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.773+0000 7efefb3fa700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efef4107970 0x7efef419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.773+0000 7efefb3fa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efef41192b0 con 0x7efef406b740 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.773+0000 7efefb3fa700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efef4119130 con 0x7efef4107970 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.773+0000 7efefb3fa700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efef4119430 con 0x7efef4105480 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9196700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efef406b740 0x7efef41a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9196700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efef406b740 0x7efef41a5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46400/0 (socket says 172.21.15.98:46400) 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9196700 1 -- 172.21.15.98:0/3556964625 learned_addr learned my addr 172.21.15.98:0/3556964625 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef8995700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef4105480 0x7efef41a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9997700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efef4107970 0x7efef419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9196700 1 -- 172.21.15.98:0/3556964625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef4105480 msgr2=0x7efef41a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9196700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef4105480 0x7efef41a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9196700 1 -- 172.21.15.98:0/3556964625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efef4107970 msgr2=0x7efef419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9196700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efef4107970 0x7efef419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9196700 1 -- 172.21.15.98:0/3556964625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef419ff90 con 0x7efef406b740 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9997700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efef4107970 0x7efef419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:35:45.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efef9196700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efef406b740 0x7efef41a5540 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efef000b530 tx=0x7efef000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efeea7fc700 1 -- 172.21.15.98:0/3556964625 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0015020 con 0x7efef406b740 2024-04-25T17:35:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efeea7fc700 1 -- 172.21.15.98:0/3556964625 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efef0011ce0 con 0x7efef406b740 2024-04-25T17:35:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efefb3fa700 1 -- 172.21.15.98:0/3556964625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef41a0280 con 0x7efef406b740 2024-04-25T17:35:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efeea7fc700 1 -- 172.21.15.98:0/3556964625 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0010d20 con 0x7efef406b740 2024-04-25T17:35:45.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.774+0000 7efefb3fa700 1 -- 172.21.15.98:0/3556964625 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef41ac4c0 con 0x7efef406b740 2024-04-25T17:35:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.776+0000 7efeea7fc700 1 -- 172.21.15.98:0/3556964625 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7efef0018020 con 0x7efef406b740 2024-04-25T17:35:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.777+0000 7efeea7fc700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7efee00a3cc0 0x7efee00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.777+0000 7efef8995700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7efee00a3cc0 0x7efee00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.777+0000 7efeea7fc700 1 -- 172.21.15.98:0/3556964625 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7efef00cfbd0 con 0x7efef406b740 2024-04-25T17:35:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.777+0000 7efef8995700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7efee00a3cc0 0x7efee00a6180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efee4006fd0 tx=0x7efee4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7eff05c83b80 1 Processor -- start 2024-04-25T17:35:45.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7eff05c83b80 1 -- start start 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e7501d0b0 0x563e7501d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75028340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e75028880 0x563e7502ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7eff05c83b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563e74f70c40 con 0x563e7501d9d0 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7eff05c83b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563e74f178e0 con 0x563e75028880 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7eff05c83b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563e74f6cc00 con 0x563e7501d0b0 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef8995700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75028340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef8995700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75028340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46404/0 (socket says 172.21.15.98:46404) 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef8995700 1 -- 172.21.15.98:0/2802384943 learned_addr learned my addr 172.21.15.98:0/2802384943 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef9997700 1 --2- 172.21.15.98:0/2802384943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e75028880 0x563e7502ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef9196700 1 --2- 172.21.15.98:0/2802384943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e7501d0b0 0x563e7501d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef8995700 1 -- 172.21.15.98:0/2802384943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e7501d0b0 msgr2=0x563e7501d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef8995700 1 --2- 172.21.15.98:0/2802384943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e7501d0b0 0x563e7501d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef8995700 1 -- 172.21.15.98:0/2802384943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e75028880 msgr2=0x563e7502ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:45.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef8995700 1 --2- 172.21.15.98:0/2802384943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e75028880 0x563e7502ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef8995700 1 -- 172.21.15.98:0/2802384943 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e74edb6f0 con 0x563e7501d9d0 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.778+0000 7efef9997700 1 --2- 172.21.15.98:0/2802384943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e75028880 0x563e7502ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.779+0000 7efef8995700 1 --2- 172.21.15.98:0/2802384943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75028340 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7efee400d6a0 tx=0x7efee4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a6f21864273dbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.779+0000 7efedf7fe700 1 -- 172.21.15.98:0/2802384943 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4024070 con 0x563e7501d9d0 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.779+0000 7efedf7fe700 1 -- 172.21.15.98:0/2802384943 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee4014af0 con 0x563e7501d9d0 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.779+0000 7efedf7fe700 1 -- 172.21.15.98:0/2802384943 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee401fb50 con 0x563e7501d9d0 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.779+0000 7eff05c83b80 1 -- 172.21.15.98:0/2802384943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 msgr2=0x563e75028340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 --2- 172.21.15.98:0/2802384943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75028340 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7efee400d6a0 tx=0x7efee4010000 comp rx=0 tx=0).stop 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 -- 172.21.15.98:0/2802384943 shutdown_connections 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 --2- 172.21.15.98:0/2802384943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e7501d0b0 0x563e7501d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 --2- 172.21.15.98:0/2802384943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e75028880 0x563e7502ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 --2- 172.21.15.98:0/2802384943 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75028340 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 -- 172.21.15.98:0/2802384943 >> 172.21.15.98:0/2802384943 conn(0x563e74f4a840 msgr2=0x563e75019580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 -- 172.21.15.98:0/2802384943 shutdown_connections 2024-04-25T17:35:45.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 -- 172.21.15.98:0/2802384943 wait complete. 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 -- start start 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 Processor -- start 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d0b0 0x563e75027f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e7501d9d0 0x563e75025280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e75028880 0x563e750257c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563e7502b370 con 0x563e7501d0b0 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563e74f2a560 con 0x563e7501d9d0 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.780+0000 7eff05c83b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563e7502b4f0 con 0x563e75028880 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7eff05c83b80 10 client.0 _ll_register_callbacks cb 0x563e74f098d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efef9196700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d0b0 0x563e75027f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efef9196700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d0b0 0x563e75027f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46418/0 (socket says 172.21.15.98:46418) 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efef9196700 1 -- 172.21.15.98:0/4122947281 learned_addr learned my addr 172.21.15.98:0/4122947281 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efef9997700 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e75028880 0x563e750257c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efef9196700 1 -- 172.21.15.98:0/4122947281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e75028880 msgr2=0x563e750257c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efef9196700 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e75028880 0x563e750257c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efef9196700 1 -- 172.21.15.98:0/4122947281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e7501d9d0 msgr2=0x563e75025280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efef9196700 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e7501d9d0 0x563e75025280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efef9196700 1 -- 172.21.15.98:0/4122947281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e74ebe880 con 0x563e7501d0b0 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efef9196700 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d0b0 0x563e75027f80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7efef0098320 tx=0x7efef009b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f3f6ff9dec742c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efedcff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efedcff9700 1 -- 172.21.15.98:0/4122947281 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef00a0270 con 0x563e7501d0b0 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efedcff9700 1 -- 172.21.15.98:0/4122947281 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7efef00b5260 con 0x563e7501d0b0 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7eff05c83b80 1 -- 172.21.15.98:0/4122947281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efee4014af0 con 0x563e7501d0b0 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efedcff9700 1 -- 172.21.15.98:0/4122947281 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efef00a4b30 con 0x563e7501d0b0 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7eff05c83b80 10 client.15099 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7eff05c83b80 1 -- 172.21.15.98:0/4122947281 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563e75035c40 con 0x563e7501d0b0 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.781+0000 7efedcff9700 1 -- 172.21.15.98:0/4122947281 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee4014af0 con 0x563e7501d0b0 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.782+0000 7efedcff9700 1 -- 172.21.15.98:0/4122947281 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7efef00cdba0 con 0x563e7501d0b0 2024-04-25T17:35:45.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.782+0000 7eff05c83b80 10 client.15099 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.782+0000 7eff05c83b80 10 client.15099 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T17:35:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.782+0000 7eff05c83b80 20 client.15099 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.782+0000 7eff05c83b80 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563e75038210 0x563e75039680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.782+0000 7eff05c83b80 4 client.15099 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:35:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.782+0000 7eff05c83b80 1 -- 172.21.15.98:0/4122947281 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e74f7e430 con 0x563e75038210 2024-04-25T17:35:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.782+0000 7efef8995700 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563e75038210 0x563e75039680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.782+0000 7efef8995700 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563e75038210 0x563e75039680 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c83ba8cae8a2098f server_cookie=474c45f423d96443 in_seq=0 out_seq=0 2024-04-25T17:35:45.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.782+0000 7efedcff9700 10 client.15099 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:35:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7efedcff9700 1 -- 172.21.15.98:0/4122947281 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e74f7e430 con 0x563e75038210 2024-04-25T17:35:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7efedcff9700 10 client.15099 handle_command_reply: tid=0 2024-04-25T17:35:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7eff05c83b80 1 client.15099 shutdown 2024-04-25T17:35:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7eff05c83b80 1 -- 172.21.15.98:0/4122947281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d0b0 msgr2=0x563e75027f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:45.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7eff05c83b80 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d0b0 0x563e75027f80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7efef0098320 tx=0x7efef009b950 comp rx=0 tx=0).stop 2024-04-25T17:35:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7eff05c83b80 1 -- 172.21.15.98:0/4122947281 shutdown_connections 2024-04-25T17:35:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7eff05c83b80 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563e75038210 0x563e75039680 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7eff05c83b80 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e75028880 0x563e750257c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7eff05c83b80 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d0b0 0x563e75027f80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7eff05c83b80 1 --2- 172.21.15.98:0/4122947281 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e7501d9d0 0x563e75025280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7eff05c83b80 1 -- 172.21.15.98:0/4122947281 >> 172.21.15.98:0/4122947281 conn(0x563e74f4a840 msgr2=0x563e750191b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.783+0000 7efedcff9700 0 client.15099 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:35:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.784+0000 7eff05c83b80 1 -- 172.21.15.98:0/4122947281 shutdown_connections 2024-04-25T17:35:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.784+0000 7eff05c83b80 1 -- 172.21.15.98:0/4122947281 wait complete. 2024-04-25T17:35:45.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.784+0000 7eff05c83b80 20 client.15099 trim_cache size 0 max 16384 2024-04-25T17:35:45.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7eff05c83b80 1 Processor -- start 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7eff05c83b80 1 -- start start 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e7501d0b0 0x563e75027ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75022540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e75028880 0x563e75022a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7eff05c83b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563e750379c0 con 0x563e7501d9d0 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7eff05c83b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563e75038090 con 0x563e75028880 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7eff05c83b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563e75038760 con 0x563e7501d0b0 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7efef8995700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75022540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7efef8995700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75022540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46420/0 (socket says 172.21.15.98:46420) 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.804+0000 7efef8995700 1 -- 172.21.15.98:0/208494850 learned_addr learned my addr 172.21.15.98:0/208494850 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.805+0000 7efef9196700 1 --2- 172.21.15.98:0/208494850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e7501d0b0 0x563e75027ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.805+0000 7efef8995700 1 -- 172.21.15.98:0/208494850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e7501d0b0 msgr2=0x563e75027ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.805+0000 7efef8995700 1 --2- 172.21.15.98:0/208494850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e7501d0b0 0x563e75027ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.805+0000 7efef8995700 1 -- 172.21.15.98:0/208494850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e75028880 msgr2=0x563e75022a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.805+0000 7efef8995700 1 --2- 172.21.15.98:0/208494850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e75028880 0x563e75022a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.805+0000 7efef8995700 1 -- 172.21.15.98:0/208494850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e7501ef20 con 0x563e7501d9d0 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.805+0000 7efef8995700 1 --2- 172.21.15.98:0/208494850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75022540 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efee4022040 tx=0x7efee4004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fd10abf8af9379 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.805+0000 7efedeffd700 1 -- 172.21.15.98:0/208494850 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4024040 con 0x563e7501d9d0 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.805+0000 7efedeffd700 1 -- 172.21.15.98:0/208494850 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee401f700 con 0x563e7501d9d0 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.805+0000 7efedeffd700 1 -- 172.21.15.98:0/208494850 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee40148a0 con 0x563e7501d9d0 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 -- 172.21.15.98:0/208494850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 msgr2=0x563e75022540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 --2- 172.21.15.98:0/208494850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75022540 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efee4022040 tx=0x7efee4004860 comp rx=0 tx=0).stop 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 -- 172.21.15.98:0/208494850 shutdown_connections 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 --2- 172.21.15.98:0/208494850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e7501d0b0 0x563e75027ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 --2- 172.21.15.98:0/208494850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e75028880 0x563e75022a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 --2- 172.21.15.98:0/208494850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75022540 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 -- 172.21.15.98:0/208494850 >> 172.21.15.98:0/208494850 conn(0x563e74f4a840 msgr2=0x563e75034040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:45.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 -- 172.21.15.98:0/208494850 shutdown_connections 2024-04-25T17:35:45.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 -- 172.21.15.98:0/208494850 wait complete. 2024-04-25T17:35:45.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 -- start start 2024-04-25T17:35:45.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.806+0000 7eff05c83b80 1 Processor -- start 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e7501d0b0 0x563e75042b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75043040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7eff05c83b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e75028880 0x563e7503baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7eff05c83b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563e75039500 con 0x563e7501d9d0 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7eff05c83b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563e75038e30 con 0x563e7501d0b0 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7eff05c83b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563e75039bd0 con 0x563e75028880 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7eff05c83b80 10 client.0 _ll_register_callbacks cb 0x563e74f098d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef8995700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75043040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef8995700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75043040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46428/0 (socket says 172.21.15.98:46428) 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef8995700 1 -- 172.21.15.98:0/3718970562 learned_addr learned my addr 172.21.15.98:0/3718970562 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:45.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef9997700 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e75028880 0x563e7503baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef8995700 1 -- 172.21.15.98:0/3718970562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e75028880 msgr2=0x563e7503baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef8995700 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e75028880 0x563e7503baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef9196700 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e7501d0b0 0x563e75042b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef8995700 1 -- 172.21.15.98:0/3718970562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e7501d0b0 msgr2=0x563e75042b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef8995700 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e7501d0b0 0x563e75042b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef8995700 1 -- 172.21.15.98:0/3718970562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e7503c0f0 con 0x563e7501d9d0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efef8995700 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75043040 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efee400e010 tx=0x7efee4005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a270c20da7e7b437 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efedcff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efedcff9700 1 -- 172.21.15.98:0/3718970562 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee403a070 con 0x563e7501d9d0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7eff05c83b80 1 -- 172.21.15.98:0/3718970562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efee401f700 con 0x563e7501d9d0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.807+0000 7efedcff9700 1 -- 172.21.15.98:0/3718970562 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7efee401f8b0 con 0x563e7501d9d0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7efedcff9700 1 -- 172.21.15.98:0/3718970562 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efee4024040 con 0x563e7501d9d0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7eff05c83b80 10 client.15105 fetch_fsmap learned FSMap version 25 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7eff05c83b80 1 -- 172.21.15.98:0/3718970562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563e7509fc80 con 0x563e7501d9d0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7efedcff9700 1 -- 172.21.15.98:0/3718970562 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee401f700 con 0x563e7501d9d0 2024-04-25T17:35:45.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7efedcff9700 1 -- 172.21.15.98:0/3718970562 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7efee4013070 con 0x563e7501d9d0 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7eff05c83b80 10 client.15105 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7eff05c83b80 10 client.15105 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7eff05c83b80 20 client.15105 populate_metadata read hostname 'smithi098' 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7eff05c83b80 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563e750a40e0 0x563e750a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7eff05c83b80 4 client.15105 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7eff05c83b80 1 -- 172.21.15.98:0/3718970562 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "status"}) -- 0x563e74f784d0 con 0x563e750a40e0 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.808+0000 7efef9196700 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563e750a40e0 0x563e750a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7efef9196700 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563e750a40e0 0x563e750a2df0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd5769b5165f9070 server_cookie=276219511550701c in_seq=0 out_seq=0 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7efedcff9700 10 client.15105 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7efedcff9700 1 -- 172.21.15.98:0/3718970562 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x563e74f784d0 con 0x563e750a40e0 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7efedcff9700 10 client.15105 handle_command_reply: tid=0 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 client.15105 shutdown 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 -- 172.21.15.98:0/3718970562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 msgr2=0x563e75043040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:45.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75043040 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efee400e010 tx=0x7efee4005b00 comp rx=0 tx=0).stop 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 -- 172.21.15.98:0/3718970562 shutdown_connections 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563e750a40e0 0x563e750a2df0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e75028880 0x563e7503baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e7501d0b0 0x563e75042b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7efedcff9700 0 client.15105 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 --2- 172.21.15.98:0/3718970562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e7501d9d0 0x563e75043040 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 -- 172.21.15.98:0/3718970562 >> 172.21.15.98:0/3718970562 conn(0x563e74f4a840 msgr2=0x563e750337b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 -- 172.21.15.98:0/3718970562 shutdown_connections 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 1 -- 172.21.15.98:0/3718970562 wait complete. 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.809+0000 7eff05c83b80 20 client.15105 trim_cache size 0 max 16384 2024-04-25T17:35:45.815 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 187.739738297, 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 25, 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 91, 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 89, 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 206.467197838 2024-04-25T17:35:45.816 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:35:45.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 -- 172.21.15.98:0/3556964625 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7efee00a3cc0 msgr2=0x7efee00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:45.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7efee00a3cc0 0x7efee00a6180 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efee4006fd0 tx=0x7efee4008040 comp rx=0 tx=0).stop 2024-04-25T17:35:45.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 -- 172.21.15.98:0/3556964625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efef406b740 msgr2=0x7efef41a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:45.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efef406b740 0x7efef41a5540 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efef000b530 tx=0x7efef000b8f0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 -- 172.21.15.98:0/3556964625 shutdown_connections 2024-04-25T17:35:45.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7efee00a3cc0 0x7efee00a6180 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efef4107970 0x7efef419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efef406b740 0x7efef41a5540 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 --2- 172.21.15.98:0/3556964625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efef4105480 0x7efef41a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:45.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 -- 172.21.15.98:0/3556964625 >> 172.21.15.98:0/3556964625 conn(0x7efef40788f0 msgr2=0x7efef4104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:45.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 -- 172.21.15.98:0/3556964625 shutdown_connections 2024-04-25T17:35:45.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:45.810+0000 7efefb3fa700 1 -- 172.21.15.98:0/3556964625 wait complete. 2024-04-25T17:35:46.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:46 smithi098 ceph-mon[29192]: pgmap v399: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:46.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:46 smithi203 ceph-mon[32834]: pgmap v399: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:46.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:46 smithi173 ceph-mon[32856]: pgmap v399: 97 pgs: 97 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-25T17:35:46.588 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T17:35:46.597 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T17:35:46.598 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T17:35:46.598 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T17:35:46.598 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-25T17:35:46.599 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T17:35:46.599 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:35:46.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-25T17:35:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.701+0000 7fdf487e8640 1 -- 172.21.15.98:0/1806674279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf40100ea0 msgr2=0x7fdf401012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:46.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.701+0000 7fdf487e8640 1 --2- 172.21.15.98:0/1806674279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf40100ea0 0x7fdf401012c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009ca0 tx=0x7fdf3c02a8c0 comp rx=0 tx=0).stop 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.701+0000 7fdf487e8640 1 -- 172.21.15.98:0/1806674279 shutdown_connections 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.701+0000 7fdf487e8640 1 --2- 172.21.15.98:0/1806674279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf40102a40 0x7fdf40109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.701+0000 7fdf487e8640 1 --2- 172.21.15.98:0/1806674279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf40102080 0x7fdf40102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.701+0000 7fdf487e8640 1 --2- 172.21.15.98:0/1806674279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf40100ea0 0x7fdf401012c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.701+0000 7fdf487e8640 1 -- 172.21.15.98:0/1806674279 >> 172.21.15.98:0/1806674279 conn(0x7fdf400fc460 msgr2=0x7fdf400fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.701+0000 7fdf487e8640 1 -- 172.21.15.98:0/1806674279 shutdown_connections 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.701+0000 7fdf487e8640 1 -- 172.21.15.98:0/1806674279 wait complete. 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf487e8640 1 Processor -- start 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf487e8640 1 -- start start 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf487e8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf40100ea0 0x7fdf401970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf487e8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf40102080 0x7fdf40197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf487e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf40102a40 0x7fdf4019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:46.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf487e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4010caa0 con 0x7fdf40102a40 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf487e8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4010c920 con 0x7fdf40100ea0 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf487e8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4010cc20 con 0x7fdf40102080 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf46d85640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf40102a40 0x7fdf4019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf46d85640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf40102a40 0x7fdf4019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46456/0 (socket says 172.21.15.98:46456) 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf46d85640 1 -- 172.21.15.98:0/802333841 learned_addr learned my addr 172.21.15.98:0/802333841 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf46584640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf40100ea0 0x7fdf401970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf46d85640 1 -- 172.21.15.98:0/802333841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf40102080 msgr2=0x7fdf40197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf45d83640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf40102080 0x7fdf40197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf46d85640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf40102080 0x7fdf40197600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf46d85640 1 -- 172.21.15.98:0/802333841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf40100ea0 msgr2=0x7fdf401970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf46d85640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf40100ea0 0x7fdf401970c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.702+0000 7fdf46d85640 1 -- 172.21.15.98:0/802333841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4019ece0 con 0x7fdf40102a40 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.703+0000 7fdf46d85640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf40102a40 0x7fdf4019e670 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800d750 tx=0x7fdf3800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:46.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.703+0000 7fdf377fe640 1 -- 172.21.15.98:0/802333841 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf38017070 con 0x7fdf40102a40 2024-04-25T17:35:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.703+0000 7fdf377fe640 1 -- 172.21.15.98:0/802333841 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf380054d0 con 0x7fdf40102a40 2024-04-25T17:35:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.703+0000 7fdf487e8640 1 -- 172.21.15.98:0/802333841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4019efd0 con 0x7fdf40102a40 2024-04-25T17:35:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.703+0000 7fdf377fe640 1 -- 172.21.15.98:0/802333841 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf38004c60 con 0x7fdf40102a40 2024-04-25T17:35:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.703+0000 7fdf487e8640 1 -- 172.21.15.98:0/802333841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf4019f4c0 con 0x7fdf40102a40 2024-04-25T17:35:46.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.704+0000 7fdf487e8640 1 -- 172.21.15.98:0/802333841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf40108240 con 0x7fdf40102a40 2024-04-25T17:35:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.705+0000 7fdf377fe640 1 -- 172.21.15.98:0/802333841 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdf38009bd0 con 0x7fdf40102a40 2024-04-25T17:35:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.706+0000 7fdf377fe640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fdf2c0a3c50 0x7fdf2c0a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.706+0000 7fdf46584640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fdf2c0a3c50 0x7fdf2c0a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.706+0000 7fdf377fe640 1 -- 172.21.15.98:0/802333841 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fdf380cfdb0 con 0x7fdf40102a40 2024-04-25T17:35:46.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.706+0000 7fdf46584640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fdf2c0a3c50 0x7fdf2c0a6110 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c000f80 tx=0x7fdf3c005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:46.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.711+0000 7fdf377fe640 1 -- 172.21.15.98:0/802333841 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf3809c7a0 con 0x7fdf40102a40 2024-04-25T17:35:46.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:46.936+0000 7fdf487e8640 1 -- 172.21.15.98:0/802333841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fdf40062bf0 con 0x7fdf40102a40 2024-04-25T17:35:47.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.162+0000 7fdf377fe640 1 -- 172.21.15.98:0/802333841 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fdf3809c0a0 con 0x7fdf40102a40 2024-04-25T17:35:47.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 -- 172.21.15.98:0/802333841 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fdf2c0a3c50 msgr2=0x7fdf2c0a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:47.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fdf2c0a3c50 0x7fdf2c0a6110 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c000f80 tx=0x7fdf3c005dc0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 -- 172.21.15.98:0/802333841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf40102a40 msgr2=0x7fdf4019e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:47.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf40102a40 0x7fdf4019e670 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800d750 tx=0x7fdf3800dc10 comp rx=0 tx=0).stop 2024-04-25T17:35:47.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 -- 172.21.15.98:0/802333841 shutdown_connections 2024-04-25T17:35:47.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fdf2c0a3c50 0x7fdf2c0a6110 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdf40102a40 0x7fdf4019e670 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf40102080 0x7fdf40197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 --2- 172.21.15.98:0/802333841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf40100ea0 0x7fdf401970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 -- 172.21.15.98:0/802333841 >> 172.21.15.98:0/802333841 conn(0x7fdf400fc460 msgr2=0x7fdf40105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:47.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 -- 172.21.15.98:0/802333841 shutdown_connections 2024-04-25T17:35:47.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.164+0000 7fdf487e8640 1 -- 172.21.15.98:0/802333841 wait complete. 2024-04-25T17:35:47.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-25T17:35:47.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.299+0000 7feef9e8a640 1 -- 172.21.15.98:0/1149293094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feef4102040 msgr2=0x7feef41024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:47.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.299+0000 7feef9e8a640 1 --2- 172.21.15.98:0/1149293094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feef4102040 0x7feef41024c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7feedc009900 tx=0x7feedc02a410 comp rx=0 tx=0).stop 2024-04-25T17:35:47.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.299+0000 7feef9e8a640 1 -- 172.21.15.98:0/1149293094 shutdown_connections 2024-04-25T17:35:47.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.299+0000 7feef9e8a640 1 --2- 172.21.15.98:0/1149293094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feef4102a00 0x7feef4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.299+0000 7feef9e8a640 1 --2- 172.21.15.98:0/1149293094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feef4102040 0x7feef41024c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.299+0000 7feef9e8a640 1 --2- 172.21.15.98:0/1149293094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feef4100e60 0x7feef4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.299+0000 7feef9e8a640 1 -- 172.21.15.98:0/1149293094 >> 172.21.15.98:0/1149293094 conn(0x7feef40fc420 msgr2=0x7feef40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:47.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.299+0000 7feef9e8a640 1 -- 172.21.15.98:0/1149293094 shutdown_connections 2024-04-25T17:35:47.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.299+0000 7feef9e8a640 1 -- 172.21.15.98:0/1149293094 wait complete. 2024-04-25T17:35:47.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef9e8a640 1 Processor -- start 2024-04-25T17:35:47.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef9e8a640 1 -- start start 2024-04-25T17:35:47.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef9e8a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feef4100e60 0x7feef406a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:47.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef9e8a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feef4102040 0x7feef419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef9e8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feef4102a00 0x7feef419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef9e8a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feef410ca10 con 0x7feef4102a00 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef9e8a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feef410c890 con 0x7feef4102040 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef9e8a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feef410cb90 con 0x7feef4100e60 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feef4102a00 0x7feef419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feef4102a00 0x7feef419f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46506/0 (socket says 172.21.15.98:46506) 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef3fff640 1 -- 172.21.15.98:0/7362283 learned_addr learned my addr 172.21.15.98:0/7362283 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef37fe640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feef4100e60 0x7feef406a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef2ffd640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feef4102040 0x7feef419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef3fff640 1 -- 172.21.15.98:0/7362283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feef4100e60 msgr2=0x7feef406a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef3fff640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feef4100e60 0x7feef406a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef3fff640 1 -- 172.21.15.98:0/7362283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feef4102040 msgr2=0x7feef419ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef3fff640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feef4102040 0x7feef419ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.300+0000 7feef3fff640 1 -- 172.21.15.98:0/7362283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef419fde0 con 0x7feef4102a00 2024-04-25T17:35:47.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.301+0000 7feef3fff640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feef4102a00 0x7feef419f4d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7feee400e970 tx=0x7feee400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:47.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.301+0000 7feef0ff9640 1 -- 172.21.15.98:0/7362283 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee400ccc0 con 0x7feef4102a00 2024-04-25T17:35:47.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.301+0000 7feef0ff9640 1 -- 172.21.15.98:0/7362283 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feee400ce60 con 0x7feef4102a00 2024-04-25T17:35:47.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.301+0000 7feef9e8a640 1 -- 172.21.15.98:0/7362283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef41a0070 con 0x7feef4102a00 2024-04-25T17:35:47.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.301+0000 7feef0ff9640 1 -- 172.21.15.98:0/7362283 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4010430 con 0x7feef4102a00 2024-04-25T17:35:47.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.301+0000 7feef9e8a640 1 -- 172.21.15.98:0/7362283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef41a0540 con 0x7feef4102a00 2024-04-25T17:35:47.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.302+0000 7feef9e8a640 1 -- 172.21.15.98:0/7362283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef4108280 con 0x7feef4102a00 2024-04-25T17:35:47.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.303+0000 7feef0ff9640 1 -- 172.21.15.98:0/7362283 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7feee4004ff0 con 0x7feef4102a00 2024-04-25T17:35:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.304+0000 7feef0ff9640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feee00a3ca0 0x7feee00a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.304+0000 7feef37fe640 1 -- 172.21.15.98:0/7362283 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feee00a3ca0 msgr2=0x7feee00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4145487482 2024-04-25T17:35:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.304+0000 7feef0ff9640 1 -- 172.21.15.98:0/7362283 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7feee40d08a0 con 0x7feef4102a00 2024-04-25T17:35:47.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.304+0000 7feef37fe640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feee00a3ca0 0x7feee00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:35:47.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.308+0000 7feef0ff9640 1 -- 172.21.15.98:0/7362283 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feee409d230 con 0x7feef4102a00 2024-04-25T17:35:47.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:47 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/802333841' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T17:35:47.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: ignoring --setuser ceph since I am not root 2024-04-25T17:35:47.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: ignoring --setgroup ceph since I am not root 2024-04-25T17:35:47.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:47.361+0000 7f683274d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T17:35:47.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:47 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/802333841' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T17:35:47.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: ignoring --setuser ceph since I am not root 2024-04-25T17:35:47.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: ignoring --setgroup ceph since I am not root 2024-04-25T17:35:47.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:47 smithi203 ceph-mgr[34647]: -- 172.21.15.203:0/1115078232 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557d1f5765a0 con 0x557d1f54ac00 2024-04-25T17:35:47.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:47.347+0000 7f8558d07200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T17:35:47.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.494+0000 7feef9e8a640 1 -- 172.21.15.98:0/7362283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7feef4062b50 con 0x7feef4102a00 2024-04-25T17:35:47.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.500+0000 7feef0ff9640 1 -- 172.21.15.98:0/7362283 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v27)=0 v27) ==== 138+0+0 (secure 0 0 0) 0x7feee409cb30 con 0x7feef4102a00 2024-04-25T17:35:47.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.502+0000 7feef9e8a640 1 -- 172.21.15.98:0/7362283 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feee00a3ca0 msgr2=0x7feee00a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.502+0000 7feef9e8a640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feee00a3ca0 0x7feee00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.502+0000 7feef9e8a640 1 -- 172.21.15.98:0/7362283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feef4102a00 msgr2=0x7feef419f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.503+0000 7feef9e8a640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feef4102a00 0x7feef419f4d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7feee400e970 tx=0x7feee400ed70 comp rx=0 tx=0).stop 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.503+0000 7feef9e8a640 1 -- 172.21.15.98:0/7362283 shutdown_connections 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.503+0000 7feef9e8a640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7feee00a3ca0 0x7feee00a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.503+0000 7feef9e8a640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feef4102a00 0x7feef419f4d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.503+0000 7feef9e8a640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feef4102040 0x7feef419ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.503+0000 7feef9e8a640 1 --2- 172.21.15.98:0/7362283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feef4100e60 0x7feef406a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.503+0000 7feef9e8a640 1 -- 172.21.15.98:0/7362283 >> 172.21.15.98:0/7362283 conn(0x7feef40fc420 msgr2=0x7feef4105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.503+0000 7feef9e8a640 1 -- 172.21.15.98:0/7362283 shutdown_connections 2024-04-25T17:35:47.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.503+0000 7feef9e8a640 1 -- 172.21.15.98:0/7362283 wait complete. 2024-04-25T17:35:47.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-25T17:35:47.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:47 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/802333841' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T17:35:47.522 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: ignoring --setuser ceph since I am not root 2024-04-25T17:35:47.523 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: ignoring --setgroup ceph since I am not root 2024-04-25T17:35:47.523 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:47 smithi173 ceph-mgr[34285]: -- 172.21.15.173:0/1089370540 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fa43eee5a0 con 0x55fa43ec3000 2024-04-25T17:35:47.523 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:47.341+0000 7f850d23d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T17:35:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.616+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/839933698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8105360 msgr2=0x7fb2e8107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.616+0000 7fb2ecbe5640 1 --2- 172.21.15.98:0/839933698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8105360 0x7fb2e8107750 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009900 tx=0x7fb2d802a410 comp rx=0 tx=0).stop 2024-04-25T17:35:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.616+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/839933698 shutdown_connections 2024-04-25T17:35:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.616+0000 7fb2ecbe5640 1 --2- 172.21.15.98:0/839933698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8107c90 0x7fb2e810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.616+0000 7fb2ecbe5640 1 --2- 172.21.15.98:0/839933698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8105360 0x7fb2e8107750 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.616+0000 7fb2ecbe5640 1 --2- 172.21.15.98:0/839933698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2e8102a30 0x7fb2e8104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.616+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/839933698 >> 172.21.15.98:0/839933698 conn(0x7fb2e80fc420 msgr2=0x7fb2e80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.616+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/839933698 shutdown_connections 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.616+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/839933698 wait complete. 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2ecbe5640 1 Processor -- start 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2ecbe5640 1 -- start start 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2ecbe5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8102a30 0x7fb2e8198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2ecbe5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2e8105360 0x7fb2e8198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2ecbe5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8107c90 0x7fb2e819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2ecbe5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e810daf0 con 0x7fb2e8107c90 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2ecbe5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e810d970 con 0x7fb2e8105360 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2ecbe5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e810dc70 con 0x7fb2e8102a30 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2e6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8107c90 0x7fb2e819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2e6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8107c90 0x7fb2e819f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46526/0 (socket says 172.21.15.98:46526) 2024-04-25T17:35:47.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2e6d9d640 1 -- 172.21.15.98:0/3361187444 learned_addr learned my addr 172.21.15.98:0/3361187444 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2e659c640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8102a30 0x7fb2e8198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2e5d9b640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2e8105360 0x7fb2e8198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2e6d9d640 1 -- 172.21.15.98:0/3361187444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8102a30 msgr2=0x7fb2e8198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2e6d9d640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8102a30 0x7fb2e8198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2e6d9d640 1 -- 172.21.15.98:0/3361187444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2e8105360 msgr2=0x7fb2e8198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2e6d9d640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2e8105360 0x7fb2e8198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.617+0000 7fb2e6d9d640 1 -- 172.21.15.98:0/3361187444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e819fd60 con 0x7fb2e8107c90 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.618+0000 7fb2e6d9d640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8107c90 0x7fb2e819f6f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000eb10 tx=0x7fb2e000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.618+0000 7fb2d77fe640 1 -- 172.21.15.98:0/3361187444 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0004b20 con 0x7fb2e8107c90 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.618+0000 7fb2d77fe640 1 -- 172.21.15.98:0/3361187444 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2e0004cc0 con 0x7fb2e8107c90 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.618+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/3361187444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e819fff0 con 0x7fb2e8107c90 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.618+0000 7fb2d77fe640 1 -- 172.21.15.98:0/3361187444 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0010670 con 0x7fb2e8107c90 2024-04-25T17:35:47.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.618+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/3361187444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e81a0510 con 0x7fb2e8107c90 2024-04-25T17:35:47.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.619+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/3361187444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e80fdf10 con 0x7fb2e8107c90 2024-04-25T17:35:47.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.620+0000 7fb2d77fe640 1 -- 172.21.15.98:0/3361187444 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fb2e0005020 con 0x7fb2e8107c90 2024-04-25T17:35:47.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.620+0000 7fb2d77fe640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb2d00a3ca0 0x7fb2d00a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:47.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.620+0000 7fb2e659c640 1 -- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb2d00a3ca0 msgr2=0x7fb2d00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4145487482 2024-04-25T17:35:47.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.621+0000 7fb2d77fe640 1 -- 172.21.15.98:0/3361187444 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fb2e00d0920 con 0x7fb2e8107c90 2024-04-25T17:35:47.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.621+0000 7fb2e659c640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb2d00a3ca0 0x7fb2d00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:35:47.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.625+0000 7fb2d77fe640 1 -- 172.21.15.98:0/3361187444 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2e009d310 con 0x7fb2e8107c90 2024-04-25T17:35:47.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.816+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/3361187444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fb2e8062bc0 con 0x7fb2e8107c90 2024-04-25T17:35:47.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.822+0000 7fb2e659c640 1 -- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb2d00a3ca0 msgr2=0x7fb2d00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4145487482 2024-04-25T17:35:47.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.822+0000 7fb2e659c640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb2d00a3ca0 0x7fb2d00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T17:35:47.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.822+0000 7fb2d77fe640 1 -- 172.21.15.98:0/3361187444 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v28)=0 v28) ==== 133+0+0 (secure 0 0 0) 0x7fb2e009cc10 con 0x7fb2e8107c90 2024-04-25T17:35:47.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb2d00a3ca0 msgr2=0x7fb2d00a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:35:47.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb2d00a3ca0 0x7fb2d00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8107c90 msgr2=0x7fb2e819f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:47.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8107c90 0x7fb2e819f6f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000eb10 tx=0x7fb2e000c2d0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/3361187444 shutdown_connections 2024-04-25T17:35:47.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fb2d00a3ca0 0x7fb2d00a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2e8107c90 0x7fb2e819f6f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2e8105360 0x7fb2e8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 --2- 172.21.15.98:0/3361187444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2e8102a30 0x7fb2e8198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:47.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/3361187444 >> 172.21.15.98:0/3361187444 conn(0x7fb2e80fc420 msgr2=0x7fb2e8103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:47.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/3361187444 shutdown_connections 2024-04-25T17:35:47.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:47.824+0000 7fb2ecbe5640 1 -- 172.21.15.98:0/3361187444 wait complete. 2024-04-25T17:35:47.834 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:47.551+0000 7f683274d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:35:47.835 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:47.667+0000 7f683274d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T17:35:47.836 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T17:35:47.860 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T17:35:47.861 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-25T17:35:47.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:35:47.861 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-25T17:35:47.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:35:47.876 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-04-25T17:35:47.930 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:47.524+0000 7f850d23d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:35:47.930 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:47.625+0000 7f850d23d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T17:35:47.942 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2024-04-25T17:35:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:35:47.944 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v2', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-04-25T17:35:47.944 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T17:35:47.944 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-04-25T17:35:47.944 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-04-25T17:35:47.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T17:35:47.949 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:47.535+0000 7f8558d07200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:35:47.949 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:47.638+0000 7f8558d07200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T17:35:47.949 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:47.948+0000 7f8558d07200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T17:35:48.166 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:47.984+0000 7f683274d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T17:35:48.166 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:48 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:48.073+0000 7f683274d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T17:35:48.213 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:48 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:48.034+0000 7f8558d07200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T17:35:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:48 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/802333841' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T17:35:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:48 smithi203 ceph-mon[32834]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-25T17:35:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:48 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/7362283' entity='client.admin' 2024-04-25T17:35:48.213 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:48 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3361187444' entity='client.admin' 2024-04-25T17:35:48.238 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:48.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:48 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/802333841' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T17:35:48.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:48 smithi173 ceph-mon[32856]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-25T17:35:48.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:48 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/7362283' entity='client.admin' 2024-04-25T17:35:48.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:48 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3361187444' entity='client.admin' 2024-04-25T17:35:48.273 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:47.929+0000 7f850d23d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T17:35:48.273 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:48 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:48.013+0000 7f850d23d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T17:35:48.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:48 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/802333841' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T17:35:48.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:48 smithi098 ceph-mon[29192]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-25T17:35:48.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:48 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/7362283' entity='client.admin' 2024-04-25T17:35:48.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:48 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3361187444' entity='client.admin' 2024-04-25T17:35:49.518 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:49 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:49.180+0000 7f850d23d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T17:35:49.518 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:49 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:49.347+0000 7f850d23d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T17:35:49.565 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:49 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:49.223+0000 7f8558d07200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T17:35:49.565 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:49 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:49.394+0000 7f8558d07200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T17:35:49.565 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:49 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:49.564+0000 7f8558d07200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T17:35:49.641 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:49.299+0000 7f683274d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T17:35:49.641 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:49.472+0000 7f683274d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T17:35:49.772 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:49 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:49.517+0000 7f850d23d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T17:35:49.950 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:49.650+0000 7f683274d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T17:35:49.950 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:49.949+0000 7f683274d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T17:35:50.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 -- 172.21.15.98:0/1584629958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed80101930 msgr2=0x7fed80101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 --2- 172.21.15.98:0/1584629958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed80101930 0x7fed80101db0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009990 tx=0x7fed6c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 -- 172.21.15.98:0/1584629958 shutdown_connections 2024-04-25T17:35:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 --2- 172.21.15.98:0/1584629958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed801022f0 0x7fed8010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 --2- 172.21.15.98:0/1584629958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80107980 0x7fed80107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 --2- 172.21.15.98:0/1584629958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed80101930 0x7fed80101db0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 -- 172.21.15.98:0/1584629958 >> 172.21.15.98:0/1584629958 conn(0x7fed800788b0 msgr2=0x7fed80078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 -- 172.21.15.98:0/1584629958 shutdown_connections 2024-04-25T17:35:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 -- 172.21.15.98:0/1584629958 wait complete. 2024-04-25T17:35:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 Processor -- start 2024-04-25T17:35:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 -- start start 2024-04-25T17:35:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed80101930 0x7fed80199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed801022f0 0x7fed8019a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.060+0000 7fed86b17700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80107980 0x7fed801a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed86b17700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed80111ae0 con 0x7fed80101930 2024-04-25T17:35:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed86b17700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed80111960 con 0x7fed80107980 2024-04-25T17:35:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed86b17700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fed80111c60 con 0x7fed801022f0 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed848b3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed80101930 0x7fed80199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed848b3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed80101930 0x7fed80199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46544/0 (socket says 172.21.15.98:46544) 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed848b3700 1 -- 172.21.15.98:0/3094223379 learned_addr learned my addr 172.21.15.98:0/3094223379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed850b4700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80107980 0x7fed801a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed7ffff700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed801022f0 0x7fed8019a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed848b3700 1 -- 172.21.15.98:0/3094223379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed801022f0 msgr2=0x7fed8019a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed848b3700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed801022f0 0x7fed8019a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed848b3700 1 -- 172.21.15.98:0/3094223379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80107980 msgr2=0x7fed801a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed848b3700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80107980 0x7fed801a4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed848b3700 1 -- 172.21.15.98:0/3094223379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed801a54e0 con 0x7fed80101930 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed848b3700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed80101930 0x7fed80199f20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fed7400d980 tx=0x7fed7400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed7dffb700 1 -- 172.21.15.98:0/3094223379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed74016070 con 0x7fed80101930 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed86b17700 1 -- 172.21.15.98:0/3094223379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed80197d70 con 0x7fed80101930 2024-04-25T17:35:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed7dffb700 1 -- 172.21.15.98:0/3094223379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed74004830 con 0x7fed80101930 2024-04-25T17:35:50.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.061+0000 7fed7dffb700 1 -- 172.21.15.98:0/3094223379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed74005540 con 0x7fed80101930 2024-04-25T17:35:50.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.062+0000 7fed86b17700 1 -- 172.21.15.98:0/3094223379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed801982d0 con 0x7fed80101930 2024-04-25T17:35:50.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.062+0000 7fed86b17700 1 -- 172.21.15.98:0/3094223379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed80069730 con 0x7fed80101930 2024-04-25T17:35:50.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.063+0000 7fed7dffb700 1 -- 172.21.15.98:0/3094223379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fed74004220 con 0x7fed80101930 2024-04-25T17:35:50.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.064+0000 7fed7dffb700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fed700a3c90 0x7fed700a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.064+0000 7fed7ffff700 1 -- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fed700a3c90 msgr2=0x7fed700a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4145487482 2024-04-25T17:35:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.064+0000 7fed7dffb700 1 -- 172.21.15.98:0/3094223379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7fed740d0160 con 0x7fed80101930 2024-04-25T17:35:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.064+0000 7fed7ffff700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fed700a3c90 0x7fed700a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:35:50.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.069+0000 7fed7dffb700 1 -- 172.21.15.98:0/3094223379 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed7401f020 con 0x7fed80101930 2024-04-25T17:35:50.095 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:49 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:49.801+0000 7f850d23d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T17:35:50.095 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:49 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:49.892+0000 7f850d23d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T17:35:50.151 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:49 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:49.853+0000 7f8558d07200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T17:35:50.151 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:49 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:49.946+0000 7f8558d07200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T17:35:50.206 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:50.048+0000 7f683274d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T17:35:50.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.265+0000 7fed7ffff700 1 -- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fed700a3c90 msgr2=0x7fed700a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4145487482 2024-04-25T17:35:50.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.265+0000 7fed7ffff700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fed700a3c90 0x7fed700a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T17:35:50.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.323+0000 7fed86b17700 1 -- 172.21.15.98:0/3094223379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fed8002df80 con 0x7fed80101930 2024-04-25T17:35:50.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.324+0000 7fed7dffb700 1 -- 172.21.15.98:0/3094223379 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7fed7409cb50 con 0x7fed80101930 2024-04-25T17:35:50.324 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T17:35:50.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 -- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fed700a3c90 msgr2=0x7fed700a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:35:50.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fed700a3c90 0x7fed700a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 -- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed80101930 msgr2=0x7fed80199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:50.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed80101930 0x7fed80199f20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fed7400d980 tx=0x7fed7400dc90 comp rx=0 tx=0).stop 2024-04-25T17:35:50.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 -- 172.21.15.98:0/3094223379 shutdown_connections 2024-04-25T17:35:50.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7fed700a3c90 0x7fed700a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed80107980 0x7fed801a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed80101930 0x7fed80199f20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 --2- 172.21.15.98:0/3094223379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed801022f0 0x7fed8019a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:50.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 -- 172.21.15.98:0/3094223379 >> 172.21.15.98:0/3094223379 conn(0x7fed800788b0 msgr2=0x7fed800ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:50.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 -- 172.21.15.98:0/3094223379 shutdown_connections 2024-04-25T17:35:50.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:50.326+0000 7fed86b17700 1 -- 172.21.15.98:0/3094223379 wait complete. 2024-04-25T17:35:50.363 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:50 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:50.094+0000 7f850d23d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T17:35:50.364 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:50 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:50.295+0000 7f850d23d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:35:50.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:50 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3094223379' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T17:35:50.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:50.150+0000 7f8558d07200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T17:35:50.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:50.345+0000 7f8558d07200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:35:50.703 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:50 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3094223379' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T17:35:50.703 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:50.260+0000 7f683274d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T17:35:50.703 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:50.464+0000 7f683274d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T17:35:50.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:50 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3094223379' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T17:35:51.067 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T17:35:51.067 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T17:35:51.067 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-25T17:35:51.067 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T17:35:51.067 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-04-25T17:35:51.067 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T17:35:51.067 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T17:35:51.067 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T17:35:51.067 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-04-25T17:35:51.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:35:51.068 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-25T17:35:51.083 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T17:35:51.084 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-25T17:35:51.085 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T17:35:51.085 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-25T17:35:51.085 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T17:35:51.085 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42023sec preferred_lft 42023sec 2024-04-25T17:35:51.085 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::c454:7d32:cfc2:7ded/64 scope link noprefixroute 2024-04-25T17:35:51.085 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-25T17:35:51.085 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T17:35:51.085 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T17:35:51.086 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T17:35:51.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-04-25T17:35:51.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-04-25T17:35:51.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-04-25T17:35:51.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T17:35:51.086 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T17:35:51.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T17:35:51.212 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:50.948+0000 7f8558d07200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T17:35:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T17:35:51.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:35:51.266 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T17:35:51.272 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:50 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:50.916+0000 7f850d23d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T17:35:51.289 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:51.077+0000 7f683274d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T17:35:51.292 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-25T17:35:51.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:35:51.294 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-25T17:35:51.378 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-25T17:35:51.378 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T17:35:51.378 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T17:35:51.378 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T17:35:51.378 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T17:35:51.379 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T17:35:51.379 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T17:35:51.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T17:35:51.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T17:35:51.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T17:35:51.379 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T17:35:51.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T17:35:51.569 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:51.534+0000 7f683274d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T17:35:51.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T17:35:51.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:35:51.647 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-25T17:35:51.648 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:51 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:51.350+0000 7f850d23d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T17:35:51.648 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:51 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:51.446+0000 7f850d23d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T17:35:51.648 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:51 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:51.646+0000 7f850d23d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T17:35:51.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:35:51.663 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-04-25T17:35:51.673 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:51 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:51.373+0000 7f8558d07200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T17:35:51.673 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:51 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:51.470+0000 7f8558d07200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T17:35:51.673 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:51 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:51.672+0000 7f8558d07200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T17:35:51.718 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T17:35:51.718 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T17:35:51.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--tmp-cephfs 2024-04-25T17:35:51.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-04-25T17:35:51.718 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T17:35:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T17:35:51.825 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:51.639+0000 7f683274d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T17:35:51.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T17:35:51.835 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-04-25T17:35:51.835 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T17:35:51.835 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T17:35:51.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-04-25T17:35:51.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T17:35:51.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-04-25T17:35:51.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-04-25T17:35:51.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-04-25T17:35:51.836 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T17:35:51.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T17:35:51.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T17:35:51.957 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T17:35:51.957 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T17:35:51.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-25T17:35:51.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T17:35:51.958 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T17:35:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T17:35:52.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T17:35:52.037 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T17:35:52.037 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /tmp/cephfs 2024-04-25T17:35:52.052 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/tmp/cephfs' 2024-04-25T17:35:52.052 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T17:35:52.053 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /tmp/cephfs 2024-04-25T17:35:52.107 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-25T17:35:52.107 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-25T17:35:52.107 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/8b073cd4-d70f-47d4-966e-81c60974204b 2024-04-25T17:35:52.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:35:52.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/8b073cd4-d70f-47d4-966e-81c60974204b /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,context=system_u:object_r:postgresql_db_t:s0,nofallback 2024-04-25T17:35:52.168 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:51.856+0000 7f683274d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T17:35:52.226 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-04-25T17:35:52.226 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-25T17:35:52.329 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-04-25T17:35:52.329 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-25T17:35:52.329 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.173:3300,172.21.15.203:3300" 2024-04-25T17:35:52.329 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 0@79754014-0328-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/8b073cd4-d70f-47d4-966e-81c60974204b 2024-04-25T17:35:52.329 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0",name=0,key=0,mon_addr=172.21.15.98:3300/172.21.15.173:3300/172.21.15.203:3300" will pass to kernel 2024-04-25T17:35:52.332 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T17:35:52.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:35:52.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /tmp/cephfs 2024-04-25T17:35:52.401 DEBUG:teuthology.orchestra.run.smithi098:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T17:35:52.455 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:52.416+0000 7f683274d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T17:35:52.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:52.203+0000 7f8558d07200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T17:35:52.463 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:52.298+0000 7f8558d07200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T17:35:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T17:35:52.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T17:35:52.498 INFO:teuthology.orchestra.run.smithi098.stdout:client24803 2024-04-25T17:35:52.499 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:35:52.499 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T17:35:52.512 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:52.171+0000 7f850d23d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T17:35:52.512 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:52.265+0000 7f850d23d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T17:35:52.556 DEBUG:teuthology.orchestra.run.smithi098:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T17:35:52.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T17:35:52.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T17:35:52.653 INFO:teuthology.orchestra.run.smithi098.stdout:client24803 2024-04-25T17:35:52.654 DEBUG:teuthology.orchestra.run.smithi098:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T17:35:52.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T17:35:52.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T17:35:52.710 INFO:teuthology.orchestra.run.smithi098.stdout:client24803 2024-04-25T17:35:52.711 DEBUG:teuthology.orchestra.run.smithi098:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/79754014-0328-11ef-bc93-c7b262605968.client24803/status') 2024-04-25T17:35:52.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T17:35:52.767 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:52.516+0000 7f683274d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T17:35:52.768 INFO:teuthology.orchestra.run.smithi098.stdout:instance: client.24803 (3)172.21.15.98:0/1014226601 2024-04-25T17:35:52.768 INFO:teuthology.orchestra.run.smithi098.stdout:blocklisted: false 2024-04-25T17:35:52.769 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records in 2024-04-25T17:35:52.769 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records out 2024-04-25T17:35:52.769 INFO:teuthology.orchestra.run.smithi098.stderr:71 bytes copied, 2.2041e-05 s, 3.2 MB/s 2024-04-25T17:35:52.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T17:35:52.772 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:52.511+0000 7f850d23d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T17:35:52.772 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:52.645+0000 7f850d23d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T17:35:52.772 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:52.740+0000 7f850d23d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T17:35:52.776 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T17:35:52.793 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T17:35:52.793 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:35:52.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-25T17:35:52.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.895+0000 7ff23b605640 1 -- 172.21.15.98:0/899748299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff234102040 msgr2=0x7ff2341024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:52.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.895+0000 7ff23b605640 1 --2- 172.21.15.98:0/899748299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff234102040 0x7ff2341024c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff230009900 tx=0x7ff23002a410 comp rx=0 tx=0).stop 2024-04-25T17:35:52.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.895+0000 7ff23b605640 1 -- 172.21.15.98:0/899748299 shutdown_connections 2024-04-25T17:35:52.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.895+0000 7ff23b605640 1 --2- 172.21.15.98:0/899748299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff234102a00 0x7ff234109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:52.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.895+0000 7ff23b605640 1 --2- 172.21.15.98:0/899748299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff234102040 0x7ff2341024c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:52.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.895+0000 7ff23b605640 1 --2- 172.21.15.98:0/899748299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff234100e60 0x7ff234101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:52.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.895+0000 7ff23b605640 1 -- 172.21.15.98:0/899748299 >> 172.21.15.98:0/899748299 conn(0x7ff2340fc420 msgr2=0x7ff2340fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:52.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.895+0000 7ff23b605640 1 -- 172.21.15.98:0/899748299 shutdown_connections 2024-04-25T17:35:52.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.895+0000 7ff23b605640 1 -- 172.21.15.98:0/899748299 wait complete. 2024-04-25T17:35:52.897 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:52.547+0000 7f8558d07200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T17:35:52.897 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:52.681+0000 7f8558d07200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T17:35:52.897 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:52.778+0000 7f8558d07200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T17:35:52.897 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:52.895+0000 7f8558d07200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T17:35:52.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.896+0000 7ff23b605640 1 Processor -- start 2024-04-25T17:35:52.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.896+0000 7ff23b605640 1 -- start start 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.896+0000 7ff23b605640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff234100e60 0x7ff234197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.896+0000 7ff23b605640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff234102040 0x7ff2341975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.896+0000 7ff23b605640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff234102a00 0x7ff23419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.896+0000 7ff23b605640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff23410c9f0 con 0x7ff234102a00 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.896+0000 7ff23b605640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff23410c870 con 0x7ff234102040 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.896+0000 7ff23b605640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff23410cb70 con 0x7ff234100e60 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.896+0000 7ff239ba2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff234102a00 0x7ff23419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff239ba2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff234102a00 0x7ff23419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46558/0 (socket says 172.21.15.98:46558) 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff2393a1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff234100e60 0x7ff234197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff239ba2640 1 -- 172.21.15.98:0/3922800133 learned_addr learned my addr 172.21.15.98:0/3922800133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff238ba0640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff234102040 0x7ff2341975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff239ba2640 1 -- 172.21.15.98:0/3922800133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff234100e60 msgr2=0x7ff234197090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff239ba2640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff234100e60 0x7ff234197090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff239ba2640 1 -- 172.21.15.98:0/3922800133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff234102040 msgr2=0x7ff2341975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:52.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff239ba2640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff234102040 0x7ff2341975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:52.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff239ba2640 1 -- 172.21.15.98:0/3922800133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff23419ecb0 con 0x7ff234102a00 2024-04-25T17:35:52.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff239ba2640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff234102a00 0x7ff23419e640 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009fd0 tx=0x7ff22c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:52.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff22a7fc640 1 -- 172.21.15.98:0/3922800133 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c017070 con 0x7ff234102a00 2024-04-25T17:35:52.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff22a7fc640 1 -- 172.21.15.98:0/3922800133 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff22c003e20 con 0x7ff234102a00 2024-04-25T17:35:52.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff23b605640 1 -- 172.21.15.98:0/3922800133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff23419efa0 con 0x7ff234102a00 2024-04-25T17:35:52.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff22a7fc640 1 -- 172.21.15.98:0/3922800133 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c010460 con 0x7ff234102a00 2024-04-25T17:35:52.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.897+0000 7ff23b605640 1 -- 172.21.15.98:0/3922800133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff23419f410 con 0x7ff234102a00 2024-04-25T17:35:52.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.898+0000 7ff23b605640 1 -- 172.21.15.98:0/3922800133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff234108250 con 0x7ff234102a00 2024-04-25T17:35:52.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.899+0000 7ff22a7fc640 1 -- 172.21.15.98:0/3922800133 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7ff22c003970 con 0x7ff234102a00 2024-04-25T17:35:52.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.900+0000 7ff22a7fc640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff2200a3ca0 0x7ff2200a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:52.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.900+0000 7ff2393a1640 1 -- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff2200a3ca0 msgr2=0x7ff2200a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4145487482 2024-04-25T17:35:52.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.900+0000 7ff22a7fc640 1 -- 172.21.15.98:0/3922800133 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7ff22c0d0120 con 0x7ff234102a00 2024-04-25T17:35:52.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.900+0000 7ff2393a1640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff2200a3ca0 0x7ff2200a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:35:52.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:52.908+0000 7ff22a7fc640 1 -- 172.21.15.98:0/3922800133 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff22c09c4c0 con 0x7ff234102a00 2024-04-25T17:35:53.026 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:52.777+0000 7f683274d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T17:35:53.026 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:52.925+0000 7f683274d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T17:35:53.026 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:53.025+0000 7f683274d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T17:35:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.101+0000 7ff2393a1640 1 -- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff2200a3ca0 msgr2=0x7ff2200a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4145487482 2024-04-25T17:35:53.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.101+0000 7ff2393a1640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff2200a3ca0 0x7ff2200a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T17:35:53.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.132+0000 7ff23b605640 1 -- 172.21.15.98:0/3922800133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7ff23402d660 con 0x7ff234102a00 2024-04-25T17:35:53.163 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:52.857+0000 7f850d23d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T17:35:53.163 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:53.162+0000 7f850d23d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T17:35:53.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.184+0000 7ff22a7fc640 1 -- 172.21.15.98:0/3922800133 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v26) ==== 117+0+0 (secure 0 0 0) 0x7ff22c0a07c0 con 0x7ff234102a00 2024-04-25T17:35:53.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.186+0000 7ff23b605640 1 -- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff2200a3ca0 msgr2=0x7ff2200a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.186+0000 7ff23b605640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff2200a3ca0 0x7ff2200a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.186+0000 7ff23b605640 1 -- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff234102a00 msgr2=0x7ff23419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.186+0000 7ff23b605640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff234102a00 0x7ff23419e640 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009fd0 tx=0x7ff22c00b260 comp rx=0 tx=0).stop 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.187+0000 7ff23b605640 1 -- 172.21.15.98:0/3922800133 shutdown_connections 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.187+0000 7ff23b605640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7ff2200a3ca0 0x7ff2200a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.187+0000 7ff23b605640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff234102a00 0x7ff23419e640 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.187+0000 7ff23b605640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff234102040 0x7ff2341975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.187+0000 7ff23b605640 1 --2- 172.21.15.98:0/3922800133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff234100e60 0x7ff234197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.187+0000 7ff23b605640 1 -- 172.21.15.98:0/3922800133 >> 172.21.15.98:0/3922800133 conn(0x7ff2340fc420 msgr2=0x7ff234105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.187+0000 7ff23b605640 1 -- 172.21.15.98:0/3922800133 shutdown_connections 2024-04-25T17:35:53.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.187+0000 7ff23b605640 1 -- 172.21.15.98:0/3922800133 wait complete. 2024-04-25T17:35:53.199 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T17:35:53.211 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T17:35:53.222 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T17:35:53.222 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:35:53.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-25T17:35:53.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.326+0000 7f0a9f14b640 1 -- 172.21.15.98:0/3584441981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a98074930 msgr2=0x7f0a98074d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:53.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.326+0000 7f0a9f14b640 1 --2- 172.21.15.98:0/3584441981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a98074930 0x7f0a98074d90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0a84009900 tx=0x7f0a8402a480 comp rx=0 tx=0).stop 2024-04-25T17:35:53.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.326+0000 7f0a9f14b640 1 -- 172.21.15.98:0/3584441981 shutdown_connections 2024-04-25T17:35:53.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.326+0000 7f0a9f14b640 1 --2- 172.21.15.98:0/3584441981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a98109e80 0x7f0a9810c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.326+0000 7f0a9f14b640 1 --2- 172.21.15.98:0/3584441981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a98074930 0x7f0a98074d90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.326+0000 7f0a9f14b640 1 --2- 172.21.15.98:0/3584441981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a98075f10 0x7f0a98074360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.326+0000 7f0a9f14b640 1 -- 172.21.15.98:0/3584441981 >> 172.21.15.98:0/3584441981 conn(0x7f0a980fd540 msgr2=0x7f0a980ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.326+0000 7f0a9f14b640 1 -- 172.21.15.98:0/3584441981 shutdown_connections 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.326+0000 7f0a9f14b640 1 -- 172.21.15.98:0/3584441981 wait complete. 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9f14b640 1 Processor -- start 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9f14b640 1 -- start start 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9f14b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a98074930 0x7f0a9819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9f14b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a98075f10 0x7f0a9819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9f14b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a98109e80 0x7f0a981a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9f14b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9810fd70 con 0x7f0a98109e80 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9f14b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9810fbf0 con 0x7f0a98075f10 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9f14b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9810fef0 con 0x7f0a98074930 2024-04-25T17:35:53.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9d6e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a98109e80 0x7f0a981a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9d6e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a98109e80 0x7f0a981a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46578/0 (socket says 172.21.15.98:46578) 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a97fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a98075f10 0x7f0a9819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9d6e8640 1 -- 172.21.15.98:0/1451896959 learned_addr learned my addr 172.21.15.98:0/1451896959 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9cee7640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a98074930 0x7f0a9819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9d6e8640 1 -- 172.21.15.98:0/1451896959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a98074930 msgr2=0x7f0a9819a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9d6e8640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a98074930 0x7f0a9819a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9d6e8640 1 -- 172.21.15.98:0/1451896959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a98075f10 msgr2=0x7f0a9819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9d6e8640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a98075f10 0x7f0a9819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9d6e8640 1 -- 172.21.15.98:0/1451896959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a981a1f70 con 0x7f0a98109e80 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a97fff640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a98075f10 0x7f0a9819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.327+0000 7f0a9d6e8640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a98109e80 0x7f0a981a1900 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000eb10 tx=0x7f0a9000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.328+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a90004b20 con 0x7f0a98109e80 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.328+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a90004cc0 con 0x7f0a98109e80 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.328+0000 7f0a9f14b640 1 -- 172.21.15.98:0/1451896959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a981a2260 con 0x7f0a98109e80 2024-04-25T17:35:53.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.328+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a90010670 con 0x7f0a98109e80 2024-04-25T17:35:53.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.328+0000 7f0a9f14b640 1 -- 172.21.15.98:0/1451896959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a981a2730 con 0x7f0a98109e80 2024-04-25T17:35:53.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.329+0000 7f0a9f14b640 1 -- 172.21.15.98:0/1451896959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a98108c40 con 0x7f0a98109e80 2024-04-25T17:35:53.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.330+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f0a90005020 con 0x7f0a98109e80 2024-04-25T17:35:53.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.331+0000 7f0a95ffb640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a880a3ca0 0x7f0a880a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:53.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.331+0000 7f0a9cee7640 1 -- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a880a3ca0 msgr2=0x7f0a880a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4145487482 2024-04-25T17:35:53.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.331+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8211+0+0 (secure 0 0 0) 0x7f0a900d0a00 con 0x7f0a98109e80 2024-04-25T17:35:53.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.331+0000 7f0a9cee7640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a880a3ca0 0x7f0a880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T17:35:53.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.336+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a9009d3f0 con 0x7f0a98109e80 2024-04-25T17:35:53.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:53 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3922800133' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T17:35:53.456 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:53.148+0000 7f683274d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T17:35:53.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:53 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3922800133' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T17:35:53.462 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:53.200+0000 7f8558d07200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T17:35:53.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:53 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3922800133' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T17:35:53.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.532+0000 7f0a9cee7640 1 -- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a880a3ca0 msgr2=0x7f0a880a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4145487482 2024-04-25T17:35:53.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.532+0000 7f0a9cee7640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a880a3ca0 0x7f0a880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T17:35:53.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.545+0000 7f0a9f14b640 1 -- 172.21.15.98:0/1451896959 --> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f0a98062390 con 0x7f0a880a3ca0 2024-04-25T17:35:53.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.933+0000 7f0a9cee7640 1 -- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a880a3ca0 msgr2=0x7f0a880a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4145487482 2024-04-25T17:35:53.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:53.933+0000 7f0a9cee7640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a880a3ca0 0x7f0a880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T17:35:53.955 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:53.462+0000 7f683274d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T17:35:54.022 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:53.763+0000 7f850d23d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T17:35:54.023 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:35:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y[34281]: 2024-04-25T17:35:53.976+0000 7f850d23d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T17:35:54.193 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:53.811+0000 7f8558d07200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T17:35:54.193 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z[34609]: 2024-04-25T17:35:54.029+0000 7f8558d07200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T17:35:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:54.205+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7f0a90099360 con 0x7f0a98109e80 2024-04-25T17:35:54.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:54.312+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f0a90005310 con 0x7f0a98109e80 2024-04-25T17:35:54.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:54.313+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a880a3ca0 msgr2=0x7f0a880a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:35:54.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:54.313+0000 7f0a95ffb640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/4145487482,v1:172.21.15.98:6801/4145487482] conn(0x7f0a880a3ca0 0x7f0a880a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:54.440 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.074+0000 7f683274d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T17:35:54.440 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.300+0000 7f683274d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T17:35:54.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3922800133' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T17:35:54.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:35:54.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: Standby manager daemon y restarted 2024-04-25T17:35:54.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: Standby manager daemon y started 2024-04-25T17:35:54.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T17:35:54.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:35:54.440 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T17:35:54.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:35:54.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: Standby manager daemon z restarted 2024-04-25T17:35:54.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: Standby manager daemon z started 2024-04-25T17:35:54.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T17:35:54.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:35:54.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T17:35:54.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-mon[29192]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:35:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3922800133' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T17:35:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:35:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: Standby manager daemon y restarted 2024-04-25T17:35:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: Standby manager daemon y started 2024-04-25T17:35:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T17:35:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:35:54.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T17:35:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:35:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: Standby manager daemon z restarted 2024-04-25T17:35:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: Standby manager daemon z started 2024-04-25T17:35:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T17:35:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:35:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T17:35:54.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:54 smithi203 ceph-mon[32834]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:35:54.508 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3922800133' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: Standby manager daemon y restarted 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: Standby manager daemon y started 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.173:0/1143314928' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: Standby manager daemon z restarted 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: Standby manager daemon z started 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T17:35:54.509 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:54 smithi173 ceph-mon[32856]: from='mgr.? 172.21.15.203:0/492624338' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T17:35:54.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.537+0000 7f67940c5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:35:54.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.537+0000 7f67940c5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:35:54.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.537+0000 7f67940c5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:35:54.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.537+0000 7f67940c5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:35:54.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.537+0000 7f67940c5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:35:54.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.537+0000 7f67920c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:35:54.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.537+0000 7f67920c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:35:54.706 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.537+0000 7f67920c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:35:54.707 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.537+0000 7f67920c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:35:54.707 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:35:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:35:54.537+0000 7f67920c1700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:35:55.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.320+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 <== mon.0 v2:172.21.15.98:3300/0 9 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f0a90078250 con 0x7f0a98109e80 2024-04-25T17:35:55.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.321+0000 7f0a95ffb640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0a88035660 0x7f0a88035a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:55.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.321+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 --> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f0a9009c3a0 con 0x7f0a88035660 2024-04-25T17:35:55.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.325+0000 7f0a9cee7640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0a88035660 0x7f0a88035a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:55.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.326+0000 7f0a9cee7640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0a88035660 0x7f0a88035a40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00b6e0 tx=0x7f0a8c00b390 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.383+0000 7f0a95ffb640 1 -- 172.21.15.98:0/1451896959 <== mgr.15129 v2:172.21.15.98:6800/844767972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0a9009c3a0 con 0x7f0a88035660 2024-04-25T17:35:55.385 INFO:teuthology.orchestra.run.smithi098.stdout:Schedule set for path / 2024-04-25T17:35:55.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.385+0000 7f0a9f14b640 1 -- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0a88035660 msgr2=0x7f0a88035a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:55.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.385+0000 7f0a9f14b640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0a88035660 0x7f0a88035a40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00b6e0 tx=0x7f0a8c00b390 comp rx=0 tx=0).stop 2024-04-25T17:35:55.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.385+0000 7f0a9f14b640 1 -- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a98109e80 msgr2=0x7f0a981a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:55.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.385+0000 7f0a9f14b640 1 --2- 172.21.15.98:0/1451896959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a98109e80 0x7f0a981a1900 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000eb10 tx=0x7f0a9000c2d0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.385+0000 7f0a9d6e8640 1 -- 172.21.15.98:0/1451896959 reap_dead start 2024-04-25T17:35:55.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.385+0000 7f0a9f14b640 1 -- 172.21.15.98:0/1451896959 shutdown_connections 2024-04-25T17:35:55.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.385+0000 7f0a9f14b640 1 -- 172.21.15.98:0/1451896959 >> 172.21.15.98:0/1451896959 conn(0x7f0a980fd540 msgr2=0x7f0a98105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:55.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.386+0000 7f0a9f14b640 1 -- 172.21.15.98:0/1451896959 shutdown_connections 2024-04-25T17:35:55.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.386+0000 7f0a9f14b640 1 -- 172.21.15.98:0/1451896959 wait complete. 2024-04-25T17:35:55.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-25T17:35:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T17:35:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: Active manager daemon x restarted 2024-04-25T17:35:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: Activating manager daemon x 2024-04-25T17:35:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T17:35:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: mgrmap e21: x(active, starting, since 0.0114683s), standbys: z, y 2024-04-25T17:35:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:35:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:35:55.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T17:35:55.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T17:35:55.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: Manager daemon x is now available 2024-04-25T17:35:55.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:55.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T17:35:55.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:55 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T17:35:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T17:35:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: Active manager daemon x restarted 2024-04-25T17:35:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: Activating manager daemon x 2024-04-25T17:35:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T17:35:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: mgrmap e21: x(active, starting, since 0.0114683s), standbys: z, y 2024-04-25T17:35:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:35:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:35:55.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T17:35:55.464 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T17:35:55.465 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T17:35:55.466 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: Manager daemon x is now available 2024-04-25T17:35:55.466 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:55.466 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T17:35:55.466 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:55 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T17:35:55.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.500+0000 7f202b660640 1 -- 172.21.15.98:0/1845577591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2024101fb0 msgr2=0x7f20241023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:55.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.500+0000 7f202b660640 1 --2- 172.21.15.98:0/1845577591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2024101fb0 0x7f20241023d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2020009900 tx=0x7f202002a410 comp rx=0 tx=0).stop 2024-04-25T17:35:55.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.500+0000 7f202b660640 1 -- 172.21.15.98:0/1845577591 shutdown_connections 2024-04-25T17:35:55.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.500+0000 7f202b660640 1 --2- 172.21.15.98:0/1845577591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2024103b60 0x7f202410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.500+0000 7f202b660640 1 --2- 172.21.15.98:0/1845577591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f20241031a0 0x7f2024103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.500+0000 7f202b660640 1 --2- 172.21.15.98:0/1845577591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2024101fb0 0x7f20241023d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.500+0000 7f202b660640 1 -- 172.21.15.98:0/1845577591 >> 172.21.15.98:0/1845577591 conn(0x7f20240fd540 msgr2=0x7f20240ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.500+0000 7f202b660640 1 -- 172.21.15.98:0/1845577591 shutdown_connections 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.500+0000 7f202b660640 1 -- 172.21.15.98:0/1845577591 wait complete. 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f202b660640 1 Processor -- start 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f202b660640 1 -- start start 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f202b660640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2024101fb0 0x7f2024198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f202b660640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20241031a0 0x7f20241985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f202b660640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2024103b60 0x7f202419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f202b660640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f202410daf0 con 0x7f2024101fb0 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f202b660640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f202410d970 con 0x7f2024103b60 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f202b660640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f202410dc70 con 0x7f20241031a0 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f20293fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2024101fb0 0x7f2024198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f20293fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2024101fb0 0x7f2024198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45386/0 (socket says 172.21.15.98:45386) 2024-04-25T17:35:55.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f20293fc640 1 -- 172.21.15.98:0/1610304205 learned_addr learned my addr 172.21.15.98:0/1610304205 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f2029bfd640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2024103b60 0x7f202419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f20293fc640 1 -- 172.21.15.98:0/1610304205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20241031a0 msgr2=0x7f20241985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f20293fc640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20241031a0 0x7f20241985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f20293fc640 1 -- 172.21.15.98:0/1610304205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2024103b60 msgr2=0x7f202419f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f20293fc640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2024103b60 0x7f202419f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f20293fc640 1 -- 172.21.15.98:0/1610304205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202419fcb0 con 0x7f2024101fb0 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.501+0000 7f20293fc640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2024101fb0 0x7f2024198090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2020006010 tx=0x7f20200042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.502+0000 7f201a7fc640 1 -- 172.21.15.98:0/1610304205 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202003f030 con 0x7f2024101fb0 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.502+0000 7f201a7fc640 1 -- 172.21.15.98:0/1610304205 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20200318f0 con 0x7f2024101fb0 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.502+0000 7f202b660640 1 -- 172.21.15.98:0/1610304205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202419ff40 con 0x7f2024101fb0 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.502+0000 7f201a7fc640 1 -- 172.21.15.98:0/1610304205 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020035070 con 0x7f2024101fb0 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.502+0000 7f202b660640 1 -- 172.21.15.98:0/1610304205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20241a03a0 con 0x7f2024101fb0 2024-04-25T17:35:55.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.503+0000 7f202b660640 1 -- 172.21.15.98:0/1610304205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2024109270 con 0x7f2024101fb0 2024-04-25T17:35:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.504+0000 7f201a7fc640 1 -- 172.21.15.98:0/1610304205 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f2020042030 con 0x7f2024101fb0 2024-04-25T17:35:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.504+0000 7f201a7fc640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f20100a3a60 0x7f20100a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.504+0000 7f2028bfb640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f20100a3a60 0x7f20100a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:55.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.505+0000 7f201a7fc640 1 -- 172.21.15.98:0/1610304205 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f20200f0b00 con 0x7f2024101fb0 2024-04-25T17:35:55.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.505+0000 7f2028bfb640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f20100a3a60 0x7f20100a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f20241995a0 tx=0x7f2014005c70 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:55.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.508+0000 7f201a7fc640 1 -- 172.21.15.98:0/1610304205 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20200bd6e0 con 0x7f2024101fb0 2024-04-25T17:35:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T17:35:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: Active manager daemon x restarted 2024-04-25T17:35:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: Activating manager daemon x 2024-04-25T17:35:55.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T17:35:55.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: mgrmap e21: x(active, starting, since 0.0114683s), standbys: z, y 2024-04-25T17:35:55.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T17:35:55.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T17:35:55.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T17:35:55.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T17:35:55.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T17:35:55.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T17:35:55.525 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T17:35:55.525 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T17:35:55.525 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T17:35:55.525 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T17:35:55.525 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T17:35:55.525 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:35:55.526 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T17:35:55.526 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T17:35:55.526 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T17:35:55.526 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T17:35:55.526 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T17:35:55.526 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T17:35:55.527 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T17:35:55.527 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T17:35:55.527 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T17:35:55.527 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T17:35:55.527 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T17:35:55.527 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T17:35:55.527 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T17:35:55.528 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T17:35:55.528 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T17:35:55.528 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T17:35:55.528 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T17:35:55.528 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T17:35:55.528 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T17:35:55.529 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T17:35:55.529 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T17:35:55.529 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: Manager daemon x is now available 2024-04-25T17:35:55.529 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:55.529 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T17:35:55.529 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:55 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T17:35:55.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.718+0000 7f202b660640 1 -- 172.21.15.98:0/1610304205 --> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f20240624f0 con 0x7f20100a3a60 2024-04-25T17:35:55.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.772+0000 7f201a7fc640 1 -- 172.21.15.98:0/1610304205 <== mgr.15129 v2:172.21.15.98:6800/844767972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f20240624f0 con 0x7f20100a3a60 2024-04-25T17:35:55.774 INFO:teuthology.orchestra.run.smithi098.stdout:Retention added to path / 2024-04-25T17:35:55.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 -- 172.21.15.98:0/1610304205 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f20100a3a60 msgr2=0x7f20100a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f20100a3a60 0x7f20100a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f20241995a0 tx=0x7f2014005c70 comp rx=0 tx=0).stop 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 -- 172.21.15.98:0/1610304205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2024101fb0 msgr2=0x7f2024198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2024101fb0 0x7f2024198090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2020006010 tx=0x7f20200042f0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 -- 172.21.15.98:0/1610304205 shutdown_connections 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f20100a3a60 0x7f20100a5f20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2024103b60 0x7f202419f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20241031a0 0x7f20241985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 --2- 172.21.15.98:0/1610304205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2024101fb0 0x7f2024198090 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 -- 172.21.15.98:0/1610304205 >> 172.21.15.98:0/1610304205 conn(0x7f20240fd540 msgr2=0x7f2024106440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 -- 172.21.15.98:0/1610304205 shutdown_connections 2024-04-25T17:35:55.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.774+0000 7f202b660640 1 -- 172.21.15.98:0/1610304205 wait complete. 2024-04-25T17:35:55.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.887+0000 7f04aeffe640 1 -- 172.21.15.98:0/4270110343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04a8102a90 msgr2=0x7f04a8104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.887+0000 7f04aeffe640 1 --2- 172.21.15.98:0/4270110343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04a8102a90 0x7f04a8104e80 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009900 tx=0x7f04a402a410 comp rx=0 tx=0).stop 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.887+0000 7f04aeffe640 1 -- 172.21.15.98:0/4270110343 shutdown_connections 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.887+0000 7f04aeffe640 1 --2- 172.21.15.98:0/4270110343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a8107cf0 0x7f04a810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 --2- 172.21.15.98:0/4270110343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a81053c0 0x7f04a81077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 --2- 172.21.15.98:0/4270110343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04a8102a90 0x7f04a8104e80 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 -- 172.21.15.98:0/4270110343 >> 172.21.15.98:0/4270110343 conn(0x7f04a80fc420 msgr2=0x7f04a80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 -- 172.21.15.98:0/4270110343 shutdown_connections 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 -- 172.21.15.98:0/4270110343 wait complete. 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 Processor -- start 2024-04-25T17:35:55.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 -- start start 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a8102a90 0x7f04a81981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a81053c0 0x7f04a81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04a8107cf0 0x7f04a819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04a810daf0 con 0x7f04a8107cf0 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f04a810d970 con 0x7f04a8102a90 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04aeffe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f04a810dc70 con 0x7f04a81053c0 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04ad59b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04a8107cf0 0x7f04a819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04ad59b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04a8107cf0 0x7f04a819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45408/0 (socket says 172.21.15.98:45408) 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04ad59b640 1 -- 172.21.15.98:0/3485585966 learned_addr learned my addr 172.21.15.98:0/3485585966 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f049ffff640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a81053c0 0x7f04a81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.888+0000 7f04acd9a640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a8102a90 0x7f04a81981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f04ad59b640 1 -- 172.21.15.98:0/3485585966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a81053c0 msgr2=0x7f04a81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f04ad59b640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a81053c0 0x7f04a81986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f04ad59b640 1 -- 172.21.15.98:0/3485585966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a8102a90 msgr2=0x7f04a81981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:55.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f04ad59b640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a8102a90 0x7f04a81981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:55.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f04ad59b640 1 -- 172.21.15.98:0/3485585966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a819fdc0 con 0x7f04a8107cf0 2024-04-25T17:35:55.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f04ad59b640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04a8107cf0 0x7f04a819f750 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f04a000eb10 tx=0x7f04a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:55.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f049dffb640 1 -- 172.21.15.98:0/3485585966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0004b20 con 0x7f04a8107cf0 2024-04-25T17:35:55.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f049dffb640 1 -- 172.21.15.98:0/3485585966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04a0004cc0 con 0x7f04a8107cf0 2024-04-25T17:35:55.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f04aeffe640 1 -- 172.21.15.98:0/3485585966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a81a00b0 con 0x7f04a8107cf0 2024-04-25T17:35:55.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f049dffb640 1 -- 172.21.15.98:0/3485585966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0010670 con 0x7f04a8107cf0 2024-04-25T17:35:55.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.889+0000 7f04aeffe640 1 -- 172.21.15.98:0/3485585966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a81a0650 con 0x7f04a8107cf0 2024-04-25T17:35:55.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.890+0000 7f04aeffe640 1 -- 172.21.15.98:0/3485585966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a80fdf10 con 0x7f04a8107cf0 2024-04-25T17:35:55.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.891+0000 7f049dffb640 1 -- 172.21.15.98:0/3485585966 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f04a0005020 con 0x7f04a8107cf0 2024-04-25T17:35:55.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.891+0000 7f049dffb640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f04900a3ae0 0x7f04900a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:55.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.892+0000 7f04acd9a640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f04900a3ae0 0x7f04900a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:55.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.892+0000 7f049dffb640 1 -- 172.21.15.98:0/3485585966 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f04a00d0640 con 0x7f04a8107cf0 2024-04-25T17:35:55.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.892+0000 7f04acd9a640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f04900a3ae0 0x7f04900a5fa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f04a4000c00 tx=0x7f04a40314a0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:55.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:55.896+0000 7f049dffb640 1 -- 172.21.15.98:0/3485585966 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04a009d220 con 0x7f04a8107cf0 2024-04-25T17:35:56.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.103+0000 7f04aeffe640 1 -- 172.21.15.98:0/3485585966 --> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f04a8062390 con 0x7f04900a3ae0 2024-04-25T17:35:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.111+0000 7f049dffb640 1 -- 172.21.15.98:0/3485585966 <== mgr.15129 v2:172.21.15.98:6800/844767972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f04a8062390 con 0x7f04900a3ae0 2024-04-25T17:35:56.113 INFO:teuthology.orchestra.run.smithi098.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T17:35:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-25T17:35:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 -- 172.21.15.98:0/3485585966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f04900a3ae0 msgr2=0x7f04900a5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f04900a3ae0 0x7f04900a5fa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f04a4000c00 tx=0x7f04a40314a0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 -- 172.21.15.98:0/3485585966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04a8107cf0 msgr2=0x7f04a819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04a8107cf0 0x7f04a819f750 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f04a000eb10 tx=0x7f04a000c2d0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 -- 172.21.15.98:0/3485585966 shutdown_connections 2024-04-25T17:35:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f04900a3ae0 0x7f04900a5fa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04a8107cf0 0x7f04a819f750 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04a81053c0 0x7f04a81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 --2- 172.21.15.98:0/3485585966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04a8102a90 0x7f04a81981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 -- 172.21.15.98:0/3485585966 >> 172.21.15.98:0/3485585966 conn(0x7f04a80fc420 msgr2=0x7f04a8103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 -- 172.21.15.98:0/3485585966 shutdown_connections 2024-04-25T17:35:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.113+0000 7f04aeffe640 1 -- 172.21.15.98:0/3485585966 wait complete. 2024-04-25T17:35:56.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T17:35:56.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.235+0000 7f99db16f640 1 -- 172.21.15.98:0/39355367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4100e60 msgr2=0x7f99d4101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:56.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.235+0000 7f99db16f640 1 --2- 172.21.15.98:0/39355367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4100e60 0x7f99d4101280 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009900 tx=0x7f99c802a410 comp rx=0 tx=0).stop 2024-04-25T17:35:56.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.235+0000 7f99db16f640 1 -- 172.21.15.98:0/39355367 shutdown_connections 2024-04-25T17:35:56.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.235+0000 7f99db16f640 1 --2- 172.21.15.98:0/39355367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99d4102a00 0x7f99d4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.235+0000 7f99db16f640 1 --2- 172.21.15.98:0/39355367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4102040 0x7f99d41024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.235+0000 7f99db16f640 1 --2- 172.21.15.98:0/39355367 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4100e60 0x7f99d4101280 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.235+0000 7f99db16f640 1 -- 172.21.15.98:0/39355367 >> 172.21.15.98:0/39355367 conn(0x7f99d40fc420 msgr2=0x7f99d40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.235+0000 7f99db16f640 1 -- 172.21.15.98:0/39355367 shutdown_connections 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.235+0000 7f99db16f640 1 -- 172.21.15.98:0/39355367 wait complete. 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99db16f640 1 Processor -- start 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99db16f640 1 -- start start 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99db16f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4100e60 0x7f99d406a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99db16f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4102040 0x7f99d419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99db16f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99d4102a00 0x7f99d419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99db16f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f99d410c9f0 con 0x7f99d4102040 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99db16f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f99d410c870 con 0x7f99d4100e60 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99db16f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f99d410cb70 con 0x7f99d4102a00 2024-04-25T17:35:56.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99d3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4102040 0x7f99d419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99d3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4102040 0x7f99d419cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45422/0 (socket says 172.21.15.98:45422) 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99d8f0b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4100e60 0x7f99d406a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99d3fff640 1 -- 172.21.15.98:0/2686347725 learned_addr learned my addr 172.21.15.98:0/2686347725 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99d970c640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99d4102a00 0x7f99d419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99d3fff640 1 -- 172.21.15.98:0/2686347725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99d4102a00 msgr2=0x7f99d419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99d3fff640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99d4102a00 0x7f99d419f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99d3fff640 1 -- 172.21.15.98:0/2686347725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4100e60 msgr2=0x7f99d406a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99d3fff640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4100e60 0x7f99d406a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.236+0000 7f99d3fff640 1 -- 172.21.15.98:0/2686347725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d419fdc0 con 0x7f99d4102040 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.237+0000 7f99d3fff640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4102040 0x7f99d419cf30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f99c000d950 tx=0x7f99c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.237+0000 7f99d1ffb640 1 -- 172.21.15.98:0/2686347725 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0017030 con 0x7f99d4102040 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.237+0000 7f99d1ffb640 1 -- 172.21.15.98:0/2686347725 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99c0004020 con 0x7f99d4102040 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.237+0000 7f99db16f640 1 -- 172.21.15.98:0/2686347725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d41a00b0 con 0x7f99d4102040 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.237+0000 7f99d1ffb640 1 -- 172.21.15.98:0/2686347725 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c00047a0 con 0x7f99d4102040 2024-04-25T17:35:56.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.237+0000 7f99db16f640 1 -- 172.21.15.98:0/2686347725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d41a0600 con 0x7f99d4102040 2024-04-25T17:35:56.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.238+0000 7f99db16f640 1 -- 172.21.15.98:0/2686347725 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d41082d0 con 0x7f99d4102040 2024-04-25T17:35:56.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.239+0000 7f99d1ffb640 1 -- 172.21.15.98:0/2686347725 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f99c0009a30 con 0x7f99d4102040 2024-04-25T17:35:56.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.240+0000 7f99d1ffb640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f99c40a3a60 0x7f99c40a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:56.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.240+0000 7f99d8f0b640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f99c40a3a60 0x7f99c40a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:56.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.240+0000 7f99d1ffb640 1 -- 172.21.15.98:0/2686347725 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f99c00cfb00 con 0x7f99d4102040 2024-04-25T17:35:56.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.240+0000 7f99d8f0b640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f99c40a3a60 0x7f99c40a5f20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009230 tx=0x7f99c80314a0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:56.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.245+0000 7f99d1ffb640 1 -- 172.21.15.98:0/2686347725 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99c009c6e0 con 0x7f99d4102040 2024-04-25T17:35:56.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:56 smithi098 ceph-mon[29192]: [25/Apr/2024:17:35:54] ENGINE Bus STARTING 2024-04-25T17:35:56.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:56 smithi098 ceph-mon[29192]: [25/Apr/2024:17:35:55] ENGINE Serving on http://172.21.15.98:8765 2024-04-25T17:35:56.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:56 smithi098 ceph-mon[29192]: [25/Apr/2024:17:35:55] ENGINE Serving on https://172.21.15.98:7150 2024-04-25T17:35:56.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:56 smithi098 ceph-mon[29192]: [25/Apr/2024:17:35:55] ENGINE Bus STARTED 2024-04-25T17:35:56.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:56 smithi098 ceph-mon[29192]: mgrmap e22: x(active, since 1.02094s), standbys: z, y 2024-04-25T17:35:56.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:56 smithi203 ceph-mon[32834]: [25/Apr/2024:17:35:54] ENGINE Bus STARTING 2024-04-25T17:35:56.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:56 smithi203 ceph-mon[32834]: [25/Apr/2024:17:35:55] ENGINE Serving on http://172.21.15.98:8765 2024-04-25T17:35:56.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:56 smithi203 ceph-mon[32834]: [25/Apr/2024:17:35:55] ENGINE Serving on https://172.21.15.98:7150 2024-04-25T17:35:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:56 smithi203 ceph-mon[32834]: [25/Apr/2024:17:35:55] ENGINE Bus STARTED 2024-04-25T17:35:56.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:56 smithi203 ceph-mon[32834]: mgrmap e22: x(active, since 1.02094s), standbys: z, y 2024-04-25T17:35:56.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.474+0000 7f99db16f640 1 -- 172.21.15.98:0/2686347725 --> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f99d4062390 con 0x7f99c40a3a60 2024-04-25T17:35:56.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.481+0000 7f99d1ffb640 1 -- 172.21.15.98:0/2686347725 <== mgr.15129 v2:172.21.15.98:6800/844767972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f99d4062390 con 0x7f99c40a3a60 2024-04-25T17:35:56.483 INFO:teuthology.orchestra.run.smithi098.stdout:/ 1m 6m3h 2024-04-25T17:35:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 -- 172.21.15.98:0/2686347725 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f99c40a3a60 msgr2=0x7f99c40a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f99c40a3a60 0x7f99c40a5f20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009230 tx=0x7f99c80314a0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 -- 172.21.15.98:0/2686347725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4102040 msgr2=0x7f99d419cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4102040 0x7f99d419cf30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f99c000d950 tx=0x7f99c000de10 comp rx=0 tx=0).stop 2024-04-25T17:35:56.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 -- 172.21.15.98:0/2686347725 shutdown_connections 2024-04-25T17:35:56.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f99c40a3a60 0x7f99c40a5f20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99d4102a00 0x7f99d419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99d4102040 0x7f99d419cf30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 --2- 172.21.15.98:0/2686347725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99d4100e60 0x7f99d406a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:56.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 -- 172.21.15.98:0/2686347725 >> 172.21.15.98:0/2686347725 conn(0x7f99d40fc420 msgr2=0x7f99d41054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:56.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 -- 172.21.15.98:0/2686347725 shutdown_connections 2024-04-25T17:35:56.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:56.484+0000 7f99db16f640 1 -- 172.21.15.98:0/2686347725 wait complete. 2024-04-25T17:35:56.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-25T17:35:56.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:56 smithi173 ceph-mon[32856]: [25/Apr/2024:17:35:54] ENGINE Bus STARTING 2024-04-25T17:35:56.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:56 smithi173 ceph-mon[32856]: [25/Apr/2024:17:35:55] ENGINE Serving on http://172.21.15.98:8765 2024-04-25T17:35:56.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:56 smithi173 ceph-mon[32856]: [25/Apr/2024:17:35:55] ENGINE Serving on https://172.21.15.98:7150 2024-04-25T17:35:56.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:56 smithi173 ceph-mon[32856]: [25/Apr/2024:17:35:55] ENGINE Bus STARTED 2024-04-25T17:35:56.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:56 smithi173 ceph-mon[32856]: mgrmap e22: x(active, since 1.02094s), standbys: z, y 2024-04-25T17:35:56.524 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-25T17:35:56.535 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-25T17:35:56.552 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:35:56.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-25T17:35:56.567 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-25T17:35:56.578 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T17:35:56.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T17:35:56.822 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:35:57.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:57 smithi173 ceph-mon[32856]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:57.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:57 smithi173 ceph-mon[32856]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:35:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:57 smithi173 ceph-mon[32856]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:57 smithi173 ceph-mon[32856]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:57 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:57 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:57 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:57 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:57 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:57 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:57 smithi098 ceph-mon[29192]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:57.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:57 smithi098 ceph-mon[29192]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:35:57.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:57 smithi098 ceph-mon[29192]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:57.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:57 smithi098 ceph-mon[29192]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:57.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:57 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:57 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:57 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:57 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:57 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:57 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:57 smithi203 ceph-mon[32834]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:57.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:57 smithi203 ceph-mon[32834]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:35:57.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:57 smithi203 ceph-mon[32834]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:57.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:57 smithi203 ceph-mon[32834]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:57.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:57 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:57 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:57 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:57 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:57 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:57.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:57 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:35:58.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:58.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T17:35:58.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:58.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:58 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:58.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:58.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:58 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.677+0000 7fc378b27700 1 -- 172.21.15.98:0/308816349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741023a0 msgr2=0x7fc374102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:58.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.677+0000 7fc378b27700 1 --2- 172.21.15.98:0/308816349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741023a0 0x7fc374102780 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc3640099c0 tx=0x7fc364009cd0 comp rx=0 tx=0).stop 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.677+0000 7fc378b27700 1 -- 172.21.15.98:0/308816349 shutdown_connections 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.678+0000 7fc378b27700 1 --2- 172.21.15.98:0/308816349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc374106cc0 0x7fc37410e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.678+0000 7fc378b27700 1 --2- 172.21.15.98:0/308816349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741023a0 0x7fc374102780 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.678+0000 7fc378b27700 1 --2- 172.21.15.98:0/308816349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc374102d50 0x7fc3741065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.678+0000 7fc378b27700 1 -- 172.21.15.98:0/308816349 >> 172.21.15.98:0/308816349 conn(0x7fc374077fb0 msgr2=0x7fc3740ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.678+0000 7fc378b27700 1 -- 172.21.15.98:0/308816349 shutdown_connections 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.678+0000 7fc378b27700 1 -- 172.21.15.98:0/308816349 wait complete. 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.678+0000 7fc378b27700 1 Processor -- start 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.678+0000 7fc378b27700 1 -- start start 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc378b27700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741023a0 0x7fc3741991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc378b27700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc374102d50 0x7fc374199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:58.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc378b27700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc374106cc0 0x7fc37419da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc378b27700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc374111bb0 con 0x7fc3741023a0 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc378b27700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc374111a30 con 0x7fc374106cc0 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc378b27700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc374111d30 con 0x7fc374102d50 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc371d9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc374102d50 0x7fc374199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc37259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741023a0 0x7fc3741991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc371d9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc374102d50 0x7fc374199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:47032/0 (socket says 172.21.15.98:47032) 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc371d9b700 1 -- 172.21.15.98:0/230236482 learned_addr learned my addr 172.21.15.98:0/230236482 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc372d9d700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc374106cc0 0x7fc37419da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc371d9b700 1 -- 172.21.15.98:0/230236482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc374106cc0 msgr2=0x7fc37419da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc371d9b700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc374106cc0 0x7fc37419da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc371d9b700 1 -- 172.21.15.98:0/230236482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741023a0 msgr2=0x7fc3741991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc371d9b700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741023a0 0x7fc3741991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc371d9b700 1 -- 172.21.15.98:0/230236482 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc37419e180 con 0x7fc374102d50 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc372d9d700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc374106cc0 0x7fc37419da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:35:58.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.679+0000 7fc371d9b700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc374102d50 0x7fc374199700 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e970 tx=0x7fc36800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.680+0000 7fc3637fe700 1 -- 172.21.15.98:0/230236482 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36800cbd0 con 0x7fc374102d50 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.680+0000 7fc378b27700 1 -- 172.21.15.98:0/230236482 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc37419e470 con 0x7fc374102d50 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.680+0000 7fc3637fe700 1 -- 172.21.15.98:0/230236482 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc36800cd70 con 0x7fc374102d50 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.680+0000 7fc3637fe700 1 -- 172.21.15.98:0/230236482 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc368005680 con 0x7fc374102d50 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.680+0000 7fc378b27700 1 -- 172.21.15.98:0/230236482 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3740fd350 con 0x7fc374102d50 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.681+0000 7fc378b27700 1 -- 172.21.15.98:0/230236482 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc374103150 con 0x7fc374102d50 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.682+0000 7fc3637fe700 1 -- 172.21.15.98:0/230236482 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7fc368004750 con 0x7fc374102d50 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.682+0000 7fc3637fe700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc35c0a3db0 0x7fc35c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.683+0000 7fc37259c700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc35c0a3db0 0x7fc35c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.683+0000 7fc3637fe700 1 -- 172.21.15.98:0/230236482 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fc3680d0b80 con 0x7fc374102d50 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.683+0000 7fc37259c700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc35c0a3db0 0x7fc35c0a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc364003760 tx=0x7fc364005040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:35:58.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.686+0000 7fc3637fe700 1 -- 172.21.15.98:0/230236482 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc36809d1f0 con 0x7fc374102d50 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:58.715 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:58 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:58.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.902+0000 7fc378b27700 1 -- 172.21.15.98:0/230236482 --> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc374105c00 con 0x7fc35c0a3db0 2024-04-25T17:35:58.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.912+0000 7fc3637fe700 1 -- 172.21.15.98:0/230236482 <== mgr.15129 v2:172.21.15.98:6800/844767972 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258002 (secure 0 0 0) 0x7fc374105c00 con 0x7fc35c0a3db0 2024-04-25T17:35:58.918 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:35:58.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 -- 172.21.15.98:0/230236482 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc35c0a3db0 msgr2=0x7fc35c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc35c0a3db0 0x7fc35c0a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc364003760 tx=0x7fc364005040 comp rx=0 tx=0).stop 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 -- 172.21.15.98:0/230236482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc374102d50 msgr2=0x7fc374199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc374102d50 0x7fc374199700 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e970 tx=0x7fc36800ec80 comp rx=0 tx=0).stop 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 -- 172.21.15.98:0/230236482 shutdown_connections 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc35c0a3db0 0x7fc35c0a6270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc374106cc0 0x7fc37419da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3741023a0 0x7fc3741991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 --2- 172.21.15.98:0/230236482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc374102d50 0x7fc374199700 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 -- 172.21.15.98:0/230236482 >> 172.21.15.98:0/230236482 conn(0x7fc374077fb0 msgr2=0x7fc3741012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 -- 172.21.15.98:0/230236482 shutdown_connections 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:35:58.915+0000 7fc378b27700 1 -- 172.21.15.98:0/230236482 wait complete. 2024-04-25T17:35:58.920 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T17:35:59.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Unable to set osd_memory_target on smithi173 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi203 to 176.2M 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Unable to set osd_memory_target on smithi203 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:59.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:59.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:59.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:35:59 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T17:35:59.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Unable to set osd_memory_target on smithi173 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-04-25T17:35:59.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi203 to 176.2M 2024-04-25T17:35:59.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Unable to set osd_memory_target on smithi203 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-25T17:35:59.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T17:35:59.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T17:35:59.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T17:35:59.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T17:35:59.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T17:35:59.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:59.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:35:59 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.656 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T17:35:58.319984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188638,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1334,"num_read_kb":827,"num_write":480,"num_write_kb":4381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357404,"kb_used_data":12372,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584420,"statfs":{"total":1151940427776,"available":1151574446080,"internally_reserved":0,"allocated":12668928,"data_stored":9066753,"data_compressed":198489,"data_compressed_allocated":258048,"data_compressed_original":3022848,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991076"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549366+0000","last_change":"2024-04-25T17:32:35.075338+0000","last_active":"2024-04-25T17:35:54.549366+0000","last_peered":"2024-04-25T17:35:54.549366+0000","last_clean":"2024-04-25T17:35:54.549366+0000","last_became_active":"2024-04-25T17:32:35.075029+0000","last_became_peered":"2024-04-25T17:32:35.075029+0000","last_unstale":"2024-04-25T17:35:54.549366+0000","last_undegraded":"2024-04-25T17:35:54.549366+0000","last_fullsized":"2024-04-25T17:35:54.549366+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:23.837337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543054+0000","last_change":"2024-04-25T17:32:35.075422+0000","last_active":"2024-04-25T17:35:54.543054+0000","last_peered":"2024-04-25T17:35:54.543054+0000","last_clean":"2024-04-25T17:35:54.543054+0000","last_became_active":"2024-04-25T17:32:35.071975+0000","last_became_peered":"2024-04-25T17:32:35.071975+0000","last_unstale":"2024-04-25T17:35:54.543054+0000","last_undegraded":"2024-04-25T17:35:54.543054+0000","last_fullsized":"2024-04-25T17:35:54.543054+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:20.548869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542762+0000","last_change":"2024-04-25T17:32:35.068153+0000","last_active":"2024-04-25T17:35:54.542762+0000","last_peered":"2024-04-25T17:35:54.542762+0000","last_clean":"2024-04-25T17:35:54.542762+0000","last_became_active":"2024-04-25T17:32:35.065873+0000","last_became_peered":"2024-04-25T17:32:35.065873+0000","last_unstale":"2024-04-25T17:35:54.542762+0000","last_undegraded":"2024-04-25T17:35:54.542762+0000","last_fullsized":"2024-04-25T17:35:54.542762+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:31.381875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310699+0000","last_change":"2024-04-25T17:32:35.073055+0000","last_active":"2024-04-25T17:35:54.310699+0000","last_peered":"2024-04-25T17:35:54.310699+0000","last_clean":"2024-04-25T17:35:54.310699+0000","last_became_active":"2024-04-25T17:32:35.072596+0000","last_became_peered":"2024-04-25T17:32:35.072596+0000","last_unstale":"2024-04-25T17:35:54.310699+0000","last_undegraded":"2024-04-25T17:35:54.310699+0000","last_fullsized":"2024-04-25T17:35:54.310699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:37.529698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309402+0000","last_change":"2024-04-25T17:32:35.072721+0000","last_active":"2024-04-25T17:35:54.309402+0000","last_peered":"2024-04-25T17:35:54.309402+0000","last_clean":"2024-04-25T17:35:54.309402+0000","last_became_active":"2024-04-25T17:32:35.072370+0000","last_became_peered":"2024-04-25T17:32:35.072370+0000","last_unstale":"2024-04-25T17:35:54.309402+0000","last_undegraded":"2024-04-25T17:35:54.309402+0000","last_fullsized":"2024-04-25T17:35:54.309402+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:34.429985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350742+0000","last_change":"2024-04-25T17:32:35.071538+0000","last_active":"2024-04-25T17:35:54.350742+0000","last_peered":"2024-04-25T17:35:54.350742+0000","last_clean":"2024-04-25T17:35:54.350742+0000","last_became_active":"2024-04-25T17:32:35.067427+0000","last_became_peered":"2024-04-25T17:32:35.067427+0000","last_unstale":"2024-04-25T17:35:54.350742+0000","last_undegraded":"2024-04-25T17:35:54.350742+0000","last_fullsized":"2024-04-25T17:35:54.350742+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:36.052762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542299+0000","last_change":"2024-04-25T17:32:35.079476+0000","last_active":"2024-04-25T17:35:54.542299+0000","last_peered":"2024-04-25T17:35:54.542299+0000","last_clean":"2024-04-25T17:35:54.542299+0000","last_became_active":"2024-04-25T17:32:35.079170+0000","last_became_peered":"2024-04-25T17:32:35.079170+0000","last_unstale":"2024-04-25T17:35:54.542299+0000","last_undegraded":"2024-04-25T17:35:54.542299+0000","last_fullsized":"2024-04-25T17:35:54.542299+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:30.937670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395435+0000","last_change":"2024-04-25T17:32:35.077261+0000","last_active":"2024-04-25T17:35:54.395435+0000","last_peered":"2024-04-25T17:35:54.395435+0000","last_clean":"2024-04-25T17:35:54.395435+0000","last_became_active":"2024-04-25T17:32:35.076896+0000","last_became_peered":"2024-04-25T17:32:35.076896+0000","last_unstale":"2024-04-25T17:35:54.395435+0000","last_undegraded":"2024-04-25T17:35:54.395435+0000","last_fullsized":"2024-04-25T17:35:54.395435+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:24.082550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352775+0000","last_change":"2024-04-25T17:32:35.072229+0000","last_active":"2024-04-25T17:35:54.352775+0000","last_peered":"2024-04-25T17:35:54.352775+0000","last_clean":"2024-04-25T17:35:54.352775+0000","last_became_active":"2024-04-25T17:32:35.071586+0000","last_became_peered":"2024-04-25T17:32:35.071586+0000","last_unstale":"2024-04-25T17:35:54.352775+0000","last_undegraded":"2024-04-25T17:35:54.352775+0000","last_fullsized":"2024-04-25T17:35:54.352775+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:01.195638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.351018+0000","last_change":"2024-04-25T17:32:35.075274+0000","last_active":"2024-04-25T17:35:54.351018+0000","last_peered":"2024-04-25T17:35:54.351018+0000","last_clean":"2024-04-25T17:35:54.351018+0000","last_became_active":"2024-04-25T17:32:35.073821+0000","last_became_peered":"2024-04-25T17:32:35.073821+0000","last_unstale":"2024-04-25T17:35:54.351018+0000","last_undegraded":"2024-04-25T17:35:54.351018+0000","last_fullsized":"2024-04-25T17:35:54.351018+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:43:04.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542610+0000","last_change":"2024-04-25T17:32:35.080747+0000","last_active":"2024-04-25T17:35:54.542610+0000","last_peered":"2024-04-25T17:35:54.542610+0000","last_clean":"2024-04-25T17:35:54.542610+0000","last_became_active":"2024-04-25T17:32:35.080383+0000","last_became_peered":"2024-04-25T17:32:35.080383+0000","last_unstale":"2024-04-25T17:35:54.542610+0000","last_undegraded":"2024-04-25T17:35:54.542610+0000","last_fullsized":"2024-04-25T17:35:54.542610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:42.128892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310582+0000","last_change":"2024-04-25T17:32:35.061442+0000","last_active":"2024-04-25T17:35:54.310582+0000","last_peered":"2024-04-25T17:35:54.310582+0000","last_clean":"2024-04-25T17:35:54.310582+0000","last_became_active":"2024-04-25T17:32:35.061030+0000","last_became_peered":"2024-04-25T17:32:35.061030+0000","last_unstale":"2024-04-25T17:35:54.310582+0000","last_undegraded":"2024-04-25T17:35:54.310582+0000","last_fullsized":"2024-04-25T17:35:54.310582+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:03.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416138+0000","last_change":"2024-04-25T17:32:35.069698+0000","last_active":"2024-04-25T17:35:54.416138+0000","last_peered":"2024-04-25T17:35:54.416138+0000","last_clean":"2024-04-25T17:35:54.416138+0000","last_became_active":"2024-04-25T17:32:35.069374+0000","last_became_peered":"2024-04-25T17:32:35.069374+0000","last_unstale":"2024-04-25T17:35:54.416138+0000","last_undegraded":"2024-04-25T17:35:54.416138+0000","last_fullsized":"2024-04-25T17:35:54.416138+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:03.214525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542531+0000","last_change":"2024-04-25T17:32:35.069001+0000","last_active":"2024-04-25T17:35:54.542531+0000","last_peered":"2024-04-25T17:35:54.542531+0000","last_clean":"2024-04-25T17:35:54.542531+0000","last_became_active":"2024-04-25T17:32:35.064882+0000","last_became_peered":"2024-04-25T17:32:35.064882+0000","last_unstale":"2024-04-25T17:35:54.542531+0000","last_undegraded":"2024-04-25T17:35:54.542531+0000","last_fullsized":"2024-04-25T17:35:54.542531+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:32.122251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542441+0000","last_change":"2024-04-25T17:32:35.065042+0000","last_active":"2024-04-25T17:35:54.542441+0000","last_peered":"2024-04-25T17:35:54.542441+0000","last_clean":"2024-04-25T17:35:54.542441+0000","last_became_active":"2024-04-25T17:32:35.064719+0000","last_became_peered":"2024-04-25T17:32:35.064719+0000","last_unstale":"2024-04-25T17:35:54.542441+0000","last_undegraded":"2024-04-25T17:35:54.542441+0000","last_fullsized":"2024-04-25T17:35:54.542441+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:54.715385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352288+0000","last_change":"2024-04-25T17:32:35.067390+0000","last_active":"2024-04-25T17:35:54.352288+0000","last_peered":"2024-04-25T17:35:54.352288+0000","last_clean":"2024-04-25T17:35:54.352288+0000","last_became_active":"2024-04-25T17:32:35.066846+0000","last_became_peered":"2024-04-25T17:32:35.066846+0000","last_unstale":"2024-04-25T17:35:54.352288+0000","last_undegraded":"2024-04-25T17:35:54.352288+0000","last_fullsized":"2024-04-25T17:35:54.352288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:24.305767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549475+0000","last_change":"2024-04-25T17:32:35.075349+0000","last_active":"2024-04-25T17:35:54.549475+0000","last_peered":"2024-04-25T17:35:54.549475+0000","last_clean":"2024-04-25T17:35:54.549475+0000","last_became_active":"2024-04-25T17:32:35.075046+0000","last_became_peered":"2024-04-25T17:32:35.075046+0000","last_unstale":"2024-04-25T17:35:54.549475+0000","last_undegraded":"2024-04-25T17:35:54.549475+0000","last_fullsized":"2024-04-25T17:35:54.549475+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:51.703406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307381+0000","last_change":"2024-04-25T17:32:35.077597+0000","last_active":"2024-04-25T17:35:54.307381+0000","last_peered":"2024-04-25T17:35:54.307381+0000","last_clean":"2024-04-25T17:35:54.307381+0000","last_became_active":"2024-04-25T17:32:35.077232+0000","last_became_peered":"2024-04-25T17:32:35.077232+0000","last_unstale":"2024-04-25T17:35:54.307381+0000","last_undegraded":"2024-04-25T17:35:54.307381+0000","last_fullsized":"2024-04-25T17:35:54.307381+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:46.842673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352862+0000","last_change":"2024-04-25T17:32:35.078229+0000","last_active":"2024-04-25T17:35:54.352862+0000","last_peered":"2024-04-25T17:35:54.352862+0000","last_clean":"2024-04-25T17:35:54.352862+0000","last_became_active":"2024-04-25T17:32:35.077899+0000","last_became_peered":"2024-04-25T17:32:35.077899+0000","last_unstale":"2024-04-25T17:35:54.352862+0000","last_undegraded":"2024-04-25T17:35:54.352862+0000","last_fullsized":"2024-04-25T17:35:54.352862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:22.776371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307342+0000","last_change":"2024-04-25T17:32:32.045385+0000","last_active":"2024-04-25T17:35:54.307342+0000","last_peered":"2024-04-25T17:35:54.307342+0000","last_clean":"2024-04-25T17:35:54.307342+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:35:54.307342+0000","last_undegraded":"2024-04-25T17:35:54.307342+0000","last_fullsized":"2024-04-25T17:35:54.307342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:29.732001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395800+0000","last_change":"2024-04-25T17:32:35.079966+0000","last_active":"2024-04-25T17:35:54.395800+0000","last_peered":"2024-04-25T17:35:54.395800+0000","last_clean":"2024-04-25T17:35:54.395800+0000","last_became_active":"2024-04-25T17:32:35.079649+0000","last_became_peered":"2024-04-25T17:32:35.079649+0000","last_unstale":"2024-04-25T17:35:54.395800+0000","last_undegraded":"2024-04-25T17:35:54.395800+0000","last_fullsized":"2024-04-25T17:35:54.395800+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:19:45.679273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.396029+0000","last_change":"2024-04-25T17:32:32.285667+0000","last_active":"2024-04-25T17:35:54.396029+0000","last_peered":"2024-04-25T17:35:54.396029+0000","last_clean":"2024-04-25T17:35:54.396029+0000","last_became_active":"2024-04-25T17:32:32.285330+0000","last_became_peered":"2024-04-25T17:32:32.285330+0000","last_unstale":"2024-04-25T17:35:54.396029+0000","last_undegraded":"2024-04-25T17:35:54.396029+0000","last_fullsized":"2024-04-25T17:35:54.396029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.975280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543368+0000","last_change":"2024-04-25T17:32:35.068841+0000","last_active":"2024-04-25T17:35:54.543368+0000","last_peered":"2024-04-25T17:35:54.543368+0000","last_clean":"2024-04-25T17:35:54.543368+0000","last_became_active":"2024-04-25T17:32:35.068518+0000","last_became_peered":"2024-04-25T17:32:35.068518+0000","last_unstale":"2024-04-25T17:35:54.543368+0000","last_undegraded":"2024-04-25T17:35:54.543368+0000","last_fullsized":"2024-04-25T17:35:54.543368+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:23.290990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309712+0000","last_change":"2024-04-25T17:32:32.045627+0000","last_active":"2024-04-25T17:35:54.309712+0000","last_peered":"2024-04-25T17:35:54.309712+0000","last_clean":"2024-04-25T17:35:54.309712+0000","last_became_active":"2024-04-25T17:32:32.044971+0000","last_became_peered":"2024-04-25T17:32:32.044971+0000","last_unstale":"2024-04-25T17:35:54.309712+0000","last_undegraded":"2024-04-25T17:35:54.309712+0000","last_fullsized":"2024-04-25T17:35:54.309712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:50.590435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352609+0000","last_change":"2024-04-25T17:32:35.070862+0000","last_active":"2024-04-25T17:35:54.352609+0000","last_peered":"2024-04-25T17:35:54.352609+0000","last_clean":"2024-04-25T17:35:54.352609+0000","last_became_active":"2024-04-25T17:32:35.069927+0000","last_became_peered":"2024-04-25T17:32:35.069927+0000","last_unstale":"2024-04-25T17:35:54.352609+0000","last_undegraded":"2024-04-25T17:35:54.352609+0000","last_fullsized":"2024-04-25T17:35:54.352609+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:40:01.799015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309802+0000","last_change":"2024-04-25T17:32:32.046062+0000","last_active":"2024-04-25T17:35:54.309802+0000","last_peered":"2024-04-25T17:35:54.309802+0000","last_clean":"2024-04-25T17:35:54.309802+0000","last_became_active":"2024-04-25T17:32:32.044903+0000","last_became_peered":"2024-04-25T17:32:32.044903+0000","last_unstale":"2024-04-25T17:35:54.309802+0000","last_undegraded":"2024-04-25T17:35:54.309802+0000","last_fullsized":"2024-04-25T17:35:54.309802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:51.099460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402607+0000","last_change":"2024-04-25T17:32:35.075452+0000","last_active":"2024-04-25T17:35:54.402607+0000","last_peered":"2024-04-25T17:35:54.402607+0000","last_clean":"2024-04-25T17:35:54.402607+0000","last_became_active":"2024-04-25T17:32:35.075136+0000","last_became_peered":"2024-04-25T17:32:35.075136+0000","last_unstale":"2024-04-25T17:35:54.402607+0000","last_undegraded":"2024-04-25T17:35:54.402607+0000","last_fullsized":"2024-04-25T17:35:54.402607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:02.290926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":218,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415513+0000","last_change":"2024-04-25T17:32:32.042019+0000","last_active":"2024-04-25T17:35:54.415513+0000","last_peered":"2024-04-25T17:35:54.415513+0000","last_clean":"2024-04-25T17:35:54.415513+0000","last_became_active":"2024-04-25T17:32:32.041641+0000","last_became_peered":"2024-04-25T17:32:32.041641+0000","last_unstale":"2024-04-25T17:35:54.415513+0000","last_undegraded":"2024-04-25T17:35:54.415513+0000","last_fullsized":"2024-04-25T17:35:54.415513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:02.025915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309350+0000","last_change":"2024-04-25T17:32:35.069715+0000","last_active":"2024-04-25T17:35:54.309350+0000","last_peered":"2024-04-25T17:35:54.309350+0000","last_clean":"2024-04-25T17:35:54.309350+0000","last_became_active":"2024-04-25T17:32:35.069339+0000","last_became_peered":"2024-04-25T17:32:35.069339+0000","last_unstale":"2024-04-25T17:35:54.309350+0000","last_undegraded":"2024-04-25T17:35:54.309350+0000","last_fullsized":"2024-04-25T17:35:54.309350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:48:13.572261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350662+0000","last_change":"2024-04-25T17:32:32.042445+0000","last_active":"2024-04-25T17:35:54.350662+0000","last_peered":"2024-04-25T17:35:54.350662+0000","last_clean":"2024-04-25T17:35:54.350662+0000","last_became_active":"2024-04-25T17:32:32.042033+0000","last_became_peered":"2024-04-25T17:32:32.042033+0000","last_unstale":"2024-04-25T17:35:54.350662+0000","last_undegraded":"2024-04-25T17:35:54.350662+0000","last_fullsized":"2024-04-25T17:35:54.350662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.968383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402497+0000","last_change":"2024-04-25T17:32:35.074492+0000","last_active":"2024-04-25T17:35:54.402497+0000","last_peered":"2024-04-25T17:35:54.402497+0000","last_clean":"2024-04-25T17:35:54.402497+0000","last_became_active":"2024-04-25T17:32:35.073652+0000","last_became_peered":"2024-04-25T17:32:35.073652+0000","last_unstale":"2024-04-25T17:35:54.402497+0000","last_undegraded":"2024-04-25T17:35:54.402497+0000","last_fullsized":"2024-04-25T17:35:54.402497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:00.300453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415757+0000","last_change":"2024-04-25T17:32:32.039568+0000","last_active":"2024-04-25T17:35:54.415757+0000","last_peered":"2024-04-25T17:35:54.415757+0000","last_clean":"2024-04-25T17:35:54.415757+0000","last_became_active":"2024-04-25T17:32:32.039228+0000","last_became_peered":"2024-04-25T17:32:32.039228+0000","last_unstale":"2024-04-25T17:35:54.415757+0000","last_undegraded":"2024-04-25T17:35:54.415757+0000","last_fullsized":"2024-04-25T17:35:54.415757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:17:29.079749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395508+0000","last_change":"2024-04-25T17:32:35.077619+0000","last_active":"2024-04-25T17:35:54.395508+0000","last_peered":"2024-04-25T17:35:54.395508+0000","last_clean":"2024-04-25T17:35:54.395508+0000","last_became_active":"2024-04-25T17:32:35.077309+0000","last_became_peered":"2024-04-25T17:32:35.077309+0000","last_unstale":"2024-04-25T17:35:54.395508+0000","last_undegraded":"2024-04-25T17:35:54.395508+0000","last_fullsized":"2024-04-25T17:35:54.395508+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:41.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402343+0000","last_change":"2024-04-25T17:32:32.044650+0000","last_active":"2024-04-25T17:35:54.402343+0000","last_peered":"2024-04-25T17:35:54.402343+0000","last_clean":"2024-04-25T17:35:54.402343+0000","last_became_active":"2024-04-25T17:32:32.043999+0000","last_became_peered":"2024-04-25T17:32:32.043999+0000","last_unstale":"2024-04-25T17:35:54.402343+0000","last_undegraded":"2024-04-25T17:35:54.402343+0000","last_fullsized":"2024-04-25T17:35:54.402343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:19.463941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307201+0000","last_change":"2024-04-25T17:32:35.080598+0000","last_active":"2024-04-25T17:35:54.307201+0000","last_peered":"2024-04-25T17:35:54.307201+0000","last_clean":"2024-04-25T17:35:54.307201+0000","last_became_active":"2024-04-25T17:32:35.080300+0000","last_became_peered":"2024-04-25T17:32:35.080300+0000","last_unstale":"2024-04-25T17:35:54.307201+0000","last_undegraded":"2024-04-25T17:35:54.307201+0000","last_fullsized":"2024-04-25T17:35:54.307201+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:42.149746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":212,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350925+0000","last_change":"2024-04-25T17:32:32.043889+0000","last_active":"2024-04-25T17:35:54.350925+0000","last_peered":"2024-04-25T17:35:54.350925+0000","last_clean":"2024-04-25T17:35:54.350925+0000","last_became_active":"2024-04-25T17:32:32.043520+0000","last_became_peered":"2024-04-25T17:32:32.043520+0000","last_unstale":"2024-04-25T17:35:54.350925+0000","last_undegraded":"2024-04-25T17:35:54.350925+0000","last_fullsized":"2024-04-25T17:35:54.350925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:25.134797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":179,"num_read_kb":179,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352641+0000","last_change":"2024-04-25T17:32:35.071733+0000","last_active":"2024-04-25T17:35:54.352641+0000","last_peered":"2024-04-25T17:35:54.352641+0000","last_clean":"2024-04-25T17:35:54.352641+0000","last_became_active":"2024-04-25T17:32:35.071383+0000","last_became_peered":"2024-04-25T17:32:35.071383+0000","last_unstale":"2024-04-25T17:35:54.352641+0000","last_undegraded":"2024-04-25T17:35:54.352641+0000","last_fullsized":"2024-04-25T17:35:54.352641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:09:49.433675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542701+0000","last_change":"2024-04-25T17:32:32.044823+0000","last_active":"2024-04-25T17:35:54.542701+0000","last_peered":"2024-04-25T17:35:54.542701+0000","last_clean":"2024-04-25T17:35:54.542701+0000","last_became_active":"2024-04-25T17:32:32.044497+0000","last_became_peered":"2024-04-25T17:32:32.044497+0000","last_unstale":"2024-04-25T17:35:54.542701+0000","last_undegraded":"2024-04-25T17:35:54.542701+0000","last_fullsized":"2024-04-25T17:35:54.542701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:47.811172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543284+0000","last_change":"2024-04-25T17:32:35.071109+0000","last_active":"2024-04-25T17:35:54.543284+0000","last_peered":"2024-04-25T17:35:54.543284+0000","last_clean":"2024-04-25T17:35:54.543284+0000","last_became_active":"2024-04-25T17:32:35.070782+0000","last_became_peered":"2024-04-25T17:32:35.070782+0000","last_unstale":"2024-04-25T17:35:54.543284+0000","last_undegraded":"2024-04-25T17:35:54.543284+0000","last_fullsized":"2024-04-25T17:35:54.543284+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:37.247274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402726+0000","last_change":"2024-04-25T17:32:32.288683+0000","last_active":"2024-04-25T17:35:54.402726+0000","last_peered":"2024-04-25T17:35:54.402726+0000","last_clean":"2024-04-25T17:35:54.402726+0000","last_became_active":"2024-04-25T17:32:32.288261+0000","last_became_peered":"2024-04-25T17:32:32.288261+0000","last_unstale":"2024-04-25T17:35:54.402726+0000","last_undegraded":"2024-04-25T17:35:54.402726+0000","last_fullsized":"2024-04-25T17:35:54.402726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:36.502434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352607+0000","last_change":"2024-04-25T17:32:35.060828+0000","last_active":"2024-04-25T17:35:54.352607+0000","last_peered":"2024-04-25T17:35:54.352607+0000","last_clean":"2024-04-25T17:35:54.352607+0000","last_became_active":"2024-04-25T17:32:35.060433+0000","last_became_peered":"2024-04-25T17:32:35.060433+0000","last_unstale":"2024-04-25T17:35:54.352607+0000","last_undegraded":"2024-04-25T17:35:54.352607+0000","last_fullsized":"2024-04-25T17:35:54.352607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:32.382042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":436,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415711+0000","last_change":"2024-04-25T17:32:32.042979+0000","last_active":"2024-04-25T17:35:54.415711+0000","last_peered":"2024-04-25T17:35:54.415711+0000","last_clean":"2024-04-25T17:35:54.415711+0000","last_became_active":"2024-04-25T17:32:32.042257+0000","last_became_peered":"2024-04-25T17:32:32.042257+0000","last_unstale":"2024-04-25T17:35:54.415711+0000","last_undegraded":"2024-04-25T17:35:54.415711+0000","last_fullsized":"2024-04-25T17:35:54.415711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:19.316803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67597,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":371,"num_read_kb":252,"num_write":37,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309526+0000","last_change":"2024-04-25T17:32:35.069994+0000","last_active":"2024-04-25T17:35:54.309526+0000","last_peered":"2024-04-25T17:35:54.309526+0000","last_clean":"2024-04-25T17:35:54.309526+0000","last_became_active":"2024-04-25T17:32:35.069632+0000","last_became_peered":"2024-04-25T17:32:35.069632+0000","last_unstale":"2024-04-25T17:35:54.309526+0000","last_undegraded":"2024-04-25T17:35:54.309526+0000","last_fullsized":"2024-04-25T17:35:54.309526+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:51.169489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352625+0000","last_change":"2024-04-25T17:32:32.042652+0000","last_active":"2024-04-25T17:35:54.352625+0000","last_peered":"2024-04-25T17:35:54.352625+0000","last_clean":"2024-04-25T17:35:54.352625+0000","last_became_active":"2024-04-25T17:32:32.042214+0000","last_became_peered":"2024-04-25T17:32:32.042214+0000","last_unstale":"2024-04-25T17:35:54.352625+0000","last_undegraded":"2024-04-25T17:35:54.352625+0000","last_fullsized":"2024-04-25T17:35:54.352625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.662580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415731+0000","last_change":"2024-04-25T17:32:35.074449+0000","last_active":"2024-04-25T17:35:54.415731+0000","last_peered":"2024-04-25T17:35:54.415731+0000","last_clean":"2024-04-25T17:35:54.415731+0000","last_became_active":"2024-04-25T17:32:35.074116+0000","last_became_peered":"2024-04-25T17:32:35.074116+0000","last_unstale":"2024-04-25T17:35:54.415731+0000","last_undegraded":"2024-04-25T17:35:54.415731+0000","last_fullsized":"2024-04-25T17:35:54.415731+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:09.233206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402590+0000","last_change":"2024-04-25T17:32:32.041368+0000","last_active":"2024-04-25T17:35:54.402590+0000","last_peered":"2024-04-25T17:35:54.402590+0000","last_clean":"2024-04-25T17:35:54.402590+0000","last_became_active":"2024-04-25T17:32:32.040940+0000","last_became_peered":"2024-04-25T17:32:32.040940+0000","last_unstale":"2024-04-25T17:35:54.402590+0000","last_undegraded":"2024-04-25T17:35:54.402590+0000","last_fullsized":"2024-04-25T17:35:54.402590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:38.871786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542869+0000","last_change":"2024-04-25T17:32:35.073535+0000","last_active":"2024-04-25T17:35:54.542869+0000","last_peered":"2024-04-25T17:35:54.542869+0000","last_clean":"2024-04-25T17:35:54.542869+0000","last_became_active":"2024-04-25T17:32:35.073190+0000","last_became_peered":"2024-04-25T17:32:35.073190+0000","last_unstale":"2024-04-25T17:35:54.542869+0000","last_undegraded":"2024-04-25T17:35:54.542869+0000","last_fullsized":"2024-04-25T17:35:54.542869+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:19.841978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'7","reported_seq":219,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542253+0000","last_change":"2024-04-25T17:32:32.044359+0000","last_active":"2024-04-25T17:35:54.542253+0000","last_peered":"2024-04-25T17:35:54.542253+0000","last_clean":"2024-04-25T17:35:54.542253+0000","last_became_active":"2024-04-25T17:32:32.044024+0000","last_became_peered":"2024-04-25T17:32:32.044024+0000","last_unstale":"2024-04-25T17:35:54.542253+0000","last_undegraded":"2024-04-25T17:35:54.542253+0000","last_fullsized":"2024-04-25T17:35:54.542253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:58.940554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":184,"num_read_kb":5,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542252+0000","last_change":"2024-04-25T17:33:10.851665+0000","last_active":"2024-04-25T17:35:54.542252+0000","last_peered":"2024-04-25T17:35:54.542252+0000","last_clean":"2024-04-25T17:35:54.542252+0000","last_became_active":"2024-04-25T17:33:10.851217+0000","last_became_peered":"2024-04-25T17:33:10.851217+0000","last_unstale":"2024-04-25T17:35:54.542252+0000","last_undegraded":"2024-04-25T17:35:54.542252+0000","last_fullsized":"2024-04-25T17:35:54.542252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:58.436715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309339+0000","last_change":"2024-04-25T17:32:32.285823+0000","last_active":"2024-04-25T17:35:54.309339+0000","last_peered":"2024-04-25T17:35:54.309339+0000","last_clean":"2024-04-25T17:35:54.309339+0000","last_became_active":"2024-04-25T17:32:32.285416+0000","last_became_peered":"2024-04-25T17:32:32.285416+0000","last_unstale":"2024-04-25T17:35:54.309339+0000","last_undegraded":"2024-04-25T17:35:54.309339+0000","last_fullsized":"2024-04-25T17:35:54.309339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:24.473406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402426+0000","last_change":"2024-04-25T17:32:35.072626+0000","last_active":"2024-04-25T17:35:54.402426+0000","last_peered":"2024-04-25T17:35:54.402426+0000","last_clean":"2024-04-25T17:35:54.402426+0000","last_became_active":"2024-04-25T17:32:35.072264+0000","last_became_peered":"2024-04-25T17:32:35.072264+0000","last_unstale":"2024-04-25T17:35:54.402426+0000","last_undegraded":"2024-04-25T17:35:54.402426+0000","last_fullsized":"2024-04-25T17:35:54.402426+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:54.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"92'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309697+0000","last_change":"2024-04-25T17:32:32.045589+0000","last_active":"2024-04-25T17:35:54.309697+0000","last_peered":"2024-04-25T17:35:54.309697+0000","last_clean":"2024-04-25T17:35:54.309697+0000","last_became_active":"2024-04-25T17:32:32.044838+0000","last_became_peered":"2024-04-25T17:32:32.044838+0000","last_unstale":"2024-04-25T17:35:54.309697+0000","last_undegraded":"2024-04-25T17:35:54.309697+0000","last_fullsized":"2024-04-25T17:35:54.309697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:05.748335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549290+0000","last_change":"2024-04-25T17:32:35.075077+0000","last_active":"2024-04-25T17:35:54.549290+0000","last_peered":"2024-04-25T17:35:54.549290+0000","last_clean":"2024-04-25T17:35:54.549290+0000","last_became_active":"2024-04-25T17:32:35.074622+0000","last_became_peered":"2024-04-25T17:32:35.074622+0000","last_unstale":"2024-04-25T17:35:54.549290+0000","last_undegraded":"2024-04-25T17:35:54.549290+0000","last_fullsized":"2024-04-25T17:35:54.549290+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:53.435637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307183+0000","last_change":"2024-04-25T17:32:32.288707+0000","last_active":"2024-04-25T17:35:54.307183+0000","last_peered":"2024-04-25T17:35:54.307183+0000","last_clean":"2024-04-25T17:35:54.307183+0000","last_became_active":"2024-04-25T17:32:32.288321+0000","last_became_peered":"2024-04-25T17:32:32.288321+0000","last_unstale":"2024-04-25T17:35:54.307183+0000","last_undegraded":"2024-04-25T17:35:54.307183+0000","last_fullsized":"2024-04-25T17:35:54.307183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:00.113662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350774+0000","last_change":"2024-04-25T17:32:35.072100+0000","last_active":"2024-04-25T17:35:54.350774+0000","last_peered":"2024-04-25T17:35:54.350774+0000","last_clean":"2024-04-25T17:35:54.350774+0000","last_became_active":"2024-04-25T17:32:35.070706+0000","last_became_peered":"2024-04-25T17:32:35.070706+0000","last_unstale":"2024-04-25T17:35:54.350774+0000","last_undegraded":"2024-04-25T17:35:54.350774+0000","last_fullsized":"2024-04-25T17:35:54.350774+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:18.717385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542543+0000","last_change":"2024-04-25T17:32:32.042700+0000","last_active":"2024-04-25T17:35:54.542543+0000","last_peered":"2024-04-25T17:35:54.542543+0000","last_clean":"2024-04-25T17:35:54.542543+0000","last_became_active":"2024-04-25T17:32:32.042383+0000","last_became_peered":"2024-04-25T17:32:32.042383+0000","last_unstale":"2024-04-25T17:35:54.542543+0000","last_undegraded":"2024-04-25T17:35:54.542543+0000","last_fullsized":"2024-04-25T17:35:54.542543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:11.128322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350759+0000","last_change":"2024-04-25T17:32:35.072336+0000","last_active":"2024-04-25T17:35:54.350759+0000","last_peered":"2024-04-25T17:35:54.350759+0000","last_clean":"2024-04-25T17:35:54.350759+0000","last_became_active":"2024-04-25T17:32:35.069830+0000","last_became_peered":"2024-04-25T17:32:35.069830+0000","last_unstale":"2024-04-25T17:35:54.350759+0000","last_undegraded":"2024-04-25T17:35:54.350759+0000","last_fullsized":"2024-04-25T17:35:54.350759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:12.887428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402564+0000","last_change":"2024-04-25T17:32:35.071375+0000","last_active":"2024-04-25T17:35:54.402564+0000","last_peered":"2024-04-25T17:35:54.402564+0000","last_clean":"2024-04-25T17:35:54.402564+0000","last_became_active":"2024-04-25T17:32:35.071063+0000","last_became_peered":"2024-04-25T17:32:35.071063+0000","last_unstale":"2024-04-25T17:35:54.402564+0000","last_undegraded":"2024-04-25T17:35:54.402564+0000","last_fullsized":"2024-04-25T17:35:54.402564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:41.917038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'6","reported_seq":214,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416122+0000","last_change":"2024-04-25T17:32:32.286941+0000","last_active":"2024-04-25T17:35:54.416122+0000","last_peered":"2024-04-25T17:35:54.416122+0000","last_clean":"2024-04-25T17:35:54.416122+0000","last_became_active":"2024-04-25T17:32:32.286608+0000","last_became_peered":"2024-04-25T17:32:32.286608+0000","last_unstale":"2024-04-25T17:35:54.416122+0000","last_undegraded":"2024-04-25T17:35:54.416122+0000","last_fullsized":"2024-04-25T17:35:54.416122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.559724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":179,"num_read_kb":179,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395790+0000","last_change":"2024-04-25T17:32:35.068701+0000","last_active":"2024-04-25T17:35:54.395790+0000","last_peered":"2024-04-25T17:35:54.395790+0000","last_clean":"2024-04-25T17:35:54.395790+0000","last_became_active":"2024-04-25T17:32:35.068418+0000","last_became_peered":"2024-04-25T17:32:35.068418+0000","last_unstale":"2024-04-25T17:35:54.395790+0000","last_undegraded":"2024-04-25T17:35:54.395790+0000","last_fullsized":"2024-04-25T17:35:54.395790+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:42:25.431527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310665+0000","last_change":"2024-04-25T17:32:32.042167+0000","last_active":"2024-04-25T17:35:54.310665+0000","last_peered":"2024-04-25T17:35:54.310665+0000","last_clean":"2024-04-25T17:35:54.310665+0000","last_became_active":"2024-04-25T17:32:32.041765+0000","last_became_peered":"2024-04-25T17:32:32.041765+0000","last_unstale":"2024-04-25T17:35:54.310665+0000","last_undegraded":"2024-04-25T17:35:54.310665+0000","last_fullsized":"2024-04-25T17:35:54.310665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:28.901648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":363,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542379+0000","last_change":"2024-04-25T17:31:05.391870+0000","last_active":"2024-04-25T17:35:54.542379+0000","last_peered":"2024-04-25T17:35:54.542379+0000","last_clean":"2024-04-25T17:35:54.542379+0000","last_became_active":"2024-04-25T17:31:05.391426+0000","last_became_peered":"2024-04-25T17:31:05.391426+0000","last_unstale":"2024-04-25T17:35:54.542379+0000","last_undegraded":"2024-04-25T17:35:54.542379+0000","last_fullsized":"2024-04-25T17:35:54.542379+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_clean_scrub_stamp":"2024-04-25T17:27:37.393149+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:18.400543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415657+0000","last_change":"2024-04-25T17:33:10.190518+0000","last_active":"2024-04-25T17:35:54.415657+0000","last_peered":"2024-04-25T17:35:54.415657+0000","last_clean":"2024-04-25T17:35:54.415657+0000","last_became_active":"2024-04-25T17:33:10.190166+0000","last_became_peered":"2024-04-25T17:33:10.190166+0000","last_unstale":"2024-04-25T17:35:54.415657+0000","last_undegraded":"2024-04-25T17:35:54.415657+0000","last_fullsized":"2024-04-25T17:35:54.415657+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:45.707078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542429+0000","last_change":"2024-04-25T17:33:10.197412+0000","last_active":"2024-04-25T17:35:54.542429+0000","last_peered":"2024-04-25T17:35:54.542429+0000","last_clean":"2024-04-25T17:35:54.542429+0000","last_became_active":"2024-04-25T17:33:10.194388+0000","last_became_peered":"2024-04-25T17:33:10.194388+0000","last_unstale":"2024-04-25T17:35:54.542429+0000","last_undegraded":"2024-04-25T17:35:54.542429+0000","last_fullsized":"2024-04-25T17:35:54.542429+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:35.455072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309588+0000","last_change":"2024-04-25T17:32:32.285845+0000","last_active":"2024-04-25T17:35:54.309588+0000","last_peered":"2024-04-25T17:35:54.309588+0000","last_clean":"2024-04-25T17:35:54.309588+0000","last_became_active":"2024-04-25T17:32:32.285386+0000","last_became_peered":"2024-04-25T17:32:32.285386+0000","last_unstale":"2024-04-25T17:35:54.309588+0000","last_undegraded":"2024-04-25T17:35:54.309588+0000","last_fullsized":"2024-04-25T17:35:54.309588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:12.115900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416407+0000","last_change":"2024-04-25T17:32:35.065808+0000","last_active":"2024-04-25T17:35:54.416407+0000","last_peered":"2024-04-25T17:35:54.416407+0000","last_clean":"2024-04-25T17:35:54.416407+0000","last_became_active":"2024-04-25T17:32:35.065504+0000","last_became_peered":"2024-04-25T17:32:35.065504+0000","last_unstale":"2024-04-25T17:35:54.416407+0000","last_undegraded":"2024-04-25T17:35:54.416407+0000","last_fullsized":"2024-04-25T17:35:54.416407+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:33.438281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'48","reported_seq":100,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.502778+0000","last_change":"2024-04-25T17:32:32.045357+0000","last_active":"2024-04-25T17:35:54.502778+0000","last_peered":"2024-04-25T17:35:54.502778+0000","last_clean":"2024-04-25T17:35:54.502778+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:35:54.502778+0000","last_undegraded":"2024-04-25T17:35:54.502778+0000","last_fullsized":"2024-04-25T17:35:54.502778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:36.113130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":87,"num_write_kb":839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395786+0000","last_change":"2024-04-25T17:32:35.077889+0000","last_active":"2024-04-25T17:35:54.395786+0000","last_peered":"2024-04-25T17:35:54.395786+0000","last_clean":"2024-04-25T17:35:54.395786+0000","last_became_active":"2024-04-25T17:32:35.076470+0000","last_became_peered":"2024-04-25T17:32:35.076470+0000","last_unstale":"2024-04-25T17:35:54.395786+0000","last_undegraded":"2024-04-25T17:35:54.395786+0000","last_fullsized":"2024-04-25T17:35:54.395786+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:45.457146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310626+0000","last_change":"2024-04-25T17:32:32.406826+0000","last_active":"2024-04-25T17:35:54.310626+0000","last_peered":"2024-04-25T17:35:54.310626+0000","last_clean":"2024-04-25T17:35:54.310626+0000","last_became_active":"2024-04-25T17:32:32.406454+0000","last_became_peered":"2024-04-25T17:32:32.406454+0000","last_unstale":"2024-04-25T17:35:54.310626+0000","last_undegraded":"2024-04-25T17:35:54.310626+0000","last_fullsized":"2024-04-25T17:35:54.310626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:21:11.730942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543116+0000","last_change":"2024-04-25T17:32:35.075961+0000","last_active":"2024-04-25T17:35:54.543116+0000","last_peered":"2024-04-25T17:35:54.543116+0000","last_clean":"2024-04-25T17:35:54.543116+0000","last_became_active":"2024-04-25T17:32:35.075595+0000","last_became_peered":"2024-04-25T17:32:35.075595+0000","last_unstale":"2024-04-25T17:35:54.543116+0000","last_undegraded":"2024-04-25T17:35:54.543116+0000","last_fullsized":"2024-04-25T17:35:54.543116+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:25.513096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352364+0000","last_change":"2024-04-25T17:32:32.289123+0000","last_active":"2024-04-25T17:35:54.352364+0000","last_peered":"2024-04-25T17:35:54.352364+0000","last_clean":"2024-04-25T17:35:54.352364+0000","last_became_active":"2024-04-25T17:32:32.288783+0000","last_became_peered":"2024-04-25T17:32:32.288783+0000","last_unstale":"2024-04-25T17:35:54.352364+0000","last_undegraded":"2024-04-25T17:35:54.352364+0000","last_fullsized":"2024-04-25T17:35:54.352364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:52.222440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309346+0000","last_change":"2024-04-25T17:32:35.066364+0000","last_active":"2024-04-25T17:35:54.309346+0000","last_peered":"2024-04-25T17:35:54.309346+0000","last_clean":"2024-04-25T17:35:54.309346+0000","last_became_active":"2024-04-25T17:32:35.065489+0000","last_became_peered":"2024-04-25T17:32:35.065489+0000","last_unstale":"2024-04-25T17:35:54.309346+0000","last_undegraded":"2024-04-25T17:35:54.309346+0000","last_fullsized":"2024-04-25T17:35:54.309346+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:16.817796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395705+0000","last_change":"2024-04-25T17:32:32.040662+0000","last_active":"2024-04-25T17:35:54.395705+0000","last_peered":"2024-04-25T17:35:54.395705+0000","last_clean":"2024-04-25T17:35:54.395705+0000","last_became_active":"2024-04-25T17:32:32.040334+0000","last_became_peered":"2024-04-25T17:32:32.040334+0000","last_unstale":"2024-04-25T17:35:54.395705+0000","last_undegraded":"2024-04-25T17:35:54.395705+0000","last_fullsized":"2024-04-25T17:35:54.395705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:03.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549074+0000","last_change":"2024-04-25T17:32:35.079183+0000","last_active":"2024-04-25T17:35:54.549074+0000","last_peered":"2024-04-25T17:35:54.549074+0000","last_clean":"2024-04-25T17:35:54.549074+0000","last_became_active":"2024-04-25T17:32:35.078472+0000","last_became_peered":"2024-04-25T17:32:35.078472+0000","last_unstale":"2024-04-25T17:35:54.549074+0000","last_undegraded":"2024-04-25T17:35:54.549074+0000","last_fullsized":"2024-04-25T17:35:54.549074+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:39:59.120020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549072+0000","last_change":"2024-04-25T17:33:10.201283+0000","last_active":"2024-04-25T17:35:54.549072+0000","last_peered":"2024-04-25T17:35:54.549072+0000","last_clean":"2024-04-25T17:35:54.549072+0000","last_became_active":"2024-04-25T17:33:10.196192+0000","last_became_peered":"2024-04-25T17:33:10.196192+0000","last_unstale":"2024-04-25T17:35:54.549072+0000","last_undegraded":"2024-04-25T17:35:54.549072+0000","last_fullsized":"2024-04-25T17:35:54.549072+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:03.716366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395607+0000","last_change":"2024-04-25T17:32:35.081593+0000","last_active":"2024-04-25T17:35:54.395607+0000","last_peered":"2024-04-25T17:35:54.395607+0000","last_clean":"2024-04-25T17:35:54.395607+0000","last_became_active":"2024-04-25T17:32:35.081248+0000","last_became_peered":"2024-04-25T17:32:35.081248+0000","last_unstale":"2024-04-25T17:35:54.395607+0000","last_undegraded":"2024-04-25T17:35:54.395607+0000","last_fullsized":"2024-04-25T17:35:54.395607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:41:01.177899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549097+0000","last_change":"2024-04-25T17:33:10.198901+0000","last_active":"2024-04-25T17:35:54.549097+0000","last_peered":"2024-04-25T17:35:54.549097+0000","last_clean":"2024-04-25T17:35:54.549097+0000","last_became_active":"2024-04-25T17:33:10.192191+0000","last_became_peered":"2024-04-25T17:33:10.192191+0000","last_unstale":"2024-04-25T17:35:54.549097+0000","last_undegraded":"2024-04-25T17:35:54.549097+0000","last_fullsized":"2024-04-25T17:35:54.549097+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:39.760642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542586+0000","last_change":"2024-04-25T17:32:35.068592+0000","last_active":"2024-04-25T17:35:54.542586+0000","last_peered":"2024-04-25T17:35:54.542586+0000","last_clean":"2024-04-25T17:35:54.542586+0000","last_became_active":"2024-04-25T17:32:35.068240+0000","last_became_peered":"2024-04-25T17:32:35.068240+0000","last_unstale":"2024-04-25T17:35:54.542586+0000","last_undegraded":"2024-04-25T17:35:54.542586+0000","last_fullsized":"2024-04-25T17:35:54.542586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:48.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352804+0000","last_change":"2024-04-25T17:32:32.406702+0000","last_active":"2024-04-25T17:35:54.352804+0000","last_peered":"2024-04-25T17:35:54.352804+0000","last_clean":"2024-04-25T17:35:54.352804+0000","last_became_active":"2024-04-25T17:32:32.406263+0000","last_became_peered":"2024-04-25T17:32:32.406263+0000","last_unstale":"2024-04-25T17:35:54.352804+0000","last_undegraded":"2024-04-25T17:35:54.352804+0000","last_fullsized":"2024-04-25T17:35:54.352804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:27.627002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542741+0000","last_change":"2024-04-25T17:32:35.073585+0000","last_active":"2024-04-25T17:35:54.542741+0000","last_peered":"2024-04-25T17:35:54.542741+0000","last_clean":"2024-04-25T17:35:54.542741+0000","last_became_active":"2024-04-25T17:32:35.072902+0000","last_became_peered":"2024-04-25T17:32:35.072902+0000","last_unstale":"2024-04-25T17:35:54.542741+0000","last_undegraded":"2024-04-25T17:35:54.542741+0000","last_fullsized":"2024-04-25T17:35:54.542741+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:17.277129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'39","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.500632+0000","last_change":"2024-04-25T17:32:32.042546+0000","last_active":"2024-04-25T17:35:54.500632+0000","last_peered":"2024-04-25T17:35:54.500632+0000","last_clean":"2024-04-25T17:35:54.500632+0000","last_became_active":"2024-04-25T17:32:32.042191+0000","last_became_peered":"2024-04-25T17:32:32.042191+0000","last_unstale":"2024-04-25T17:35:54.500632+0000","last_undegraded":"2024-04-25T17:35:54.500632+0000","last_fullsized":"2024-04-25T17:35:54.500632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:58.126260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542396+0000","last_change":"2024-04-25T17:32:35.077067+0000","last_active":"2024-04-25T17:35:54.542396+0000","last_peered":"2024-04-25T17:35:54.542396+0000","last_clean":"2024-04-25T17:35:54.542396+0000","last_became_active":"2024-04-25T17:32:35.076701+0000","last_became_peered":"2024-04-25T17:32:35.076701+0000","last_unstale":"2024-04-25T17:35:54.542396+0000","last_undegraded":"2024-04-25T17:35:54.542396+0000","last_fullsized":"2024-04-25T17:35:54.542396+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:06.646419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416631+0000","last_change":"2024-04-25T17:33:10.610032+0000","last_active":"2024-04-25T17:35:54.416631+0000","last_peered":"2024-04-25T17:35:54.416631+0000","last_clean":"2024-04-25T17:35:54.416631+0000","last_became_active":"2024-04-25T17:33:10.607260+0000","last_became_peered":"2024-04-25T17:33:10.607260+0000","last_unstale":"2024-04-25T17:35:54.416631+0000","last_undegraded":"2024-04-25T17:35:54.416631+0000","last_fullsized":"2024-04-25T17:35:54.416631+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:42:41.509257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542556+0000","last_change":"2024-04-25T17:32:35.071351+0000","last_active":"2024-04-25T17:35:54.542556+0000","last_peered":"2024-04-25T17:35:54.542556+0000","last_clean":"2024-04-25T17:35:54.542556+0000","last_became_active":"2024-04-25T17:32:35.070559+0000","last_became_peered":"2024-04-25T17:32:35.070559+0000","last_unstale":"2024-04-25T17:35:54.542556+0000","last_undegraded":"2024-04-25T17:35:54.542556+0000","last_fullsized":"2024-04-25T17:35:54.542556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:23.864826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549145+0000","last_change":"2024-04-25T17:32:35.073326+0000","last_active":"2024-04-25T17:35:54.549145+0000","last_peered":"2024-04-25T17:35:54.549145+0000","last_clean":"2024-04-25T17:35:54.549145+0000","last_became_active":"2024-04-25T17:32:35.072904+0000","last_became_peered":"2024-04-25T17:32:35.072904+0000","last_unstale":"2024-04-25T17:35:54.549145+0000","last_undegraded":"2024-04-25T17:35:54.549145+0000","last_fullsized":"2024-04-25T17:35:54.549145+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:50.586127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352580+0000","last_change":"2024-04-25T17:33:10.195444+0000","last_active":"2024-04-25T17:35:54.352580+0000","last_peered":"2024-04-25T17:35:54.352580+0000","last_clean":"2024-04-25T17:35:54.352580+0000","last_became_active":"2024-04-25T17:33:10.195048+0000","last_became_peered":"2024-04-25T17:33:10.195048+0000","last_unstale":"2024-04-25T17:35:54.352580+0000","last_undegraded":"2024-04-25T17:35:54.352580+0000","last_fullsized":"2024-04-25T17:35:54.352580+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:03.098618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352519+0000","last_change":"2024-04-25T17:32:35.072053+0000","last_active":"2024-04-25T17:35:54.352519+0000","last_peered":"2024-04-25T17:35:54.352519+0000","last_clean":"2024-04-25T17:35:54.352519+0000","last_became_active":"2024-04-25T17:32:35.071712+0000","last_became_peered":"2024-04-25T17:32:35.071712+0000","last_unstale":"2024-04-25T17:35:54.352519+0000","last_undegraded":"2024-04-25T17:35:54.352519+0000","last_fullsized":"2024-04-25T17:35:54.352519+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:42.928854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542632+0000","last_change":"2024-04-25T17:32:35.061408+0000","last_active":"2024-04-25T17:35:54.542632+0000","last_peered":"2024-04-25T17:35:54.542632+0000","last_clean":"2024-04-25T17:35:54.542632+0000","last_became_active":"2024-04-25T17:32:35.061105+0000","last_became_peered":"2024-04-25T17:32:35.061105+0000","last_unstale":"2024-04-25T17:35:54.542632+0000","last_undegraded":"2024-04-25T17:35:54.542632+0000","last_fullsized":"2024-04-25T17:35:54.542632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:48.242157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416347+0000","last_change":"2024-04-25T17:33:10.687122+0000","last_active":"2024-04-25T17:35:54.416347+0000","last_peered":"2024-04-25T17:35:54.416347+0000","last_clean":"2024-04-25T17:35:54.416347+0000","last_became_active":"2024-04-25T17:33:10.686755+0000","last_became_peered":"2024-04-25T17:33:10.686755+0000","last_unstale":"2024-04-25T17:35:54.416347+0000","last_undegraded":"2024-04-25T17:35:54.416347+0000","last_fullsized":"2024-04-25T17:35:54.416347+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.999655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549326+0000","last_change":"2024-04-25T17:32:35.077433+0000","last_active":"2024-04-25T17:35:54.549326+0000","last_peered":"2024-04-25T17:35:54.549326+0000","last_clean":"2024-04-25T17:35:54.549326+0000","last_became_active":"2024-04-25T17:32:35.076201+0000","last_became_peered":"2024-04-25T17:32:35.076201+0000","last_unstale":"2024-04-25T17:35:54.549326+0000","last_undegraded":"2024-04-25T17:35:54.549326+0000","last_fullsized":"2024-04-25T17:35:54.549326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:42.677022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352836+0000","last_change":"2024-04-25T17:32:35.068017+0000","last_active":"2024-04-25T17:35:54.352836+0000","last_peered":"2024-04-25T17:35:54.352836+0000","last_clean":"2024-04-25T17:35:54.352836+0000","last_became_active":"2024-04-25T17:32:35.067647+0000","last_became_peered":"2024-04-25T17:32:35.067647+0000","last_unstale":"2024-04-25T17:35:54.352836+0000","last_undegraded":"2024-04-25T17:35:54.352836+0000","last_fullsized":"2024-04-25T17:35:54.352836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:28:11.339402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395781+0000","last_change":"2024-04-25T17:32:35.075192+0000","last_active":"2024-04-25T17:35:54.395781+0000","last_peered":"2024-04-25T17:35:54.395781+0000","last_clean":"2024-04-25T17:35:54.395781+0000","last_became_active":"2024-04-25T17:32:35.074850+0000","last_became_peered":"2024-04-25T17:32:35.074850+0000","last_unstale":"2024-04-25T17:35:54.395781+0000","last_undegraded":"2024-04-25T17:35:54.395781+0000","last_fullsized":"2024-04-25T17:35:54.395781+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:04:41.343312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402408+0000","last_change":"2024-04-25T17:32:35.078324+0000","last_active":"2024-04-25T17:35:54.402408+0000","last_peered":"2024-04-25T17:35:54.402408+0000","last_clean":"2024-04-25T17:35:54.402408+0000","last_became_active":"2024-04-25T17:32:35.077958+0000","last_became_peered":"2024-04-25T17:32:35.077958+0000","last_unstale":"2024-04-25T17:35:54.402408+0000","last_undegraded":"2024-04-25T17:35:54.402408+0000","last_fullsized":"2024-04-25T17:35:54.402408+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:10.148060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395538+0000","last_change":"2024-04-25T17:32:35.078023+0000","last_active":"2024-04-25T17:35:54.395538+0000","last_peered":"2024-04-25T17:35:54.395538+0000","last_clean":"2024-04-25T17:35:54.395538+0000","last_became_active":"2024-04-25T17:32:35.077702+0000","last_became_peered":"2024-04-25T17:32:35.077702+0000","last_unstale":"2024-04-25T17:35:54.395538+0000","last_undegraded":"2024-04-25T17:35:54.395538+0000","last_fullsized":"2024-04-25T17:35:54.395538+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:44:37.933179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352546+0000","last_change":"2024-04-25T17:32:35.078388+0000","last_active":"2024-04-25T17:35:54.352546+0000","last_peered":"2024-04-25T17:35:54.352546+0000","last_clean":"2024-04-25T17:35:54.352546+0000","last_became_active":"2024-04-25T17:32:35.078058+0000","last_became_peered":"2024-04-25T17:32:35.078058+0000","last_unstale":"2024-04-25T17:35:54.352546+0000","last_undegraded":"2024-04-25T17:35:54.352546+0000","last_fullsized":"2024-04-25T17:35:54.352546+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:05.699542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416109+0000","last_change":"2024-04-25T17:32:35.068532+0000","last_active":"2024-04-25T17:35:54.416109+0000","last_peered":"2024-04-25T17:35:54.416109+0000","last_clean":"2024-04-25T17:35:54.416109+0000","last_became_active":"2024-04-25T17:32:35.068225+0000","last_became_peered":"2024-04-25T17:32:35.068225+0000","last_unstale":"2024-04-25T17:35:54.416109+0000","last_undegraded":"2024-04-25T17:35:54.416109+0000","last_fullsized":"2024-04-25T17:35:54.416109+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:09.911999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549091+0000","last_change":"2024-04-25T17:32:35.065526+0000","last_active":"2024-04-25T17:35:54.549091+0000","last_peered":"2024-04-25T17:35:54.549091+0000","last_clean":"2024-04-25T17:35:54.549091+0000","last_became_active":"2024-04-25T17:32:35.065146+0000","last_became_peered":"2024-04-25T17:32:35.065146+0000","last_unstale":"2024-04-25T17:35:54.549091+0000","last_undegraded":"2024-04-25T17:35:54.549091+0000","last_fullsized":"2024-04-25T17:35:54.549091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:33.662055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106941,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598036,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1198,"num_read_kb":708,"num_write":297,"num_write_kb":1441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1794108,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":231,"ondisk_log_size":231,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351030,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":491851,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612669,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":1283843,"data_compressed":44702,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776195,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":885303,"data_compressed":22045,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037830,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":492007,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299469,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1064960,"data_stored":1085559,"data_compressed":38546,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462994,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":491476,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":558475,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738480,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":492747,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":560915,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":1016603,"data_compressed":28201,"data_compressed_allocated":36864,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888156,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":492445,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724632,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":1215529,"data_compressed":44702,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133775,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":202089,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T17:35:59.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T17:35:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T17:35:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Unable to set osd_memory_target on smithi173 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-04-25T17:35:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi203 to 176.2M 2024-04-25T17:35:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Unable to set osd_memory_target on smithi203 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-25T17:35:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T17:35:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T17:35:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T17:35:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T17:35:59.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:35:59.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:35:59 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:35:59.901 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:00.366 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-mon[29192]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:36:00.366 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-mon[29192]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:36:00.366 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-mon[29192]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:36:00.366 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-mon[29192]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:36:00.366 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-mon[29192]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:36:00.366 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-mon[29192]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:36:00.366 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-mon[29192]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:36:00.366 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-mon[29192]: from='client.24826 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:36:00.366 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-mon[29192]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T17:36:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:00 smithi173 ceph-mon[32856]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:36:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:00 smithi173 ceph-mon[32856]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:36:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:00 smithi173 ceph-mon[32856]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:36:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:00 smithi173 ceph-mon[32856]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:36:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:00 smithi173 ceph-mon[32856]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:36:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:00 smithi173 ceph-mon[32856]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:36:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:00 smithi173 ceph-mon[32856]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:36:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:00 smithi173 ceph-mon[32856]: from='client.24826 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:36:00.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:00 smithi173 ceph-mon[32856]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T17:36:00.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:00 smithi203 ceph-mon[32834]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:36:00.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:00 smithi203 ceph-mon[32834]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:36:00.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:00 smithi203 ceph-mon[32834]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:36:00.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:00 smithi203 ceph-mon[32834]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-25T17:36:00.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:00 smithi203 ceph-mon[32834]: Updating smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:36:00.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:00 smithi203 ceph-mon[32834]: Updating smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:36:00.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:00 smithi203 ceph-mon[32834]: Updating smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T17:36:00.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:00 smithi203 ceph-mon[32834]: from='client.24826 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:36:00.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:00 smithi203 ceph-mon[32834]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T17:36:01.012 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:36:00.781+0000 7f6772c03700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:36:01.012 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:36:00.781+0000 7f6772c03700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:36:01.012 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:36:00.781+0000 7f6772c03700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:36:01.012 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:36:00.781+0000 7f6772c03700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:36:01.012 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:36:00 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:36:00.781+0000 7f6772c03700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T17:36:01.303 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:01 smithi098 ceph-mon[29192]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 65 op/s 2024-04-25T17:36:01.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:01 smithi173 ceph-mon[32856]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 65 op/s 2024-04-25T17:36:01.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 -- 172.21.15.98:0/1213066477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4150102a90 msgr2=0x7f4150102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:01.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 --2- 172.21.15.98:0/1213066477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4150102a90 0x7f4150102f10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4144009990 tx=0x7f4144009ca0 comp rx=0 tx=0).stop 2024-04-25T17:36:01.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 -- 172.21.15.98:0/1213066477 shutdown_connections 2024-04-25T17:36:01.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 --2- 172.21.15.98:0/1213066477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4150103450 0x7f415010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:01.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 --2- 172.21.15.98:0/1213066477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4150108a50 0x7f4150108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:01.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 --2- 172.21.15.98:0/1213066477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4150102a90 0x7f4150102f10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:01.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 -- 172.21.15.98:0/1213066477 >> 172.21.15.98:0/1213066477 conn(0x7f4150079ba0 msgr2=0x7f4150079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:01.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 -- 172.21.15.98:0/1213066477 shutdown_connections 2024-04-25T17:36:01.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 -- 172.21.15.98:0/1213066477 wait complete. 2024-04-25T17:36:01.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 Processor -- start 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.557+0000 7f415777c700 1 -- start start 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f415777c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4150102a90 0x7f4150074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f415777c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4150103450 0x7f4150075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f415777c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4150108a50 0x7f41500757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f415777c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4150112c20 con 0x7f4150108a50 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f415777c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4150112aa0 con 0x7f4150103450 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f415777c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4150112da0 con 0x7f4150102a90 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4155518700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4150102a90 0x7f4150074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4155d19700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4150108a50 0x7f41500757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4155518700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4150102a90 0x7f4150074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:47072/0 (socket says 172.21.15.98:47072) 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4155518700 1 -- 172.21.15.98:0/3621448185 learned_addr learned my addr 172.21.15.98:0/3621448185 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4154d17700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4150103450 0x7f4150075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:01.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4155d19700 1 -- 172.21.15.98:0/3621448185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4150102a90 msgr2=0x7f4150074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4155d19700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4150102a90 0x7f4150074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4155d19700 1 -- 172.21.15.98:0/3621448185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4150103450 msgr2=0x7f4150075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4155d19700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4150103450 0x7f4150075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4155d19700 1 -- 172.21.15.98:0/3621448185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41501a7720 con 0x7f4150108a50 2024-04-25T17:36:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f4155d19700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4150108a50 0x7f41500757c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f414000d720 tx=0x7f414000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.558+0000 7f414e7fc700 1 -- 172.21.15.98:0/3621448185 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4140016070 con 0x7f4150108a50 2024-04-25T17:36:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.559+0000 7f415777c700 1 -- 172.21.15.98:0/3621448185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41501a7a10 con 0x7f4150108a50 2024-04-25T17:36:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.559+0000 7f414e7fc700 1 -- 172.21.15.98:0/3621448185 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f414000f690 con 0x7f4150108a50 2024-04-25T17:36:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.559+0000 7f414e7fc700 1 -- 172.21.15.98:0/3621448185 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4140003a40 con 0x7f4150108a50 2024-04-25T17:36:01.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.559+0000 7f415777c700 1 -- 172.21.15.98:0/3621448185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41501a7f20 con 0x7f4150108a50 2024-04-25T17:36:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.559+0000 7f415777c700 1 -- 172.21.15.98:0/3621448185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4150069730 con 0x7f4150108a50 2024-04-25T17:36:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.561+0000 7f414e7fc700 1 -- 172.21.15.98:0/3621448185 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f4140003be0 con 0x7f4150108a50 2024-04-25T17:36:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.561+0000 7f414e7fc700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f41380a3ec0 0x7f41380a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.561+0000 7f4155518700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f41380a3ec0 0x7f41380a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.561+0000 7f414e7fc700 1 -- 172.21.15.98:0/3621448185 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f41400d0000 con 0x7f4150108a50 2024-04-25T17:36:01.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.562+0000 7f4155518700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f41380a3ec0 0x7f41380a6380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4148007fd0 tx=0x7f414800c040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:01.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.566+0000 7f414e7fc700 1 -- 172.21.15.98:0/3621448185 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f414001f070 con 0x7f4150108a50 2024-04-25T17:36:01.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:01 smithi203 ceph-mon[32834]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 65 op/s 2024-04-25T17:36:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.757+0000 7f415777c700 1 -- 172.21.15.98:0/3621448185 --> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4150062e80 con 0x7f41380a3ec0 2024-04-25T17:36:01.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.765+0000 7f414e7fc700 1 -- 172.21.15.98:0/3621448185 <== mgr.15129 v2:172.21.15.98:6800/844767972 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258018 (secure 0 0 0) 0x7f4150062e80 con 0x7f41380a3ec0 2024-04-25T17:36:01.770 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 -- 172.21.15.98:0/3621448185 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f41380a3ec0 msgr2=0x7f41380a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f41380a3ec0 0x7f41380a6380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4148007fd0 tx=0x7f414800c040 comp rx=0 tx=0).stop 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 -- 172.21.15.98:0/3621448185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4150108a50 msgr2=0x7f41500757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4150108a50 0x7f41500757c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f414000d720 tx=0x7f414000da30 comp rx=0 tx=0).stop 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 -- 172.21.15.98:0/3621448185 shutdown_connections 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f41380a3ec0 0x7f41380a6380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4150102a90 0x7f4150074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4150108a50 0x7f41500757c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 --2- 172.21.15.98:0/3621448185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4150103450 0x7f4150075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 -- 172.21.15.98:0/3621448185 >> 172.21.15.98:0/3621448185 conn(0x7f4150079ba0 msgr2=0x7f41501017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 -- 172.21.15.98:0/3621448185 shutdown_connections 2024-04-25T17:36:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:01.768+0000 7f415777c700 1 -- 172.21.15.98:0/3621448185 wait complete. 2024-04-25T17:36:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T17:36:02.414 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-25T17:36:00.320305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188638,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1564,"num_read_kb":1034,"num_write":579,"num_write_kb":4829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":923,"ondisk_log_size":923,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357436,"kb_used_data":12396,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584388,"statfs":{"total":1151940427776,"available":1151574413312,"internally_reserved":0,"allocated":12693504,"data_stored":9263776,"data_compressed":207821,"data_compressed_allocated":270336,"data_compressed_original":3207168,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":230,"num_read_kb":207,"num_write":99,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.991398"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549366+0000","last_change":"2024-04-25T17:32:35.075338+0000","last_active":"2024-04-25T17:35:54.549366+0000","last_peered":"2024-04-25T17:35:54.549366+0000","last_clean":"2024-04-25T17:35:54.549366+0000","last_became_active":"2024-04-25T17:32:35.075029+0000","last_became_peered":"2024-04-25T17:32:35.075029+0000","last_unstale":"2024-04-25T17:35:54.549366+0000","last_undegraded":"2024-04-25T17:35:54.549366+0000","last_fullsized":"2024-04-25T17:35:54.549366+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:23.837337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543054+0000","last_change":"2024-04-25T17:32:35.075422+0000","last_active":"2024-04-25T17:35:54.543054+0000","last_peered":"2024-04-25T17:35:54.543054+0000","last_clean":"2024-04-25T17:35:54.543054+0000","last_became_active":"2024-04-25T17:32:35.071975+0000","last_became_peered":"2024-04-25T17:32:35.071975+0000","last_unstale":"2024-04-25T17:35:54.543054+0000","last_undegraded":"2024-04-25T17:35:54.543054+0000","last_fullsized":"2024-04-25T17:35:54.543054+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:20.548869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542762+0000","last_change":"2024-04-25T17:32:35.068153+0000","last_active":"2024-04-25T17:35:54.542762+0000","last_peered":"2024-04-25T17:35:54.542762+0000","last_clean":"2024-04-25T17:35:54.542762+0000","last_became_active":"2024-04-25T17:32:35.065873+0000","last_became_peered":"2024-04-25T17:32:35.065873+0000","last_unstale":"2024-04-25T17:35:54.542762+0000","last_undegraded":"2024-04-25T17:35:54.542762+0000","last_fullsized":"2024-04-25T17:35:54.542762+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:31.381875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310699+0000","last_change":"2024-04-25T17:32:35.073055+0000","last_active":"2024-04-25T17:35:54.310699+0000","last_peered":"2024-04-25T17:35:54.310699+0000","last_clean":"2024-04-25T17:35:54.310699+0000","last_became_active":"2024-04-25T17:32:35.072596+0000","last_became_peered":"2024-04-25T17:32:35.072596+0000","last_unstale":"2024-04-25T17:35:54.310699+0000","last_undegraded":"2024-04-25T17:35:54.310699+0000","last_fullsized":"2024-04-25T17:35:54.310699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:37.529698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309402+0000","last_change":"2024-04-25T17:32:35.072721+0000","last_active":"2024-04-25T17:35:54.309402+0000","last_peered":"2024-04-25T17:35:54.309402+0000","last_clean":"2024-04-25T17:35:54.309402+0000","last_became_active":"2024-04-25T17:32:35.072370+0000","last_became_peered":"2024-04-25T17:32:35.072370+0000","last_unstale":"2024-04-25T17:35:54.309402+0000","last_undegraded":"2024-04-25T17:35:54.309402+0000","last_fullsized":"2024-04-25T17:35:54.309402+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:34.429985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350742+0000","last_change":"2024-04-25T17:32:35.071538+0000","last_active":"2024-04-25T17:35:54.350742+0000","last_peered":"2024-04-25T17:35:54.350742+0000","last_clean":"2024-04-25T17:35:54.350742+0000","last_became_active":"2024-04-25T17:32:35.067427+0000","last_became_peered":"2024-04-25T17:32:35.067427+0000","last_unstale":"2024-04-25T17:35:54.350742+0000","last_undegraded":"2024-04-25T17:35:54.350742+0000","last_fullsized":"2024-04-25T17:35:54.350742+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:36.052762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542299+0000","last_change":"2024-04-25T17:32:35.079476+0000","last_active":"2024-04-25T17:35:54.542299+0000","last_peered":"2024-04-25T17:35:54.542299+0000","last_clean":"2024-04-25T17:35:54.542299+0000","last_became_active":"2024-04-25T17:32:35.079170+0000","last_became_peered":"2024-04-25T17:32:35.079170+0000","last_unstale":"2024-04-25T17:35:54.542299+0000","last_undegraded":"2024-04-25T17:35:54.542299+0000","last_fullsized":"2024-04-25T17:35:54.542299+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:30.937670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395435+0000","last_change":"2024-04-25T17:32:35.077261+0000","last_active":"2024-04-25T17:35:54.395435+0000","last_peered":"2024-04-25T17:35:54.395435+0000","last_clean":"2024-04-25T17:35:54.395435+0000","last_became_active":"2024-04-25T17:32:35.076896+0000","last_became_peered":"2024-04-25T17:32:35.076896+0000","last_unstale":"2024-04-25T17:35:54.395435+0000","last_undegraded":"2024-04-25T17:35:54.395435+0000","last_fullsized":"2024-04-25T17:35:54.395435+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:24.082550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352775+0000","last_change":"2024-04-25T17:32:35.072229+0000","last_active":"2024-04-25T17:35:54.352775+0000","last_peered":"2024-04-25T17:35:54.352775+0000","last_clean":"2024-04-25T17:35:54.352775+0000","last_became_active":"2024-04-25T17:32:35.071586+0000","last_became_peered":"2024-04-25T17:32:35.071586+0000","last_unstale":"2024-04-25T17:35:54.352775+0000","last_undegraded":"2024-04-25T17:35:54.352775+0000","last_fullsized":"2024-04-25T17:35:54.352775+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:01.195638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.351018+0000","last_change":"2024-04-25T17:32:35.075274+0000","last_active":"2024-04-25T17:35:54.351018+0000","last_peered":"2024-04-25T17:35:54.351018+0000","last_clean":"2024-04-25T17:35:54.351018+0000","last_became_active":"2024-04-25T17:32:35.073821+0000","last_became_peered":"2024-04-25T17:32:35.073821+0000","last_unstale":"2024-04-25T17:35:54.351018+0000","last_undegraded":"2024-04-25T17:35:54.351018+0000","last_fullsized":"2024-04-25T17:35:54.351018+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:43:04.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542610+0000","last_change":"2024-04-25T17:32:35.080747+0000","last_active":"2024-04-25T17:35:54.542610+0000","last_peered":"2024-04-25T17:35:54.542610+0000","last_clean":"2024-04-25T17:35:54.542610+0000","last_became_active":"2024-04-25T17:32:35.080383+0000","last_became_peered":"2024-04-25T17:32:35.080383+0000","last_unstale":"2024-04-25T17:35:54.542610+0000","last_undegraded":"2024-04-25T17:35:54.542610+0000","last_fullsized":"2024-04-25T17:35:54.542610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:42.128892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310582+0000","last_change":"2024-04-25T17:32:35.061442+0000","last_active":"2024-04-25T17:35:54.310582+0000","last_peered":"2024-04-25T17:35:54.310582+0000","last_clean":"2024-04-25T17:35:54.310582+0000","last_became_active":"2024-04-25T17:32:35.061030+0000","last_became_peered":"2024-04-25T17:32:35.061030+0000","last_unstale":"2024-04-25T17:35:54.310582+0000","last_undegraded":"2024-04-25T17:35:54.310582+0000","last_fullsized":"2024-04-25T17:35:54.310582+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:03.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416138+0000","last_change":"2024-04-25T17:32:35.069698+0000","last_active":"2024-04-25T17:35:54.416138+0000","last_peered":"2024-04-25T17:35:54.416138+0000","last_clean":"2024-04-25T17:35:54.416138+0000","last_became_active":"2024-04-25T17:32:35.069374+0000","last_became_peered":"2024-04-25T17:32:35.069374+0000","last_unstale":"2024-04-25T17:35:54.416138+0000","last_undegraded":"2024-04-25T17:35:54.416138+0000","last_fullsized":"2024-04-25T17:35:54.416138+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:03.214525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542531+0000","last_change":"2024-04-25T17:32:35.069001+0000","last_active":"2024-04-25T17:35:54.542531+0000","last_peered":"2024-04-25T17:35:54.542531+0000","last_clean":"2024-04-25T17:35:54.542531+0000","last_became_active":"2024-04-25T17:32:35.064882+0000","last_became_peered":"2024-04-25T17:32:35.064882+0000","last_unstale":"2024-04-25T17:35:54.542531+0000","last_undegraded":"2024-04-25T17:35:54.542531+0000","last_fullsized":"2024-04-25T17:35:54.542531+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:32.122251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542441+0000","last_change":"2024-04-25T17:32:35.065042+0000","last_active":"2024-04-25T17:35:54.542441+0000","last_peered":"2024-04-25T17:35:54.542441+0000","last_clean":"2024-04-25T17:35:54.542441+0000","last_became_active":"2024-04-25T17:32:35.064719+0000","last_became_peered":"2024-04-25T17:32:35.064719+0000","last_unstale":"2024-04-25T17:35:54.542441+0000","last_undegraded":"2024-04-25T17:35:54.542441+0000","last_fullsized":"2024-04-25T17:35:54.542441+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:54.715385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352288+0000","last_change":"2024-04-25T17:32:35.067390+0000","last_active":"2024-04-25T17:35:54.352288+0000","last_peered":"2024-04-25T17:35:54.352288+0000","last_clean":"2024-04-25T17:35:54.352288+0000","last_became_active":"2024-04-25T17:32:35.066846+0000","last_became_peered":"2024-04-25T17:32:35.066846+0000","last_unstale":"2024-04-25T17:35:54.352288+0000","last_undegraded":"2024-04-25T17:35:54.352288+0000","last_fullsized":"2024-04-25T17:35:54.352288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:24.305767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549475+0000","last_change":"2024-04-25T17:32:35.075349+0000","last_active":"2024-04-25T17:35:54.549475+0000","last_peered":"2024-04-25T17:35:54.549475+0000","last_clean":"2024-04-25T17:35:54.549475+0000","last_became_active":"2024-04-25T17:32:35.075046+0000","last_became_peered":"2024-04-25T17:32:35.075046+0000","last_unstale":"2024-04-25T17:35:54.549475+0000","last_undegraded":"2024-04-25T17:35:54.549475+0000","last_fullsized":"2024-04-25T17:35:54.549475+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:51.703406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307381+0000","last_change":"2024-04-25T17:32:35.077597+0000","last_active":"2024-04-25T17:35:54.307381+0000","last_peered":"2024-04-25T17:35:54.307381+0000","last_clean":"2024-04-25T17:35:54.307381+0000","last_became_active":"2024-04-25T17:32:35.077232+0000","last_became_peered":"2024-04-25T17:32:35.077232+0000","last_unstale":"2024-04-25T17:35:54.307381+0000","last_undegraded":"2024-04-25T17:35:54.307381+0000","last_fullsized":"2024-04-25T17:35:54.307381+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:46.842673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352862+0000","last_change":"2024-04-25T17:32:35.078229+0000","last_active":"2024-04-25T17:35:54.352862+0000","last_peered":"2024-04-25T17:35:54.352862+0000","last_clean":"2024-04-25T17:35:54.352862+0000","last_became_active":"2024-04-25T17:32:35.077899+0000","last_became_peered":"2024-04-25T17:32:35.077899+0000","last_unstale":"2024-04-25T17:35:54.352862+0000","last_undegraded":"2024-04-25T17:35:54.352862+0000","last_fullsized":"2024-04-25T17:35:54.352862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:22.776371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307342+0000","last_change":"2024-04-25T17:32:32.045385+0000","last_active":"2024-04-25T17:35:54.307342+0000","last_peered":"2024-04-25T17:35:54.307342+0000","last_clean":"2024-04-25T17:35:54.307342+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:35:54.307342+0000","last_undegraded":"2024-04-25T17:35:54.307342+0000","last_fullsized":"2024-04-25T17:35:54.307342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:29.732001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395800+0000","last_change":"2024-04-25T17:32:35.079966+0000","last_active":"2024-04-25T17:35:54.395800+0000","last_peered":"2024-04-25T17:35:54.395800+0000","last_clean":"2024-04-25T17:35:54.395800+0000","last_became_active":"2024-04-25T17:32:35.079649+0000","last_became_peered":"2024-04-25T17:32:35.079649+0000","last_unstale":"2024-04-25T17:35:54.395800+0000","last_undegraded":"2024-04-25T17:35:54.395800+0000","last_fullsized":"2024-04-25T17:35:54.395800+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:19:45.679273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.396029+0000","last_change":"2024-04-25T17:32:32.285667+0000","last_active":"2024-04-25T17:35:54.396029+0000","last_peered":"2024-04-25T17:35:54.396029+0000","last_clean":"2024-04-25T17:35:54.396029+0000","last_became_active":"2024-04-25T17:32:32.285330+0000","last_became_peered":"2024-04-25T17:32:32.285330+0000","last_unstale":"2024-04-25T17:35:54.396029+0000","last_undegraded":"2024-04-25T17:35:54.396029+0000","last_fullsized":"2024-04-25T17:35:54.396029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.975280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543368+0000","last_change":"2024-04-25T17:32:35.068841+0000","last_active":"2024-04-25T17:35:54.543368+0000","last_peered":"2024-04-25T17:35:54.543368+0000","last_clean":"2024-04-25T17:35:54.543368+0000","last_became_active":"2024-04-25T17:32:35.068518+0000","last_became_peered":"2024-04-25T17:32:35.068518+0000","last_unstale":"2024-04-25T17:35:54.543368+0000","last_undegraded":"2024-04-25T17:35:54.543368+0000","last_fullsized":"2024-04-25T17:35:54.543368+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:23.290990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309712+0000","last_change":"2024-04-25T17:32:32.045627+0000","last_active":"2024-04-25T17:35:54.309712+0000","last_peered":"2024-04-25T17:35:54.309712+0000","last_clean":"2024-04-25T17:35:54.309712+0000","last_became_active":"2024-04-25T17:32:32.044971+0000","last_became_peered":"2024-04-25T17:32:32.044971+0000","last_unstale":"2024-04-25T17:35:54.309712+0000","last_undegraded":"2024-04-25T17:35:54.309712+0000","last_fullsized":"2024-04-25T17:35:54.309712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:50.590435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352609+0000","last_change":"2024-04-25T17:32:35.070862+0000","last_active":"2024-04-25T17:35:54.352609+0000","last_peered":"2024-04-25T17:35:54.352609+0000","last_clean":"2024-04-25T17:35:54.352609+0000","last_became_active":"2024-04-25T17:32:35.069927+0000","last_became_peered":"2024-04-25T17:32:35.069927+0000","last_unstale":"2024-04-25T17:35:54.352609+0000","last_undegraded":"2024-04-25T17:35:54.352609+0000","last_fullsized":"2024-04-25T17:35:54.352609+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:40:01.799015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309802+0000","last_change":"2024-04-25T17:32:32.046062+0000","last_active":"2024-04-25T17:35:54.309802+0000","last_peered":"2024-04-25T17:35:54.309802+0000","last_clean":"2024-04-25T17:35:54.309802+0000","last_became_active":"2024-04-25T17:32:32.044903+0000","last_became_peered":"2024-04-25T17:32:32.044903+0000","last_unstale":"2024-04-25T17:35:54.309802+0000","last_undegraded":"2024-04-25T17:35:54.309802+0000","last_fullsized":"2024-04-25T17:35:54.309802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:51.099460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402607+0000","last_change":"2024-04-25T17:32:35.075452+0000","last_active":"2024-04-25T17:35:54.402607+0000","last_peered":"2024-04-25T17:35:54.402607+0000","last_clean":"2024-04-25T17:35:54.402607+0000","last_became_active":"2024-04-25T17:32:35.075136+0000","last_became_peered":"2024-04-25T17:32:35.075136+0000","last_unstale":"2024-04-25T17:35:54.402607+0000","last_undegraded":"2024-04-25T17:35:54.402607+0000","last_fullsized":"2024-04-25T17:35:54.402607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:02.290926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":218,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415513+0000","last_change":"2024-04-25T17:32:32.042019+0000","last_active":"2024-04-25T17:35:54.415513+0000","last_peered":"2024-04-25T17:35:54.415513+0000","last_clean":"2024-04-25T17:35:54.415513+0000","last_became_active":"2024-04-25T17:32:32.041641+0000","last_became_peered":"2024-04-25T17:32:32.041641+0000","last_unstale":"2024-04-25T17:35:54.415513+0000","last_undegraded":"2024-04-25T17:35:54.415513+0000","last_fullsized":"2024-04-25T17:35:54.415513+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:02.025915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309350+0000","last_change":"2024-04-25T17:32:35.069715+0000","last_active":"2024-04-25T17:35:54.309350+0000","last_peered":"2024-04-25T17:35:54.309350+0000","last_clean":"2024-04-25T17:35:54.309350+0000","last_became_active":"2024-04-25T17:32:35.069339+0000","last_became_peered":"2024-04-25T17:32:35.069339+0000","last_unstale":"2024-04-25T17:35:54.309350+0000","last_undegraded":"2024-04-25T17:35:54.309350+0000","last_fullsized":"2024-04-25T17:35:54.309350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:48:13.572261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350662+0000","last_change":"2024-04-25T17:32:32.042445+0000","last_active":"2024-04-25T17:35:54.350662+0000","last_peered":"2024-04-25T17:35:54.350662+0000","last_clean":"2024-04-25T17:35:54.350662+0000","last_became_active":"2024-04-25T17:32:32.042033+0000","last_became_peered":"2024-04-25T17:32:32.042033+0000","last_unstale":"2024-04-25T17:35:54.350662+0000","last_undegraded":"2024-04-25T17:35:54.350662+0000","last_fullsized":"2024-04-25T17:35:54.350662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.968383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402497+0000","last_change":"2024-04-25T17:32:35.074492+0000","last_active":"2024-04-25T17:35:54.402497+0000","last_peered":"2024-04-25T17:35:54.402497+0000","last_clean":"2024-04-25T17:35:54.402497+0000","last_became_active":"2024-04-25T17:32:35.073652+0000","last_became_peered":"2024-04-25T17:32:35.073652+0000","last_unstale":"2024-04-25T17:35:54.402497+0000","last_undegraded":"2024-04-25T17:35:54.402497+0000","last_fullsized":"2024-04-25T17:35:54.402497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:00.300453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415757+0000","last_change":"2024-04-25T17:32:32.039568+0000","last_active":"2024-04-25T17:35:54.415757+0000","last_peered":"2024-04-25T17:35:54.415757+0000","last_clean":"2024-04-25T17:35:54.415757+0000","last_became_active":"2024-04-25T17:32:32.039228+0000","last_became_peered":"2024-04-25T17:32:32.039228+0000","last_unstale":"2024-04-25T17:35:54.415757+0000","last_undegraded":"2024-04-25T17:35:54.415757+0000","last_fullsized":"2024-04-25T17:35:54.415757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:17:29.079749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395508+0000","last_change":"2024-04-25T17:32:35.077619+0000","last_active":"2024-04-25T17:35:54.395508+0000","last_peered":"2024-04-25T17:35:54.395508+0000","last_clean":"2024-04-25T17:35:54.395508+0000","last_became_active":"2024-04-25T17:32:35.077309+0000","last_became_peered":"2024-04-25T17:32:35.077309+0000","last_unstale":"2024-04-25T17:35:54.395508+0000","last_undegraded":"2024-04-25T17:35:54.395508+0000","last_fullsized":"2024-04-25T17:35:54.395508+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:41.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402343+0000","last_change":"2024-04-25T17:32:32.044650+0000","last_active":"2024-04-25T17:35:54.402343+0000","last_peered":"2024-04-25T17:35:54.402343+0000","last_clean":"2024-04-25T17:35:54.402343+0000","last_became_active":"2024-04-25T17:32:32.043999+0000","last_became_peered":"2024-04-25T17:32:32.043999+0000","last_unstale":"2024-04-25T17:35:54.402343+0000","last_undegraded":"2024-04-25T17:35:54.402343+0000","last_fullsized":"2024-04-25T17:35:54.402343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:19.463941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307201+0000","last_change":"2024-04-25T17:32:35.080598+0000","last_active":"2024-04-25T17:35:54.307201+0000","last_peered":"2024-04-25T17:35:54.307201+0000","last_clean":"2024-04-25T17:35:54.307201+0000","last_became_active":"2024-04-25T17:32:35.080300+0000","last_became_peered":"2024-04-25T17:32:35.080300+0000","last_unstale":"2024-04-25T17:35:54.307201+0000","last_undegraded":"2024-04-25T17:35:54.307201+0000","last_fullsized":"2024-04-25T17:35:54.307201+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:42.149746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":212,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350925+0000","last_change":"2024-04-25T17:32:32.043889+0000","last_active":"2024-04-25T17:35:54.350925+0000","last_peered":"2024-04-25T17:35:54.350925+0000","last_clean":"2024-04-25T17:35:54.350925+0000","last_became_active":"2024-04-25T17:32:32.043520+0000","last_became_peered":"2024-04-25T17:32:32.043520+0000","last_unstale":"2024-04-25T17:35:54.350925+0000","last_undegraded":"2024-04-25T17:35:54.350925+0000","last_fullsized":"2024-04-25T17:35:54.350925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:25.134797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":179,"num_read_kb":179,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352641+0000","last_change":"2024-04-25T17:32:35.071733+0000","last_active":"2024-04-25T17:35:54.352641+0000","last_peered":"2024-04-25T17:35:54.352641+0000","last_clean":"2024-04-25T17:35:54.352641+0000","last_became_active":"2024-04-25T17:32:35.071383+0000","last_became_peered":"2024-04-25T17:32:35.071383+0000","last_unstale":"2024-04-25T17:35:54.352641+0000","last_undegraded":"2024-04-25T17:35:54.352641+0000","last_fullsized":"2024-04-25T17:35:54.352641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:09:49.433675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542701+0000","last_change":"2024-04-25T17:32:32.044823+0000","last_active":"2024-04-25T17:35:54.542701+0000","last_peered":"2024-04-25T17:35:54.542701+0000","last_clean":"2024-04-25T17:35:54.542701+0000","last_became_active":"2024-04-25T17:32:32.044497+0000","last_became_peered":"2024-04-25T17:32:32.044497+0000","last_unstale":"2024-04-25T17:35:54.542701+0000","last_undegraded":"2024-04-25T17:35:54.542701+0000","last_fullsized":"2024-04-25T17:35:54.542701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:47.811172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543284+0000","last_change":"2024-04-25T17:32:35.071109+0000","last_active":"2024-04-25T17:35:54.543284+0000","last_peered":"2024-04-25T17:35:54.543284+0000","last_clean":"2024-04-25T17:35:54.543284+0000","last_became_active":"2024-04-25T17:32:35.070782+0000","last_became_peered":"2024-04-25T17:32:35.070782+0000","last_unstale":"2024-04-25T17:35:54.543284+0000","last_undegraded":"2024-04-25T17:35:54.543284+0000","last_fullsized":"2024-04-25T17:35:54.543284+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:37.247274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402726+0000","last_change":"2024-04-25T17:32:32.288683+0000","last_active":"2024-04-25T17:35:54.402726+0000","last_peered":"2024-04-25T17:35:54.402726+0000","last_clean":"2024-04-25T17:35:54.402726+0000","last_became_active":"2024-04-25T17:32:32.288261+0000","last_became_peered":"2024-04-25T17:32:32.288261+0000","last_unstale":"2024-04-25T17:35:54.402726+0000","last_undegraded":"2024-04-25T17:35:54.402726+0000","last_fullsized":"2024-04-25T17:35:54.402726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:36.502434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352607+0000","last_change":"2024-04-25T17:32:35.060828+0000","last_active":"2024-04-25T17:35:54.352607+0000","last_peered":"2024-04-25T17:35:54.352607+0000","last_clean":"2024-04-25T17:35:54.352607+0000","last_became_active":"2024-04-25T17:32:35.060433+0000","last_became_peered":"2024-04-25T17:32:35.060433+0000","last_unstale":"2024-04-25T17:35:54.352607+0000","last_undegraded":"2024-04-25T17:35:54.352607+0000","last_fullsized":"2024-04-25T17:35:54.352607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:32.382042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":436,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415711+0000","last_change":"2024-04-25T17:32:32.042979+0000","last_active":"2024-04-25T17:35:54.415711+0000","last_peered":"2024-04-25T17:35:54.415711+0000","last_clean":"2024-04-25T17:35:54.415711+0000","last_became_active":"2024-04-25T17:32:32.042257+0000","last_became_peered":"2024-04-25T17:32:32.042257+0000","last_unstale":"2024-04-25T17:35:54.415711+0000","last_undegraded":"2024-04-25T17:35:54.415711+0000","last_fullsized":"2024-04-25T17:35:54.415711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:19.316803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67597,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":371,"num_read_kb":252,"num_write":37,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309526+0000","last_change":"2024-04-25T17:32:35.069994+0000","last_active":"2024-04-25T17:35:54.309526+0000","last_peered":"2024-04-25T17:35:54.309526+0000","last_clean":"2024-04-25T17:35:54.309526+0000","last_became_active":"2024-04-25T17:32:35.069632+0000","last_became_peered":"2024-04-25T17:32:35.069632+0000","last_unstale":"2024-04-25T17:35:54.309526+0000","last_undegraded":"2024-04-25T17:35:54.309526+0000","last_fullsized":"2024-04-25T17:35:54.309526+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:51.169489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352625+0000","last_change":"2024-04-25T17:32:32.042652+0000","last_active":"2024-04-25T17:35:54.352625+0000","last_peered":"2024-04-25T17:35:54.352625+0000","last_clean":"2024-04-25T17:35:54.352625+0000","last_became_active":"2024-04-25T17:32:32.042214+0000","last_became_peered":"2024-04-25T17:32:32.042214+0000","last_unstale":"2024-04-25T17:35:54.352625+0000","last_undegraded":"2024-04-25T17:35:54.352625+0000","last_fullsized":"2024-04-25T17:35:54.352625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.662580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415731+0000","last_change":"2024-04-25T17:32:35.074449+0000","last_active":"2024-04-25T17:35:54.415731+0000","last_peered":"2024-04-25T17:35:54.415731+0000","last_clean":"2024-04-25T17:35:54.415731+0000","last_became_active":"2024-04-25T17:32:35.074116+0000","last_became_peered":"2024-04-25T17:32:35.074116+0000","last_unstale":"2024-04-25T17:35:54.415731+0000","last_undegraded":"2024-04-25T17:35:54.415731+0000","last_fullsized":"2024-04-25T17:35:54.415731+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:09.233206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402590+0000","last_change":"2024-04-25T17:32:32.041368+0000","last_active":"2024-04-25T17:35:54.402590+0000","last_peered":"2024-04-25T17:35:54.402590+0000","last_clean":"2024-04-25T17:35:54.402590+0000","last_became_active":"2024-04-25T17:32:32.040940+0000","last_became_peered":"2024-04-25T17:32:32.040940+0000","last_unstale":"2024-04-25T17:35:54.402590+0000","last_undegraded":"2024-04-25T17:35:54.402590+0000","last_fullsized":"2024-04-25T17:35:54.402590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:38.871786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542869+0000","last_change":"2024-04-25T17:32:35.073535+0000","last_active":"2024-04-25T17:35:54.542869+0000","last_peered":"2024-04-25T17:35:54.542869+0000","last_clean":"2024-04-25T17:35:54.542869+0000","last_became_active":"2024-04-25T17:32:35.073190+0000","last_became_peered":"2024-04-25T17:32:35.073190+0000","last_unstale":"2024-04-25T17:35:54.542869+0000","last_undegraded":"2024-04-25T17:35:54.542869+0000","last_fullsized":"2024-04-25T17:35:54.542869+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:19.841978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'7","reported_seq":219,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542253+0000","last_change":"2024-04-25T17:32:32.044359+0000","last_active":"2024-04-25T17:35:54.542253+0000","last_peered":"2024-04-25T17:35:54.542253+0000","last_clean":"2024-04-25T17:35:54.542253+0000","last_became_active":"2024-04-25T17:32:32.044024+0000","last_became_peered":"2024-04-25T17:32:32.044024+0000","last_unstale":"2024-04-25T17:35:54.542253+0000","last_undegraded":"2024-04-25T17:35:54.542253+0000","last_fullsized":"2024-04-25T17:35:54.542253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:58.940554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":184,"num_read_kb":5,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542252+0000","last_change":"2024-04-25T17:33:10.851665+0000","last_active":"2024-04-25T17:35:54.542252+0000","last_peered":"2024-04-25T17:35:54.542252+0000","last_clean":"2024-04-25T17:35:54.542252+0000","last_became_active":"2024-04-25T17:33:10.851217+0000","last_became_peered":"2024-04-25T17:33:10.851217+0000","last_unstale":"2024-04-25T17:35:54.542252+0000","last_undegraded":"2024-04-25T17:35:54.542252+0000","last_fullsized":"2024-04-25T17:35:54.542252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:58.436715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309339+0000","last_change":"2024-04-25T17:32:32.285823+0000","last_active":"2024-04-25T17:35:54.309339+0000","last_peered":"2024-04-25T17:35:54.309339+0000","last_clean":"2024-04-25T17:35:54.309339+0000","last_became_active":"2024-04-25T17:32:32.285416+0000","last_became_peered":"2024-04-25T17:32:32.285416+0000","last_unstale":"2024-04-25T17:35:54.309339+0000","last_undegraded":"2024-04-25T17:35:54.309339+0000","last_fullsized":"2024-04-25T17:35:54.309339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:24.473406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402426+0000","last_change":"2024-04-25T17:32:35.072626+0000","last_active":"2024-04-25T17:35:54.402426+0000","last_peered":"2024-04-25T17:35:54.402426+0000","last_clean":"2024-04-25T17:35:54.402426+0000","last_became_active":"2024-04-25T17:32:35.072264+0000","last_became_peered":"2024-04-25T17:32:35.072264+0000","last_unstale":"2024-04-25T17:35:54.402426+0000","last_undegraded":"2024-04-25T17:35:54.402426+0000","last_fullsized":"2024-04-25T17:35:54.402426+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:54.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"92'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309697+0000","last_change":"2024-04-25T17:32:32.045589+0000","last_active":"2024-04-25T17:35:54.309697+0000","last_peered":"2024-04-25T17:35:54.309697+0000","last_clean":"2024-04-25T17:35:54.309697+0000","last_became_active":"2024-04-25T17:32:32.044838+0000","last_became_peered":"2024-04-25T17:32:32.044838+0000","last_unstale":"2024-04-25T17:35:54.309697+0000","last_undegraded":"2024-04-25T17:35:54.309697+0000","last_fullsized":"2024-04-25T17:35:54.309697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:05.748335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549290+0000","last_change":"2024-04-25T17:32:35.075077+0000","last_active":"2024-04-25T17:35:54.549290+0000","last_peered":"2024-04-25T17:35:54.549290+0000","last_clean":"2024-04-25T17:35:54.549290+0000","last_became_active":"2024-04-25T17:32:35.074622+0000","last_became_peered":"2024-04-25T17:32:35.074622+0000","last_unstale":"2024-04-25T17:35:54.549290+0000","last_undegraded":"2024-04-25T17:35:54.549290+0000","last_fullsized":"2024-04-25T17:35:54.549290+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:53.435637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307183+0000","last_change":"2024-04-25T17:32:32.288707+0000","last_active":"2024-04-25T17:35:54.307183+0000","last_peered":"2024-04-25T17:35:54.307183+0000","last_clean":"2024-04-25T17:35:54.307183+0000","last_became_active":"2024-04-25T17:32:32.288321+0000","last_became_peered":"2024-04-25T17:32:32.288321+0000","last_unstale":"2024-04-25T17:35:54.307183+0000","last_undegraded":"2024-04-25T17:35:54.307183+0000","last_fullsized":"2024-04-25T17:35:54.307183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:00.113662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350774+0000","last_change":"2024-04-25T17:32:35.072100+0000","last_active":"2024-04-25T17:35:54.350774+0000","last_peered":"2024-04-25T17:35:54.350774+0000","last_clean":"2024-04-25T17:35:54.350774+0000","last_became_active":"2024-04-25T17:32:35.070706+0000","last_became_peered":"2024-04-25T17:32:35.070706+0000","last_unstale":"2024-04-25T17:35:54.350774+0000","last_undegraded":"2024-04-25T17:35:54.350774+0000","last_fullsized":"2024-04-25T17:35:54.350774+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:18.717385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542543+0000","last_change":"2024-04-25T17:32:32.042700+0000","last_active":"2024-04-25T17:35:54.542543+0000","last_peered":"2024-04-25T17:35:54.542543+0000","last_clean":"2024-04-25T17:35:54.542543+0000","last_became_active":"2024-04-25T17:32:32.042383+0000","last_became_peered":"2024-04-25T17:32:32.042383+0000","last_unstale":"2024-04-25T17:35:54.542543+0000","last_undegraded":"2024-04-25T17:35:54.542543+0000","last_fullsized":"2024-04-25T17:35:54.542543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:11.128322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350759+0000","last_change":"2024-04-25T17:32:35.072336+0000","last_active":"2024-04-25T17:35:54.350759+0000","last_peered":"2024-04-25T17:35:54.350759+0000","last_clean":"2024-04-25T17:35:54.350759+0000","last_became_active":"2024-04-25T17:32:35.069830+0000","last_became_peered":"2024-04-25T17:32:35.069830+0000","last_unstale":"2024-04-25T17:35:54.350759+0000","last_undegraded":"2024-04-25T17:35:54.350759+0000","last_fullsized":"2024-04-25T17:35:54.350759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:12.887428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402564+0000","last_change":"2024-04-25T17:32:35.071375+0000","last_active":"2024-04-25T17:35:54.402564+0000","last_peered":"2024-04-25T17:35:54.402564+0000","last_clean":"2024-04-25T17:35:54.402564+0000","last_became_active":"2024-04-25T17:32:35.071063+0000","last_became_peered":"2024-04-25T17:32:35.071063+0000","last_unstale":"2024-04-25T17:35:54.402564+0000","last_undegraded":"2024-04-25T17:35:54.402564+0000","last_fullsized":"2024-04-25T17:35:54.402564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:41.917038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'6","reported_seq":214,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416122+0000","last_change":"2024-04-25T17:32:32.286941+0000","last_active":"2024-04-25T17:35:54.416122+0000","last_peered":"2024-04-25T17:35:54.416122+0000","last_clean":"2024-04-25T17:35:54.416122+0000","last_became_active":"2024-04-25T17:32:32.286608+0000","last_became_peered":"2024-04-25T17:32:32.286608+0000","last_unstale":"2024-04-25T17:35:54.416122+0000","last_undegraded":"2024-04-25T17:35:54.416122+0000","last_fullsized":"2024-04-25T17:35:54.416122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.559724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":179,"num_read_kb":179,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395790+0000","last_change":"2024-04-25T17:32:35.068701+0000","last_active":"2024-04-25T17:35:54.395790+0000","last_peered":"2024-04-25T17:35:54.395790+0000","last_clean":"2024-04-25T17:35:54.395790+0000","last_became_active":"2024-04-25T17:32:35.068418+0000","last_became_peered":"2024-04-25T17:32:35.068418+0000","last_unstale":"2024-04-25T17:35:54.395790+0000","last_undegraded":"2024-04-25T17:35:54.395790+0000","last_fullsized":"2024-04-25T17:35:54.395790+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:42:25.431527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310665+0000","last_change":"2024-04-25T17:32:32.042167+0000","last_active":"2024-04-25T17:35:54.310665+0000","last_peered":"2024-04-25T17:35:54.310665+0000","last_clean":"2024-04-25T17:35:54.310665+0000","last_became_active":"2024-04-25T17:32:32.041765+0000","last_became_peered":"2024-04-25T17:32:32.041765+0000","last_unstale":"2024-04-25T17:35:54.310665+0000","last_undegraded":"2024-04-25T17:35:54.310665+0000","last_fullsized":"2024-04-25T17:35:54.310665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:28.901648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":363,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542379+0000","last_change":"2024-04-25T17:31:05.391870+0000","last_active":"2024-04-25T17:35:54.542379+0000","last_peered":"2024-04-25T17:35:54.542379+0000","last_clean":"2024-04-25T17:35:54.542379+0000","last_became_active":"2024-04-25T17:31:05.391426+0000","last_became_peered":"2024-04-25T17:31:05.391426+0000","last_unstale":"2024-04-25T17:35:54.542379+0000","last_undegraded":"2024-04-25T17:35:54.542379+0000","last_fullsized":"2024-04-25T17:35:54.542379+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_clean_scrub_stamp":"2024-04-25T17:27:37.393149+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:18.400543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415657+0000","last_change":"2024-04-25T17:33:10.190518+0000","last_active":"2024-04-25T17:35:54.415657+0000","last_peered":"2024-04-25T17:35:54.415657+0000","last_clean":"2024-04-25T17:35:54.415657+0000","last_became_active":"2024-04-25T17:33:10.190166+0000","last_became_peered":"2024-04-25T17:33:10.190166+0000","last_unstale":"2024-04-25T17:35:54.415657+0000","last_undegraded":"2024-04-25T17:35:54.415657+0000","last_fullsized":"2024-04-25T17:35:54.415657+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:45.707078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542429+0000","last_change":"2024-04-25T17:33:10.197412+0000","last_active":"2024-04-25T17:35:54.542429+0000","last_peered":"2024-04-25T17:35:54.542429+0000","last_clean":"2024-04-25T17:35:54.542429+0000","last_became_active":"2024-04-25T17:33:10.194388+0000","last_became_peered":"2024-04-25T17:33:10.194388+0000","last_unstale":"2024-04-25T17:35:54.542429+0000","last_undegraded":"2024-04-25T17:35:54.542429+0000","last_fullsized":"2024-04-25T17:35:54.542429+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:35.455072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309588+0000","last_change":"2024-04-25T17:32:32.285845+0000","last_active":"2024-04-25T17:35:54.309588+0000","last_peered":"2024-04-25T17:35:54.309588+0000","last_clean":"2024-04-25T17:35:54.309588+0000","last_became_active":"2024-04-25T17:32:32.285386+0000","last_became_peered":"2024-04-25T17:32:32.285386+0000","last_unstale":"2024-04-25T17:35:54.309588+0000","last_undegraded":"2024-04-25T17:35:54.309588+0000","last_fullsized":"2024-04-25T17:35:54.309588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:12.115900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416407+0000","last_change":"2024-04-25T17:32:35.065808+0000","last_active":"2024-04-25T17:35:54.416407+0000","last_peered":"2024-04-25T17:35:54.416407+0000","last_clean":"2024-04-25T17:35:54.416407+0000","last_became_active":"2024-04-25T17:32:35.065504+0000","last_became_peered":"2024-04-25T17:32:35.065504+0000","last_unstale":"2024-04-25T17:35:54.416407+0000","last_undegraded":"2024-04-25T17:35:54.416407+0000","last_fullsized":"2024-04-25T17:35:54.416407+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:33.438281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'101","reported_seq":222,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:56.480578+0000","last_change":"2024-04-25T17:32:32.045357+0000","last_active":"2024-04-25T17:35:56.480578+0000","last_peered":"2024-04-25T17:35:56.480578+0000","last_clean":"2024-04-25T17:35:56.480578+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:35:56.480578+0000","last_undegraded":"2024-04-25T17:35:56.480578+0000","last_fullsized":"2024-04-25T17:35:56.480578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:36.113130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":186,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395786+0000","last_change":"2024-04-25T17:32:35.077889+0000","last_active":"2024-04-25T17:35:54.395786+0000","last_peered":"2024-04-25T17:35:54.395786+0000","last_clean":"2024-04-25T17:35:54.395786+0000","last_became_active":"2024-04-25T17:32:35.076470+0000","last_became_peered":"2024-04-25T17:32:35.076470+0000","last_unstale":"2024-04-25T17:35:54.395786+0000","last_undegraded":"2024-04-25T17:35:54.395786+0000","last_fullsized":"2024-04-25T17:35:54.395786+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:45.457146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310626+0000","last_change":"2024-04-25T17:32:32.406826+0000","last_active":"2024-04-25T17:35:54.310626+0000","last_peered":"2024-04-25T17:35:54.310626+0000","last_clean":"2024-04-25T17:35:54.310626+0000","last_became_active":"2024-04-25T17:32:32.406454+0000","last_became_peered":"2024-04-25T17:32:32.406454+0000","last_unstale":"2024-04-25T17:35:54.310626+0000","last_undegraded":"2024-04-25T17:35:54.310626+0000","last_fullsized":"2024-04-25T17:35:54.310626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:21:11.730942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543116+0000","last_change":"2024-04-25T17:32:35.075961+0000","last_active":"2024-04-25T17:35:54.543116+0000","last_peered":"2024-04-25T17:35:54.543116+0000","last_clean":"2024-04-25T17:35:54.543116+0000","last_became_active":"2024-04-25T17:32:35.075595+0000","last_became_peered":"2024-04-25T17:32:35.075595+0000","last_unstale":"2024-04-25T17:35:54.543116+0000","last_undegraded":"2024-04-25T17:35:54.543116+0000","last_fullsized":"2024-04-25T17:35:54.543116+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:25.513096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352364+0000","last_change":"2024-04-25T17:32:32.289123+0000","last_active":"2024-04-25T17:35:54.352364+0000","last_peered":"2024-04-25T17:35:54.352364+0000","last_clean":"2024-04-25T17:35:54.352364+0000","last_became_active":"2024-04-25T17:32:32.288783+0000","last_became_peered":"2024-04-25T17:32:32.288783+0000","last_unstale":"2024-04-25T17:35:54.352364+0000","last_undegraded":"2024-04-25T17:35:54.352364+0000","last_fullsized":"2024-04-25T17:35:54.352364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:52.222440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309346+0000","last_change":"2024-04-25T17:32:35.066364+0000","last_active":"2024-04-25T17:35:54.309346+0000","last_peered":"2024-04-25T17:35:54.309346+0000","last_clean":"2024-04-25T17:35:54.309346+0000","last_became_active":"2024-04-25T17:32:35.065489+0000","last_became_peered":"2024-04-25T17:32:35.065489+0000","last_unstale":"2024-04-25T17:35:54.309346+0000","last_undegraded":"2024-04-25T17:35:54.309346+0000","last_fullsized":"2024-04-25T17:35:54.309346+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:16.817796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395705+0000","last_change":"2024-04-25T17:32:32.040662+0000","last_active":"2024-04-25T17:35:54.395705+0000","last_peered":"2024-04-25T17:35:54.395705+0000","last_clean":"2024-04-25T17:35:54.395705+0000","last_became_active":"2024-04-25T17:32:32.040334+0000","last_became_peered":"2024-04-25T17:32:32.040334+0000","last_unstale":"2024-04-25T17:35:54.395705+0000","last_undegraded":"2024-04-25T17:35:54.395705+0000","last_fullsized":"2024-04-25T17:35:54.395705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:03.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549074+0000","last_change":"2024-04-25T17:32:35.079183+0000","last_active":"2024-04-25T17:35:54.549074+0000","last_peered":"2024-04-25T17:35:54.549074+0000","last_clean":"2024-04-25T17:35:54.549074+0000","last_became_active":"2024-04-25T17:32:35.078472+0000","last_became_peered":"2024-04-25T17:32:35.078472+0000","last_unstale":"2024-04-25T17:35:54.549074+0000","last_undegraded":"2024-04-25T17:35:54.549074+0000","last_fullsized":"2024-04-25T17:35:54.549074+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:39:59.120020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549072+0000","last_change":"2024-04-25T17:33:10.201283+0000","last_active":"2024-04-25T17:35:54.549072+0000","last_peered":"2024-04-25T17:35:54.549072+0000","last_clean":"2024-04-25T17:35:54.549072+0000","last_became_active":"2024-04-25T17:33:10.196192+0000","last_became_peered":"2024-04-25T17:33:10.196192+0000","last_unstale":"2024-04-25T17:35:54.549072+0000","last_undegraded":"2024-04-25T17:35:54.549072+0000","last_fullsized":"2024-04-25T17:35:54.549072+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:03.716366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395607+0000","last_change":"2024-04-25T17:32:35.081593+0000","last_active":"2024-04-25T17:35:54.395607+0000","last_peered":"2024-04-25T17:35:54.395607+0000","last_clean":"2024-04-25T17:35:54.395607+0000","last_became_active":"2024-04-25T17:32:35.081248+0000","last_became_peered":"2024-04-25T17:32:35.081248+0000","last_unstale":"2024-04-25T17:35:54.395607+0000","last_undegraded":"2024-04-25T17:35:54.395607+0000","last_fullsized":"2024-04-25T17:35:54.395607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:41:01.177899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549097+0000","last_change":"2024-04-25T17:33:10.198901+0000","last_active":"2024-04-25T17:35:54.549097+0000","last_peered":"2024-04-25T17:35:54.549097+0000","last_clean":"2024-04-25T17:35:54.549097+0000","last_became_active":"2024-04-25T17:33:10.192191+0000","last_became_peered":"2024-04-25T17:33:10.192191+0000","last_unstale":"2024-04-25T17:35:54.549097+0000","last_undegraded":"2024-04-25T17:35:54.549097+0000","last_fullsized":"2024-04-25T17:35:54.549097+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:39.760642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542586+0000","last_change":"2024-04-25T17:32:35.068592+0000","last_active":"2024-04-25T17:35:54.542586+0000","last_peered":"2024-04-25T17:35:54.542586+0000","last_clean":"2024-04-25T17:35:54.542586+0000","last_became_active":"2024-04-25T17:32:35.068240+0000","last_became_peered":"2024-04-25T17:32:35.068240+0000","last_unstale":"2024-04-25T17:35:54.542586+0000","last_undegraded":"2024-04-25T17:35:54.542586+0000","last_fullsized":"2024-04-25T17:35:54.542586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:48.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352804+0000","last_change":"2024-04-25T17:32:32.406702+0000","last_active":"2024-04-25T17:35:54.352804+0000","last_peered":"2024-04-25T17:35:54.352804+0000","last_clean":"2024-04-25T17:35:54.352804+0000","last_became_active":"2024-04-25T17:32:32.406263+0000","last_became_peered":"2024-04-25T17:32:32.406263+0000","last_unstale":"2024-04-25T17:35:54.352804+0000","last_undegraded":"2024-04-25T17:35:54.352804+0000","last_fullsized":"2024-04-25T17:35:54.352804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:27.627002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542741+0000","last_change":"2024-04-25T17:32:35.073585+0000","last_active":"2024-04-25T17:35:54.542741+0000","last_peered":"2024-04-25T17:35:54.542741+0000","last_clean":"2024-04-25T17:35:54.542741+0000","last_became_active":"2024-04-25T17:32:35.072902+0000","last_became_peered":"2024-04-25T17:32:35.072902+0000","last_unstale":"2024-04-25T17:35:54.542741+0000","last_undegraded":"2024-04-25T17:35:54.542741+0000","last_fullsized":"2024-04-25T17:35:54.542741+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:17.277129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'39","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.500632+0000","last_change":"2024-04-25T17:32:32.042546+0000","last_active":"2024-04-25T17:35:54.500632+0000","last_peered":"2024-04-25T17:35:54.500632+0000","last_clean":"2024-04-25T17:35:54.500632+0000","last_became_active":"2024-04-25T17:32:32.042191+0000","last_became_peered":"2024-04-25T17:32:32.042191+0000","last_unstale":"2024-04-25T17:35:54.500632+0000","last_undegraded":"2024-04-25T17:35:54.500632+0000","last_fullsized":"2024-04-25T17:35:54.500632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:58.126260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542396+0000","last_change":"2024-04-25T17:32:35.077067+0000","last_active":"2024-04-25T17:35:54.542396+0000","last_peered":"2024-04-25T17:35:54.542396+0000","last_clean":"2024-04-25T17:35:54.542396+0000","last_became_active":"2024-04-25T17:32:35.076701+0000","last_became_peered":"2024-04-25T17:32:35.076701+0000","last_unstale":"2024-04-25T17:35:54.542396+0000","last_undegraded":"2024-04-25T17:35:54.542396+0000","last_fullsized":"2024-04-25T17:35:54.542396+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:06.646419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416631+0000","last_change":"2024-04-25T17:33:10.610032+0000","last_active":"2024-04-25T17:35:54.416631+0000","last_peered":"2024-04-25T17:35:54.416631+0000","last_clean":"2024-04-25T17:35:54.416631+0000","last_became_active":"2024-04-25T17:33:10.607260+0000","last_became_peered":"2024-04-25T17:33:10.607260+0000","last_unstale":"2024-04-25T17:35:54.416631+0000","last_undegraded":"2024-04-25T17:35:54.416631+0000","last_fullsized":"2024-04-25T17:35:54.416631+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:42:41.509257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542556+0000","last_change":"2024-04-25T17:32:35.071351+0000","last_active":"2024-04-25T17:35:54.542556+0000","last_peered":"2024-04-25T17:35:54.542556+0000","last_clean":"2024-04-25T17:35:54.542556+0000","last_became_active":"2024-04-25T17:32:35.070559+0000","last_became_peered":"2024-04-25T17:32:35.070559+0000","last_unstale":"2024-04-25T17:35:54.542556+0000","last_undegraded":"2024-04-25T17:35:54.542556+0000","last_fullsized":"2024-04-25T17:35:54.542556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:23.864826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549145+0000","last_change":"2024-04-25T17:32:35.073326+0000","last_active":"2024-04-25T17:35:54.549145+0000","last_peered":"2024-04-25T17:35:54.549145+0000","last_clean":"2024-04-25T17:35:54.549145+0000","last_became_active":"2024-04-25T17:32:35.072904+0000","last_became_peered":"2024-04-25T17:32:35.072904+0000","last_unstale":"2024-04-25T17:35:54.549145+0000","last_undegraded":"2024-04-25T17:35:54.549145+0000","last_fullsized":"2024-04-25T17:35:54.549145+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:50.586127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352580+0000","last_change":"2024-04-25T17:33:10.195444+0000","last_active":"2024-04-25T17:35:54.352580+0000","last_peered":"2024-04-25T17:35:54.352580+0000","last_clean":"2024-04-25T17:35:54.352580+0000","last_became_active":"2024-04-25T17:33:10.195048+0000","last_became_peered":"2024-04-25T17:33:10.195048+0000","last_unstale":"2024-04-25T17:35:54.352580+0000","last_undegraded":"2024-04-25T17:35:54.352580+0000","last_fullsized":"2024-04-25T17:35:54.352580+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:03.098618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352519+0000","last_change":"2024-04-25T17:32:35.072053+0000","last_active":"2024-04-25T17:35:54.352519+0000","last_peered":"2024-04-25T17:35:54.352519+0000","last_clean":"2024-04-25T17:35:54.352519+0000","last_became_active":"2024-04-25T17:32:35.071712+0000","last_became_peered":"2024-04-25T17:32:35.071712+0000","last_unstale":"2024-04-25T17:35:54.352519+0000","last_undegraded":"2024-04-25T17:35:54.352519+0000","last_fullsized":"2024-04-25T17:35:54.352519+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:42.928854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542632+0000","last_change":"2024-04-25T17:32:35.061408+0000","last_active":"2024-04-25T17:35:54.542632+0000","last_peered":"2024-04-25T17:35:54.542632+0000","last_clean":"2024-04-25T17:35:54.542632+0000","last_became_active":"2024-04-25T17:32:35.061105+0000","last_became_peered":"2024-04-25T17:32:35.061105+0000","last_unstale":"2024-04-25T17:35:54.542632+0000","last_undegraded":"2024-04-25T17:35:54.542632+0000","last_fullsized":"2024-04-25T17:35:54.542632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:48.242157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416347+0000","last_change":"2024-04-25T17:33:10.687122+0000","last_active":"2024-04-25T17:35:54.416347+0000","last_peered":"2024-04-25T17:35:54.416347+0000","last_clean":"2024-04-25T17:35:54.416347+0000","last_became_active":"2024-04-25T17:33:10.686755+0000","last_became_peered":"2024-04-25T17:33:10.686755+0000","last_unstale":"2024-04-25T17:35:54.416347+0000","last_undegraded":"2024-04-25T17:35:54.416347+0000","last_fullsized":"2024-04-25T17:35:54.416347+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.999655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549326+0000","last_change":"2024-04-25T17:32:35.077433+0000","last_active":"2024-04-25T17:35:54.549326+0000","last_peered":"2024-04-25T17:35:54.549326+0000","last_clean":"2024-04-25T17:35:54.549326+0000","last_became_active":"2024-04-25T17:32:35.076201+0000","last_became_peered":"2024-04-25T17:32:35.076201+0000","last_unstale":"2024-04-25T17:35:54.549326+0000","last_undegraded":"2024-04-25T17:35:54.549326+0000","last_fullsized":"2024-04-25T17:35:54.549326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:42.677022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352836+0000","last_change":"2024-04-25T17:32:35.068017+0000","last_active":"2024-04-25T17:35:54.352836+0000","last_peered":"2024-04-25T17:35:54.352836+0000","last_clean":"2024-04-25T17:35:54.352836+0000","last_became_active":"2024-04-25T17:32:35.067647+0000","last_became_peered":"2024-04-25T17:32:35.067647+0000","last_unstale":"2024-04-25T17:35:54.352836+0000","last_undegraded":"2024-04-25T17:35:54.352836+0000","last_fullsized":"2024-04-25T17:35:54.352836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:28:11.339402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395781+0000","last_change":"2024-04-25T17:32:35.075192+0000","last_active":"2024-04-25T17:35:54.395781+0000","last_peered":"2024-04-25T17:35:54.395781+0000","last_clean":"2024-04-25T17:35:54.395781+0000","last_became_active":"2024-04-25T17:32:35.074850+0000","last_became_peered":"2024-04-25T17:32:35.074850+0000","last_unstale":"2024-04-25T17:35:54.395781+0000","last_undegraded":"2024-04-25T17:35:54.395781+0000","last_fullsized":"2024-04-25T17:35:54.395781+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:04:41.343312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402408+0000","last_change":"2024-04-25T17:32:35.078324+0000","last_active":"2024-04-25T17:35:54.402408+0000","last_peered":"2024-04-25T17:35:54.402408+0000","last_clean":"2024-04-25T17:35:54.402408+0000","last_became_active":"2024-04-25T17:32:35.077958+0000","last_became_peered":"2024-04-25T17:32:35.077958+0000","last_unstale":"2024-04-25T17:35:54.402408+0000","last_undegraded":"2024-04-25T17:35:54.402408+0000","last_fullsized":"2024-04-25T17:35:54.402408+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:10.148060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395538+0000","last_change":"2024-04-25T17:32:35.078023+0000","last_active":"2024-04-25T17:35:54.395538+0000","last_peered":"2024-04-25T17:35:54.395538+0000","last_clean":"2024-04-25T17:35:54.395538+0000","last_became_active":"2024-04-25T17:32:35.077702+0000","last_became_peered":"2024-04-25T17:32:35.077702+0000","last_unstale":"2024-04-25T17:35:54.395538+0000","last_undegraded":"2024-04-25T17:35:54.395538+0000","last_fullsized":"2024-04-25T17:35:54.395538+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:44:37.933179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352546+0000","last_change":"2024-04-25T17:32:35.078388+0000","last_active":"2024-04-25T17:35:54.352546+0000","last_peered":"2024-04-25T17:35:54.352546+0000","last_clean":"2024-04-25T17:35:54.352546+0000","last_became_active":"2024-04-25T17:32:35.078058+0000","last_became_peered":"2024-04-25T17:32:35.078058+0000","last_unstale":"2024-04-25T17:35:54.352546+0000","last_undegraded":"2024-04-25T17:35:54.352546+0000","last_fullsized":"2024-04-25T17:35:54.352546+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:05.699542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416109+0000","last_change":"2024-04-25T17:32:35.068532+0000","last_active":"2024-04-25T17:35:54.416109+0000","last_peered":"2024-04-25T17:35:54.416109+0000","last_clean":"2024-04-25T17:35:54.416109+0000","last_became_active":"2024-04-25T17:32:35.068225+0000","last_became_peered":"2024-04-25T17:32:35.068225+0000","last_unstale":"2024-04-25T17:35:54.416109+0000","last_undegraded":"2024-04-25T17:35:54.416109+0000","last_fullsized":"2024-04-25T17:35:54.416109+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:09.911999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549091+0000","last_change":"2024-04-25T17:32:35.065526+0000","last_active":"2024-04-25T17:35:54.549091+0000","last_peered":"2024-04-25T17:35:54.549091+0000","last_clean":"2024-04-25T17:35:54.549091+0000","last_became_active":"2024-04-25T17:32:35.065146+0000","last_became_peered":"2024-04-25T17:32:35.065146+0000","last_unstale":"2024-04-25T17:35:54.549091+0000","last_undegraded":"2024-04-25T17:35:54.549091+0000","last_fullsized":"2024-04-25T17:35:54.549091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:33.662055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106941,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598036,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1428,"num_read_kb":915,"num_write":396,"num_write_kb":1889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602112,"data_stored":1990740,"data_compressed":85238,"data_compressed_allocated":110592,"data_compressed_original":1732608,"omap_allocated":0,"internal_metadata":0},"log_size":284,"ondisk_log_size":284,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351030,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":491851,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612669,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":1283843,"data_compressed":44702,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776196,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":885303,"data_compressed":22188,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037830,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":492007,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299469,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1064960,"data_stored":1085559,"data_compressed":38546,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462994,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":491476,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":558866,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":492747,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":560915,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":1213235,"data_compressed":37390,"data_compressed_allocated":49152,"data_compressed_original":708608,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888156,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":492445,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724632,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":1215529,"data_compressed":44702,"data_compressed_allocated":57344,"data_compressed_original":720896,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":133775,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":202089,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T17:36:02.418 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T17:36:02.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T17:36:02.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:02 smithi173 ceph-mon[32856]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:36:02.658 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:02.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:02 smithi098 ceph-mon[29192]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:36:02.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:02 smithi203 ceph-mon[32834]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:36:03.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:03 smithi098 ceph-mon[29192]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-25T17:36:03.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:03 smithi173 ceph-mon[32856]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-25T17:36:03.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:03 smithi203 ceph-mon[32834]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-25T17:36:04.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.326+0000 7efc59a2e700 1 -- 172.21.15.98:0/2863415433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc54100f90 msgr2=0x7efc54101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:04.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.326+0000 7efc59a2e700 1 --2- 172.21.15.98:0/2863415433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc54100f90 0x7efc54101410 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7efc480099c0 tx=0x7efc48009cd0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.326+0000 7efc59a2e700 1 -- 172.21.15.98:0/2863415433 shutdown_connections 2024-04-25T17:36:04.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.326+0000 7efc59a2e700 1 --2- 172.21.15.98:0/2863415433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc54101950 0x7efc5410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.326+0000 7efc59a2e700 1 --2- 172.21.15.98:0/2863415433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc54107a60 0x7efc54107e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.326+0000 7efc59a2e700 1 --2- 172.21.15.98:0/2863415433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc54100f90 0x7efc54101410 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.326+0000 7efc59a2e700 1 -- 172.21.15.98:0/2863415433 >> 172.21.15.98:0/2863415433 conn(0x7efc54077f40 msgr2=0x7efc540ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:04.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.326+0000 7efc59a2e700 1 -- 172.21.15.98:0/2863415433 shutdown_connections 2024-04-25T17:36:04.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.326+0000 7efc59a2e700 1 -- 172.21.15.98:0/2863415433 wait complete. 2024-04-25T17:36:04.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc59a2e700 1 Processor -- start 2024-04-25T17:36:04.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc59a2e700 1 -- start start 2024-04-25T17:36:04.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc59a2e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc54100f90 0x7efc54199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc59a2e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc54101950 0x7efc54199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc59a2e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc54107a60 0x7efc5419dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc59a2e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc54112d30 con 0x7efc54100f90 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc59a2e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc54112bb0 con 0x7efc54101950 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc59a2e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efc54112eb0 con 0x7efc54107a60 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc52ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc54100f90 0x7efc54199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc52ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc54100f90 0x7efc54199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33614/0 (socket says 172.21.15.98:33614) 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc52ffd700 1 -- 172.21.15.98:0/1161699159 learned_addr learned my addr 172.21.15.98:0/1161699159 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc537fe700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc54107a60 0x7efc5419dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.327+0000 7efc527fc700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc54101950 0x7efc54199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc52ffd700 1 -- 172.21.15.98:0/1161699159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc54107a60 msgr2=0x7efc5419dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc52ffd700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc54107a60 0x7efc5419dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc52ffd700 1 -- 172.21.15.98:0/1161699159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc54101950 msgr2=0x7efc54199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:04.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc52ffd700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc54101950 0x7efc54199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc52ffd700 1 -- 172.21.15.98:0/1161699159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc5419e210 con 0x7efc54100f90 2024-04-25T17:36:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc52ffd700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc54100f90 0x7efc54199200 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0060b0 tx=0x7efc4c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc58a2c700 1 -- 172.21.15.98:0/1161699159 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c01a070 con 0x7efc54100f90 2024-04-25T17:36:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc58a2c700 1 -- 172.21.15.98:0/1161699159 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc4c00ddd0 con 0x7efc54100f90 2024-04-25T17:36:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc59a2e700 1 -- 172.21.15.98:0/1161699159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc5419e500 con 0x7efc54100f90 2024-04-25T17:36:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc58a2c700 1 -- 172.21.15.98:0/1161699159 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c014720 con 0x7efc54100f90 2024-04-25T17:36:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.328+0000 7efc59a2e700 1 -- 172.21.15.98:0/1161699159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc541a5d80 con 0x7efc54100f90 2024-04-25T17:36:04.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.329+0000 7efc59a2e700 1 -- 172.21.15.98:0/1161699159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc54069730 con 0x7efc54100f90 2024-04-25T17:36:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.330+0000 7efc58a2c700 1 -- 172.21.15.98:0/1161699159 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7efc4c01d070 con 0x7efc54100f90 2024-04-25T17:36:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.331+0000 7efc58a2c700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7efc3c0a3ed0 0x7efc3c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.331+0000 7efc527fc700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7efc3c0a3ed0 0x7efc3c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.331+0000 7efc58a2c700 1 -- 172.21.15.98:0/1161699159 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7efc4c0d0740 con 0x7efc54100f90 2024-04-25T17:36:04.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.331+0000 7efc527fc700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7efc3c0a3ed0 0x7efc3c0a6390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efc54100500 tx=0x7efc4801a040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:04.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.335+0000 7efc58a2c700 1 -- 172.21.15.98:0/1161699159 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc4c010070 con 0x7efc54100f90 2024-04-25T17:36:04.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:04 smithi098 ceph-mon[29192]: mgrmap e25: x(active, since 8s), standbys: z, y 2024-04-25T17:36:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:04 smithi173 ceph-mon[32856]: mgrmap e25: x(active, since 8s), standbys: z, y 2024-04-25T17:36:04.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.530+0000 7efc59a2e700 1 -- 172.21.15.98:0/1161699159 --> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efc5419a560 con 0x7efc3c0a3ed0 2024-04-25T17:36:04.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.537+0000 7efc58a2c700 1 -- 172.21.15.98:0/1161699159 <== mgr.15129 v2:172.21.15.98:6800/844767972 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258036 (secure 0 0 0) 0x7efc5419a560 con 0x7efc3c0a3ed0 2024-04-25T17:36:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 -- 172.21.15.98:0/1161699159 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7efc3c0a3ed0 msgr2=0x7efc3c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7efc3c0a3ed0 0x7efc3c0a6390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efc54100500 tx=0x7efc4801a040 comp rx=0 tx=0).stop 2024-04-25T17:36:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 -- 172.21.15.98:0/1161699159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc54100f90 msgr2=0x7efc54199200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc54100f90 0x7efc54199200 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0060b0 tx=0x7efc4c00d3b0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 -- 172.21.15.98:0/1161699159 shutdown_connections 2024-04-25T17:36:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7efc3c0a3ed0 0x7efc3c0a6390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc54107a60 0x7efc5419dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc54100f90 0x7efc54199200 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 --2- 172.21.15.98:0/1161699159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc54101950 0x7efc54199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:04.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 -- 172.21.15.98:0/1161699159 >> 172.21.15.98:0/1161699159 conn(0x7efc54077f40 msgr2=0x7efc54105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 -- 172.21.15.98:0/1161699159 shutdown_connections 2024-04-25T17:36:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:04.540+0000 7efc59a2e700 1 -- 172.21.15.98:0/1161699159 wait complete. 2024-04-25T17:36:04.544 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T17:36:04.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:04 smithi203 ceph-mon[32834]: mgrmap e25: x(active, since 8s), standbys: z, y 2024-04-25T17:36:05.287 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-25T17:36:04.321392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1385270,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1721,"num_read_kb":1286,"num_write":626,"num_write_kb":5295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":960,"ondisk_log_size":960,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357512,"kb_used_data":12468,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584312,"statfs":{"total":1151940427776,"available":1151574335488,"internally_reserved":0,"allocated":12767232,"data_stored":10051038,"data_compressed":245202,"data_compressed_allocated":319488,"data_compressed_original":3969024,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":387,"num_read_kb":459,"num_write":146,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.992485"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549366+0000","last_change":"2024-04-25T17:32:35.075338+0000","last_active":"2024-04-25T17:35:54.549366+0000","last_peered":"2024-04-25T17:35:54.549366+0000","last_clean":"2024-04-25T17:35:54.549366+0000","last_became_active":"2024-04-25T17:32:35.075029+0000","last_became_peered":"2024-04-25T17:32:35.075029+0000","last_unstale":"2024-04-25T17:35:54.549366+0000","last_undegraded":"2024-04-25T17:35:54.549366+0000","last_fullsized":"2024-04-25T17:35:54.549366+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:23.837337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543054+0000","last_change":"2024-04-25T17:32:35.075422+0000","last_active":"2024-04-25T17:35:54.543054+0000","last_peered":"2024-04-25T17:35:54.543054+0000","last_clean":"2024-04-25T17:35:54.543054+0000","last_became_active":"2024-04-25T17:32:35.071975+0000","last_became_peered":"2024-04-25T17:32:35.071975+0000","last_unstale":"2024-04-25T17:35:54.543054+0000","last_undegraded":"2024-04-25T17:35:54.543054+0000","last_fullsized":"2024-04-25T17:35:54.543054+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:20.548869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542762+0000","last_change":"2024-04-25T17:32:35.068153+0000","last_active":"2024-04-25T17:35:54.542762+0000","last_peered":"2024-04-25T17:35:54.542762+0000","last_clean":"2024-04-25T17:35:54.542762+0000","last_became_active":"2024-04-25T17:32:35.065873+0000","last_became_peered":"2024-04-25T17:32:35.065873+0000","last_unstale":"2024-04-25T17:35:54.542762+0000","last_undegraded":"2024-04-25T17:35:54.542762+0000","last_fullsized":"2024-04-25T17:35:54.542762+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:31.381875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310699+0000","last_change":"2024-04-25T17:32:35.073055+0000","last_active":"2024-04-25T17:35:54.310699+0000","last_peered":"2024-04-25T17:35:54.310699+0000","last_clean":"2024-04-25T17:35:54.310699+0000","last_became_active":"2024-04-25T17:32:35.072596+0000","last_became_peered":"2024-04-25T17:32:35.072596+0000","last_unstale":"2024-04-25T17:35:54.310699+0000","last_undegraded":"2024-04-25T17:35:54.310699+0000","last_fullsized":"2024-04-25T17:35:54.310699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:37.529698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309402+0000","last_change":"2024-04-25T17:32:35.072721+0000","last_active":"2024-04-25T17:35:54.309402+0000","last_peered":"2024-04-25T17:35:54.309402+0000","last_clean":"2024-04-25T17:35:54.309402+0000","last_became_active":"2024-04-25T17:32:35.072370+0000","last_became_peered":"2024-04-25T17:32:35.072370+0000","last_unstale":"2024-04-25T17:35:54.309402+0000","last_undegraded":"2024-04-25T17:35:54.309402+0000","last_fullsized":"2024-04-25T17:35:54.309402+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:34.429985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350742+0000","last_change":"2024-04-25T17:32:35.071538+0000","last_active":"2024-04-25T17:35:54.350742+0000","last_peered":"2024-04-25T17:35:54.350742+0000","last_clean":"2024-04-25T17:35:54.350742+0000","last_became_active":"2024-04-25T17:32:35.067427+0000","last_became_peered":"2024-04-25T17:32:35.067427+0000","last_unstale":"2024-04-25T17:35:54.350742+0000","last_undegraded":"2024-04-25T17:35:54.350742+0000","last_fullsized":"2024-04-25T17:35:54.350742+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:33:36.052762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542299+0000","last_change":"2024-04-25T17:32:35.079476+0000","last_active":"2024-04-25T17:35:54.542299+0000","last_peered":"2024-04-25T17:35:54.542299+0000","last_clean":"2024-04-25T17:35:54.542299+0000","last_became_active":"2024-04-25T17:32:35.079170+0000","last_became_peered":"2024-04-25T17:32:35.079170+0000","last_unstale":"2024-04-25T17:35:54.542299+0000","last_undegraded":"2024-04-25T17:35:54.542299+0000","last_fullsized":"2024-04-25T17:35:54.542299+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:30.937670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395435+0000","last_change":"2024-04-25T17:32:35.077261+0000","last_active":"2024-04-25T17:35:54.395435+0000","last_peered":"2024-04-25T17:35:54.395435+0000","last_clean":"2024-04-25T17:35:54.395435+0000","last_became_active":"2024-04-25T17:32:35.076896+0000","last_became_peered":"2024-04-25T17:32:35.076896+0000","last_unstale":"2024-04-25T17:35:54.395435+0000","last_undegraded":"2024-04-25T17:35:54.395435+0000","last_fullsized":"2024-04-25T17:35:54.395435+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:24.082550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352775+0000","last_change":"2024-04-25T17:32:35.072229+0000","last_active":"2024-04-25T17:35:54.352775+0000","last_peered":"2024-04-25T17:35:54.352775+0000","last_clean":"2024-04-25T17:35:54.352775+0000","last_became_active":"2024-04-25T17:32:35.071586+0000","last_became_peered":"2024-04-25T17:32:35.071586+0000","last_unstale":"2024-04-25T17:35:54.352775+0000","last_undegraded":"2024-04-25T17:35:54.352775+0000","last_fullsized":"2024-04-25T17:35:54.352775+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:08:01.195638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.351018+0000","last_change":"2024-04-25T17:32:35.075274+0000","last_active":"2024-04-25T17:35:54.351018+0000","last_peered":"2024-04-25T17:35:54.351018+0000","last_clean":"2024-04-25T17:35:54.351018+0000","last_became_active":"2024-04-25T17:32:35.073821+0000","last_became_peered":"2024-04-25T17:32:35.073821+0000","last_unstale":"2024-04-25T17:35:54.351018+0000","last_undegraded":"2024-04-25T17:35:54.351018+0000","last_fullsized":"2024-04-25T17:35:54.351018+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:43:04.145158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542610+0000","last_change":"2024-04-25T17:32:35.080747+0000","last_active":"2024-04-25T17:35:54.542610+0000","last_peered":"2024-04-25T17:35:54.542610+0000","last_clean":"2024-04-25T17:35:54.542610+0000","last_became_active":"2024-04-25T17:32:35.080383+0000","last_became_peered":"2024-04-25T17:32:35.080383+0000","last_unstale":"2024-04-25T17:35:54.542610+0000","last_undegraded":"2024-04-25T17:35:54.542610+0000","last_fullsized":"2024-04-25T17:35:54.542610+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:42.128892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310582+0000","last_change":"2024-04-25T17:32:35.061442+0000","last_active":"2024-04-25T17:35:54.310582+0000","last_peered":"2024-04-25T17:35:54.310582+0000","last_clean":"2024-04-25T17:35:54.310582+0000","last_became_active":"2024-04-25T17:32:35.061030+0000","last_became_peered":"2024-04-25T17:32:35.061030+0000","last_unstale":"2024-04-25T17:35:54.310582+0000","last_undegraded":"2024-04-25T17:35:54.310582+0000","last_fullsized":"2024-04-25T17:35:54.310582+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:09:03.374809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416138+0000","last_change":"2024-04-25T17:32:35.069698+0000","last_active":"2024-04-25T17:35:54.416138+0000","last_peered":"2024-04-25T17:35:54.416138+0000","last_clean":"2024-04-25T17:35:54.416138+0000","last_became_active":"2024-04-25T17:32:35.069374+0000","last_became_peered":"2024-04-25T17:32:35.069374+0000","last_unstale":"2024-04-25T17:35:54.416138+0000","last_undegraded":"2024-04-25T17:35:54.416138+0000","last_fullsized":"2024-04-25T17:35:54.416138+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:03.214525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542531+0000","last_change":"2024-04-25T17:32:35.069001+0000","last_active":"2024-04-25T17:35:54.542531+0000","last_peered":"2024-04-25T17:35:54.542531+0000","last_clean":"2024-04-25T17:35:54.542531+0000","last_became_active":"2024-04-25T17:32:35.064882+0000","last_became_peered":"2024-04-25T17:32:35.064882+0000","last_unstale":"2024-04-25T17:35:54.542531+0000","last_undegraded":"2024-04-25T17:35:54.542531+0000","last_fullsized":"2024-04-25T17:35:54.542531+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:32.122251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542441+0000","last_change":"2024-04-25T17:32:35.065042+0000","last_active":"2024-04-25T17:35:54.542441+0000","last_peered":"2024-04-25T17:35:54.542441+0000","last_clean":"2024-04-25T17:35:54.542441+0000","last_became_active":"2024-04-25T17:32:35.064719+0000","last_became_peered":"2024-04-25T17:32:35.064719+0000","last_unstale":"2024-04-25T17:35:54.542441+0000","last_undegraded":"2024-04-25T17:35:54.542441+0000","last_fullsized":"2024-04-25T17:35:54.542441+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:23:54.715385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352288+0000","last_change":"2024-04-25T17:32:35.067390+0000","last_active":"2024-04-25T17:35:54.352288+0000","last_peered":"2024-04-25T17:35:54.352288+0000","last_clean":"2024-04-25T17:35:54.352288+0000","last_became_active":"2024-04-25T17:32:35.066846+0000","last_became_peered":"2024-04-25T17:32:35.066846+0000","last_unstale":"2024-04-25T17:35:54.352288+0000","last_undegraded":"2024-04-25T17:35:54.352288+0000","last_fullsized":"2024-04-25T17:35:54.352288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:29:24.305767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549475+0000","last_change":"2024-04-25T17:32:35.075349+0000","last_active":"2024-04-25T17:35:54.549475+0000","last_peered":"2024-04-25T17:35:54.549475+0000","last_clean":"2024-04-25T17:35:54.549475+0000","last_became_active":"2024-04-25T17:32:35.075046+0000","last_became_peered":"2024-04-25T17:32:35.075046+0000","last_unstale":"2024-04-25T17:35:54.549475+0000","last_undegraded":"2024-04-25T17:35:54.549475+0000","last_fullsized":"2024-04-25T17:35:54.549475+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:51.703406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307381+0000","last_change":"2024-04-25T17:32:35.077597+0000","last_active":"2024-04-25T17:35:54.307381+0000","last_peered":"2024-04-25T17:35:54.307381+0000","last_clean":"2024-04-25T17:35:54.307381+0000","last_became_active":"2024-04-25T17:32:35.077232+0000","last_became_peered":"2024-04-25T17:32:35.077232+0000","last_unstale":"2024-04-25T17:35:54.307381+0000","last_undegraded":"2024-04-25T17:35:54.307381+0000","last_fullsized":"2024-04-25T17:35:54.307381+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:46.842673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352862+0000","last_change":"2024-04-25T17:32:35.078229+0000","last_active":"2024-04-25T17:35:54.352862+0000","last_peered":"2024-04-25T17:35:54.352862+0000","last_clean":"2024-04-25T17:35:54.352862+0000","last_became_active":"2024-04-25T17:32:35.077899+0000","last_became_peered":"2024-04-25T17:32:35.077899+0000","last_unstale":"2024-04-25T17:35:54.352862+0000","last_undegraded":"2024-04-25T17:35:54.352862+0000","last_fullsized":"2024-04-25T17:35:54.352862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:46:22.776371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307342+0000","last_change":"2024-04-25T17:32:32.045385+0000","last_active":"2024-04-25T17:35:54.307342+0000","last_peered":"2024-04-25T17:35:54.307342+0000","last_clean":"2024-04-25T17:35:54.307342+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:35:54.307342+0000","last_undegraded":"2024-04-25T17:35:54.307342+0000","last_fullsized":"2024-04-25T17:35:54.307342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:29.732001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395800+0000","last_change":"2024-04-25T17:32:35.079966+0000","last_active":"2024-04-25T17:35:54.395800+0000","last_peered":"2024-04-25T17:35:54.395800+0000","last_clean":"2024-04-25T17:35:54.395800+0000","last_became_active":"2024-04-25T17:32:35.079649+0000","last_became_peered":"2024-04-25T17:32:35.079649+0000","last_unstale":"2024-04-25T17:35:54.395800+0000","last_undegraded":"2024-04-25T17:35:54.395800+0000","last_fullsized":"2024-04-25T17:35:54.395800+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:19:45.679273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.396029+0000","last_change":"2024-04-25T17:32:32.285667+0000","last_active":"2024-04-25T17:35:54.396029+0000","last_peered":"2024-04-25T17:35:54.396029+0000","last_clean":"2024-04-25T17:35:54.396029+0000","last_became_active":"2024-04-25T17:32:32.285330+0000","last_became_peered":"2024-04-25T17:32:32.285330+0000","last_unstale":"2024-04-25T17:35:54.396029+0000","last_undegraded":"2024-04-25T17:35:54.396029+0000","last_fullsized":"2024-04-25T17:35:54.396029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:16.975280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543368+0000","last_change":"2024-04-25T17:32:35.068841+0000","last_active":"2024-04-25T17:35:54.543368+0000","last_peered":"2024-04-25T17:35:54.543368+0000","last_clean":"2024-04-25T17:35:54.543368+0000","last_became_active":"2024-04-25T17:32:35.068518+0000","last_became_peered":"2024-04-25T17:32:35.068518+0000","last_unstale":"2024-04-25T17:35:54.543368+0000","last_undegraded":"2024-04-25T17:35:54.543368+0000","last_fullsized":"2024-04-25T17:35:54.543368+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:23.290990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309712+0000","last_change":"2024-04-25T17:32:32.045627+0000","last_active":"2024-04-25T17:35:54.309712+0000","last_peered":"2024-04-25T17:35:54.309712+0000","last_clean":"2024-04-25T17:35:54.309712+0000","last_became_active":"2024-04-25T17:32:32.044971+0000","last_became_peered":"2024-04-25T17:32:32.044971+0000","last_unstale":"2024-04-25T17:35:54.309712+0000","last_undegraded":"2024-04-25T17:35:54.309712+0000","last_fullsized":"2024-04-25T17:35:54.309712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:50.590435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352609+0000","last_change":"2024-04-25T17:32:35.070862+0000","last_active":"2024-04-25T17:35:54.352609+0000","last_peered":"2024-04-25T17:35:54.352609+0000","last_clean":"2024-04-25T17:35:54.352609+0000","last_became_active":"2024-04-25T17:32:35.069927+0000","last_became_peered":"2024-04-25T17:32:35.069927+0000","last_unstale":"2024-04-25T17:35:54.352609+0000","last_undegraded":"2024-04-25T17:35:54.352609+0000","last_fullsized":"2024-04-25T17:35:54.352609+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:40:01.799015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309802+0000","last_change":"2024-04-25T17:32:32.046062+0000","last_active":"2024-04-25T17:35:54.309802+0000","last_peered":"2024-04-25T17:35:54.309802+0000","last_clean":"2024-04-25T17:35:54.309802+0000","last_became_active":"2024-04-25T17:32:32.044903+0000","last_became_peered":"2024-04-25T17:32:32.044903+0000","last_unstale":"2024-04-25T17:35:54.309802+0000","last_undegraded":"2024-04-25T17:35:54.309802+0000","last_fullsized":"2024-04-25T17:35:54.309802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:51.099460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402607+0000","last_change":"2024-04-25T17:32:35.075452+0000","last_active":"2024-04-25T17:35:54.402607+0000","last_peered":"2024-04-25T17:35:54.402607+0000","last_clean":"2024-04-25T17:35:54.402607+0000","last_became_active":"2024-04-25T17:32:35.075136+0000","last_became_peered":"2024-04-25T17:32:35.075136+0000","last_unstale":"2024-04-25T17:35:54.402607+0000","last_undegraded":"2024-04-25T17:35:54.402607+0000","last_fullsized":"2024-04-25T17:35:54.402607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:02.290926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":224,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:58.420950+0000","last_change":"2024-04-25T17:32:32.042019+0000","last_active":"2024-04-25T17:35:58.420950+0000","last_peered":"2024-04-25T17:35:58.420950+0000","last_clean":"2024-04-25T17:35:58.420950+0000","last_became_active":"2024-04-25T17:32:32.041641+0000","last_became_peered":"2024-04-25T17:32:32.041641+0000","last_unstale":"2024-04-25T17:35:58.420950+0000","last_undegraded":"2024-04-25T17:35:58.420950+0000","last_fullsized":"2024-04-25T17:35:58.420950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:02.025915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309350+0000","last_change":"2024-04-25T17:32:35.069715+0000","last_active":"2024-04-25T17:35:54.309350+0000","last_peered":"2024-04-25T17:35:54.309350+0000","last_clean":"2024-04-25T17:35:54.309350+0000","last_became_active":"2024-04-25T17:32:35.069339+0000","last_became_peered":"2024-04-25T17:32:35.069339+0000","last_unstale":"2024-04-25T17:35:54.309350+0000","last_undegraded":"2024-04-25T17:35:54.309350+0000","last_fullsized":"2024-04-25T17:35:54.309350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:48:13.572261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350662+0000","last_change":"2024-04-25T17:32:32.042445+0000","last_active":"2024-04-25T17:35:54.350662+0000","last_peered":"2024-04-25T17:35:54.350662+0000","last_clean":"2024-04-25T17:35:54.350662+0000","last_became_active":"2024-04-25T17:32:32.042033+0000","last_became_peered":"2024-04-25T17:32:32.042033+0000","last_unstale":"2024-04-25T17:35:54.350662+0000","last_undegraded":"2024-04-25T17:35:54.350662+0000","last_fullsized":"2024-04-25T17:35:54.350662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.968383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402497+0000","last_change":"2024-04-25T17:32:35.074492+0000","last_active":"2024-04-25T17:35:54.402497+0000","last_peered":"2024-04-25T17:35:54.402497+0000","last_clean":"2024-04-25T17:35:54.402497+0000","last_became_active":"2024-04-25T17:32:35.073652+0000","last_became_peered":"2024-04-25T17:32:35.073652+0000","last_unstale":"2024-04-25T17:35:54.402497+0000","last_undegraded":"2024-04-25T17:35:54.402497+0000","last_fullsized":"2024-04-25T17:35:54.402497+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:00.300453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415757+0000","last_change":"2024-04-25T17:32:32.039568+0000","last_active":"2024-04-25T17:35:54.415757+0000","last_peered":"2024-04-25T17:35:54.415757+0000","last_clean":"2024-04-25T17:35:54.415757+0000","last_became_active":"2024-04-25T17:32:32.039228+0000","last_became_peered":"2024-04-25T17:32:32.039228+0000","last_unstale":"2024-04-25T17:35:54.415757+0000","last_undegraded":"2024-04-25T17:35:54.415757+0000","last_fullsized":"2024-04-25T17:35:54.415757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:17:29.079749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395508+0000","last_change":"2024-04-25T17:32:35.077619+0000","last_active":"2024-04-25T17:35:54.395508+0000","last_peered":"2024-04-25T17:35:54.395508+0000","last_clean":"2024-04-25T17:35:54.395508+0000","last_became_active":"2024-04-25T17:32:35.077309+0000","last_became_peered":"2024-04-25T17:32:35.077309+0000","last_unstale":"2024-04-25T17:35:54.395508+0000","last_undegraded":"2024-04-25T17:35:54.395508+0000","last_fullsized":"2024-04-25T17:35:54.395508+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:54:41.245848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402343+0000","last_change":"2024-04-25T17:32:32.044650+0000","last_active":"2024-04-25T17:35:54.402343+0000","last_peered":"2024-04-25T17:35:54.402343+0000","last_clean":"2024-04-25T17:35:54.402343+0000","last_became_active":"2024-04-25T17:32:32.043999+0000","last_became_peered":"2024-04-25T17:32:32.043999+0000","last_unstale":"2024-04-25T17:35:54.402343+0000","last_undegraded":"2024-04-25T17:35:54.402343+0000","last_fullsized":"2024-04-25T17:35:54.402343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:19.463941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307201+0000","last_change":"2024-04-25T17:32:35.080598+0000","last_active":"2024-04-25T17:35:54.307201+0000","last_peered":"2024-04-25T17:35:54.307201+0000","last_clean":"2024-04-25T17:35:54.307201+0000","last_became_active":"2024-04-25T17:32:35.080300+0000","last_became_peered":"2024-04-25T17:32:35.080300+0000","last_unstale":"2024-04-25T17:35:54.307201+0000","last_undegraded":"2024-04-25T17:35:54.307201+0000","last_fullsized":"2024-04-25T17:35:54.307201+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:33:42.149746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":217,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:58.420487+0000","last_change":"2024-04-25T17:32:32.043889+0000","last_active":"2024-04-25T17:35:58.420487+0000","last_peered":"2024-04-25T17:35:58.420487+0000","last_clean":"2024-04-25T17:35:58.420487+0000","last_became_active":"2024-04-25T17:32:32.043520+0000","last_became_peered":"2024-04-25T17:32:32.043520+0000","last_unstale":"2024-04-25T17:35:58.420487+0000","last_undegraded":"2024-04-25T17:35:58.420487+0000","last_fullsized":"2024-04-25T17:35:58.420487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:40:25.134797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":184,"num_read_kb":184,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352641+0000","last_change":"2024-04-25T17:32:35.071733+0000","last_active":"2024-04-25T17:35:54.352641+0000","last_peered":"2024-04-25T17:35:54.352641+0000","last_clean":"2024-04-25T17:35:54.352641+0000","last_became_active":"2024-04-25T17:32:35.071383+0000","last_became_peered":"2024-04-25T17:32:35.071383+0000","last_unstale":"2024-04-25T17:35:54.352641+0000","last_undegraded":"2024-04-25T17:35:54.352641+0000","last_fullsized":"2024-04-25T17:35:54.352641+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:09:49.433675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542701+0000","last_change":"2024-04-25T17:32:32.044823+0000","last_active":"2024-04-25T17:35:54.542701+0000","last_peered":"2024-04-25T17:35:54.542701+0000","last_clean":"2024-04-25T17:35:54.542701+0000","last_became_active":"2024-04-25T17:32:32.044497+0000","last_became_peered":"2024-04-25T17:32:32.044497+0000","last_unstale":"2024-04-25T17:35:54.542701+0000","last_undegraded":"2024-04-25T17:35:54.542701+0000","last_fullsized":"2024-04-25T17:35:54.542701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:35:47.811172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543284+0000","last_change":"2024-04-25T17:32:35.071109+0000","last_active":"2024-04-25T17:35:54.543284+0000","last_peered":"2024-04-25T17:35:54.543284+0000","last_clean":"2024-04-25T17:35:54.543284+0000","last_became_active":"2024-04-25T17:32:35.070782+0000","last_became_peered":"2024-04-25T17:32:35.070782+0000","last_unstale":"2024-04-25T17:35:54.543284+0000","last_undegraded":"2024-04-25T17:35:54.543284+0000","last_fullsized":"2024-04-25T17:35:54.543284+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:37.247274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402726+0000","last_change":"2024-04-25T17:32:32.288683+0000","last_active":"2024-04-25T17:35:54.402726+0000","last_peered":"2024-04-25T17:35:54.402726+0000","last_clean":"2024-04-25T17:35:54.402726+0000","last_became_active":"2024-04-25T17:32:32.288261+0000","last_became_peered":"2024-04-25T17:32:32.288261+0000","last_unstale":"2024-04-25T17:35:54.402726+0000","last_undegraded":"2024-04-25T17:35:54.402726+0000","last_fullsized":"2024-04-25T17:35:54.402726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:10:36.502434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352607+0000","last_change":"2024-04-25T17:32:35.060828+0000","last_active":"2024-04-25T17:35:54.352607+0000","last_peered":"2024-04-25T17:35:54.352607+0000","last_clean":"2024-04-25T17:35:54.352607+0000","last_became_active":"2024-04-25T17:32:35.060433+0000","last_became_peered":"2024-04-25T17:32:35.060433+0000","last_unstale":"2024-04-25T17:35:54.352607+0000","last_undegraded":"2024-04-25T17:35:54.352607+0000","last_fullsized":"2024-04-25T17:35:54.352607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:32.382042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":447,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:59.421930+0000","last_change":"2024-04-25T17:32:32.042979+0000","last_active":"2024-04-25T17:35:59.421930+0000","last_peered":"2024-04-25T17:35:59.421930+0000","last_clean":"2024-04-25T17:35:59.421930+0000","last_became_active":"2024-04-25T17:32:32.042257+0000","last_became_peered":"2024-04-25T17:32:32.042257+0000","last_unstale":"2024-04-25T17:35:59.421930+0000","last_undegraded":"2024-04-25T17:35:59.421930+0000","last_fullsized":"2024-04-25T17:35:59.421930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:19.316803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67597,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":382,"num_read_kb":258,"num_write":37,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309526+0000","last_change":"2024-04-25T17:32:35.069994+0000","last_active":"2024-04-25T17:35:54.309526+0000","last_peered":"2024-04-25T17:35:54.309526+0000","last_clean":"2024-04-25T17:35:54.309526+0000","last_became_active":"2024-04-25T17:32:35.069632+0000","last_became_peered":"2024-04-25T17:32:35.069632+0000","last_unstale":"2024-04-25T17:35:54.309526+0000","last_undegraded":"2024-04-25T17:35:54.309526+0000","last_fullsized":"2024-04-25T17:35:54.309526+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:15:51.169489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352625+0000","last_change":"2024-04-25T17:32:32.042652+0000","last_active":"2024-04-25T17:35:54.352625+0000","last_peered":"2024-04-25T17:35:54.352625+0000","last_clean":"2024-04-25T17:35:54.352625+0000","last_became_active":"2024-04-25T17:32:32.042214+0000","last_became_peered":"2024-04-25T17:32:32.042214+0000","last_unstale":"2024-04-25T17:35:54.352625+0000","last_undegraded":"2024-04-25T17:35:54.352625+0000","last_fullsized":"2024-04-25T17:35:54.352625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:43.662580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415731+0000","last_change":"2024-04-25T17:32:35.074449+0000","last_active":"2024-04-25T17:35:54.415731+0000","last_peered":"2024-04-25T17:35:54.415731+0000","last_clean":"2024-04-25T17:35:54.415731+0000","last_became_active":"2024-04-25T17:32:35.074116+0000","last_became_peered":"2024-04-25T17:32:35.074116+0000","last_unstale":"2024-04-25T17:35:54.415731+0000","last_undegraded":"2024-04-25T17:35:54.415731+0000","last_fullsized":"2024-04-25T17:35:54.415731+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:09.233206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402590+0000","last_change":"2024-04-25T17:32:32.041368+0000","last_active":"2024-04-25T17:35:54.402590+0000","last_peered":"2024-04-25T17:35:54.402590+0000","last_clean":"2024-04-25T17:35:54.402590+0000","last_became_active":"2024-04-25T17:32:32.040940+0000","last_became_peered":"2024-04-25T17:32:32.040940+0000","last_unstale":"2024-04-25T17:35:54.402590+0000","last_undegraded":"2024-04-25T17:35:54.402590+0000","last_fullsized":"2024-04-25T17:35:54.402590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:38.871786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542869+0000","last_change":"2024-04-25T17:32:35.073535+0000","last_active":"2024-04-25T17:35:54.542869+0000","last_peered":"2024-04-25T17:35:54.542869+0000","last_clean":"2024-04-25T17:35:54.542869+0000","last_became_active":"2024-04-25T17:32:35.073190+0000","last_became_peered":"2024-04-25T17:32:35.073190+0000","last_unstale":"2024-04-25T17:35:54.542869+0000","last_undegraded":"2024-04-25T17:35:54.542869+0000","last_fullsized":"2024-04-25T17:35:54.542869+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:19.841978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"92'7","reported_seq":225,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:58.421328+0000","last_change":"2024-04-25T17:32:32.044359+0000","last_active":"2024-04-25T17:35:58.421328+0000","last_peered":"2024-04-25T17:35:58.421328+0000","last_clean":"2024-04-25T17:35:58.421328+0000","last_became_active":"2024-04-25T17:32:32.044024+0000","last_became_peered":"2024-04-25T17:32:32.044024+0000","last_unstale":"2024-04-25T17:35:58.421328+0000","last_undegraded":"2024-04-25T17:35:58.421328+0000","last_fullsized":"2024-04-25T17:35:58.421328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:58.940554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":190,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542252+0000","last_change":"2024-04-25T17:33:10.851665+0000","last_active":"2024-04-25T17:35:54.542252+0000","last_peered":"2024-04-25T17:35:54.542252+0000","last_clean":"2024-04-25T17:35:54.542252+0000","last_became_active":"2024-04-25T17:33:10.851217+0000","last_became_peered":"2024-04-25T17:33:10.851217+0000","last_unstale":"2024-04-25T17:35:54.542252+0000","last_undegraded":"2024-04-25T17:35:54.542252+0000","last_fullsized":"2024-04-25T17:35:54.542252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:58.436715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309339+0000","last_change":"2024-04-25T17:32:32.285823+0000","last_active":"2024-04-25T17:35:54.309339+0000","last_peered":"2024-04-25T17:35:54.309339+0000","last_clean":"2024-04-25T17:35:54.309339+0000","last_became_active":"2024-04-25T17:32:32.285416+0000","last_became_peered":"2024-04-25T17:32:32.285416+0000","last_unstale":"2024-04-25T17:35:54.309339+0000","last_undegraded":"2024-04-25T17:35:54.309339+0000","last_fullsized":"2024-04-25T17:35:54.309339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:24.473406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402426+0000","last_change":"2024-04-25T17:32:35.072626+0000","last_active":"2024-04-25T17:35:54.402426+0000","last_peered":"2024-04-25T17:35:54.402426+0000","last_clean":"2024-04-25T17:35:54.402426+0000","last_became_active":"2024-04-25T17:32:35.072264+0000","last_became_peered":"2024-04-25T17:32:35.072264+0000","last_unstale":"2024-04-25T17:35:54.402426+0000","last_undegraded":"2024-04-25T17:35:54.402426+0000","last_fullsized":"2024-04-25T17:35:54.402426+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:54.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"92'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309697+0000","last_change":"2024-04-25T17:32:32.045589+0000","last_active":"2024-04-25T17:35:54.309697+0000","last_peered":"2024-04-25T17:35:54.309697+0000","last_clean":"2024-04-25T17:35:54.309697+0000","last_became_active":"2024-04-25T17:32:32.044838+0000","last_became_peered":"2024-04-25T17:32:32.044838+0000","last_unstale":"2024-04-25T17:35:54.309697+0000","last_undegraded":"2024-04-25T17:35:54.309697+0000","last_fullsized":"2024-04-25T17:35:54.309697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:32:05.748335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549290+0000","last_change":"2024-04-25T17:32:35.075077+0000","last_active":"2024-04-25T17:35:54.549290+0000","last_peered":"2024-04-25T17:35:54.549290+0000","last_clean":"2024-04-25T17:35:54.549290+0000","last_became_active":"2024-04-25T17:32:35.074622+0000","last_became_peered":"2024-04-25T17:32:35.074622+0000","last_unstale":"2024-04-25T17:35:54.549290+0000","last_undegraded":"2024-04-25T17:35:54.549290+0000","last_fullsized":"2024-04-25T17:35:54.549290+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:34:53.435637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.307183+0000","last_change":"2024-04-25T17:32:32.288707+0000","last_active":"2024-04-25T17:35:54.307183+0000","last_peered":"2024-04-25T17:35:54.307183+0000","last_clean":"2024-04-25T17:35:54.307183+0000","last_became_active":"2024-04-25T17:32:32.288321+0000","last_became_peered":"2024-04-25T17:32:32.288321+0000","last_unstale":"2024-04-25T17:35:54.307183+0000","last_undegraded":"2024-04-25T17:35:54.307183+0000","last_fullsized":"2024-04-25T17:35:54.307183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:08:00.113662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350774+0000","last_change":"2024-04-25T17:32:35.072100+0000","last_active":"2024-04-25T17:35:54.350774+0000","last_peered":"2024-04-25T17:35:54.350774+0000","last_clean":"2024-04-25T17:35:54.350774+0000","last_became_active":"2024-04-25T17:32:35.070706+0000","last_became_peered":"2024-04-25T17:32:35.070706+0000","last_unstale":"2024-04-25T17:35:54.350774+0000","last_undegraded":"2024-04-25T17:35:54.350774+0000","last_fullsized":"2024-04-25T17:35:54.350774+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:41:18.717385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542543+0000","last_change":"2024-04-25T17:32:32.042700+0000","last_active":"2024-04-25T17:35:54.542543+0000","last_peered":"2024-04-25T17:35:54.542543+0000","last_clean":"2024-04-25T17:35:54.542543+0000","last_became_active":"2024-04-25T17:32:32.042383+0000","last_became_peered":"2024-04-25T17:32:32.042383+0000","last_unstale":"2024-04-25T17:35:54.542543+0000","last_undegraded":"2024-04-25T17:35:54.542543+0000","last_fullsized":"2024-04-25T17:35:54.542543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:11.128322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.350759+0000","last_change":"2024-04-25T17:32:35.072336+0000","last_active":"2024-04-25T17:35:54.350759+0000","last_peered":"2024-04-25T17:35:54.350759+0000","last_clean":"2024-04-25T17:35:54.350759+0000","last_became_active":"2024-04-25T17:32:35.069830+0000","last_became_peered":"2024-04-25T17:32:35.069830+0000","last_unstale":"2024-04-25T17:35:54.350759+0000","last_undegraded":"2024-04-25T17:35:54.350759+0000","last_fullsized":"2024-04-25T17:35:54.350759+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:12.887428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402564+0000","last_change":"2024-04-25T17:32:35.071375+0000","last_active":"2024-04-25T17:35:54.402564+0000","last_peered":"2024-04-25T17:35:54.402564+0000","last_clean":"2024-04-25T17:35:54.402564+0000","last_became_active":"2024-04-25T17:32:35.071063+0000","last_became_peered":"2024-04-25T17:32:35.071063+0000","last_unstale":"2024-04-25T17:35:54.402564+0000","last_undegraded":"2024-04-25T17:35:54.402564+0000","last_fullsized":"2024-04-25T17:35:54.402564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:51:41.917038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'6","reported_seq":219,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:58.420894+0000","last_change":"2024-04-25T17:32:32.286941+0000","last_active":"2024-04-25T17:35:58.420894+0000","last_peered":"2024-04-25T17:35:58.420894+0000","last_clean":"2024-04-25T17:35:58.420894+0000","last_became_active":"2024-04-25T17:32:32.286608+0000","last_became_peered":"2024-04-25T17:32:32.286608+0000","last_unstale":"2024-04-25T17:35:58.420894+0000","last_undegraded":"2024-04-25T17:35:58.420894+0000","last_fullsized":"2024-04-25T17:35:58.420894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:01.559724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":184,"num_read_kb":184,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395790+0000","last_change":"2024-04-25T17:32:35.068701+0000","last_active":"2024-04-25T17:35:54.395790+0000","last_peered":"2024-04-25T17:35:54.395790+0000","last_clean":"2024-04-25T17:35:54.395790+0000","last_became_active":"2024-04-25T17:32:35.068418+0000","last_became_peered":"2024-04-25T17:32:35.068418+0000","last_unstale":"2024-04-25T17:35:54.395790+0000","last_undegraded":"2024-04-25T17:35:54.395790+0000","last_fullsized":"2024-04-25T17:35:54.395790+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:42:25.431527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310665+0000","last_change":"2024-04-25T17:32:32.042167+0000","last_active":"2024-04-25T17:35:54.310665+0000","last_peered":"2024-04-25T17:35:54.310665+0000","last_clean":"2024-04-25T17:35:54.310665+0000","last_became_active":"2024-04-25T17:32:32.041765+0000","last_became_peered":"2024-04-25T17:32:32.041765+0000","last_unstale":"2024-04-25T17:35:54.310665+0000","last_undegraded":"2024-04-25T17:35:54.310665+0000","last_fullsized":"2024-04-25T17:35:54.310665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:28.901648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"92'148","reported_seq":407,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:56.413572+0000","last_change":"2024-04-25T17:31:05.391870+0000","last_active":"2024-04-25T17:35:56.413572+0000","last_peered":"2024-04-25T17:35:56.413572+0000","last_clean":"2024-04-25T17:35:56.413572+0000","last_became_active":"2024-04-25T17:31:05.391426+0000","last_became_peered":"2024-04-25T17:31:05.391426+0000","last_unstale":"2024-04-25T17:35:56.413572+0000","last_undegraded":"2024-04-25T17:35:56.413572+0000","last_fullsized":"2024-04-25T17:35:56.413572+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:27:37.393149+0000","last_clean_scrub_stamp":"2024-04-25T17:27:37.393149+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:39:18.400543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.415657+0000","last_change":"2024-04-25T17:33:10.190518+0000","last_active":"2024-04-25T17:35:54.415657+0000","last_peered":"2024-04-25T17:35:54.415657+0000","last_clean":"2024-04-25T17:35:54.415657+0000","last_became_active":"2024-04-25T17:33:10.190166+0000","last_became_peered":"2024-04-25T17:33:10.190166+0000","last_unstale":"2024-04-25T17:35:54.415657+0000","last_undegraded":"2024-04-25T17:35:54.415657+0000","last_fullsized":"2024-04-25T17:35:54.415657+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:45.707078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542429+0000","last_change":"2024-04-25T17:33:10.197412+0000","last_active":"2024-04-25T17:35:54.542429+0000","last_peered":"2024-04-25T17:35:54.542429+0000","last_clean":"2024-04-25T17:35:54.542429+0000","last_became_active":"2024-04-25T17:33:10.194388+0000","last_became_peered":"2024-04-25T17:33:10.194388+0000","last_unstale":"2024-04-25T17:35:54.542429+0000","last_undegraded":"2024-04-25T17:35:54.542429+0000","last_fullsized":"2024-04-25T17:35:54.542429+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:35.455072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309588+0000","last_change":"2024-04-25T17:32:32.285845+0000","last_active":"2024-04-25T17:35:54.309588+0000","last_peered":"2024-04-25T17:35:54.309588+0000","last_clean":"2024-04-25T17:35:54.309588+0000","last_became_active":"2024-04-25T17:32:32.285386+0000","last_became_peered":"2024-04-25T17:32:32.285386+0000","last_unstale":"2024-04-25T17:35:54.309588+0000","last_undegraded":"2024-04-25T17:35:54.309588+0000","last_fullsized":"2024-04-25T17:35:54.309588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:12.115900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416407+0000","last_change":"2024-04-25T17:32:35.065808+0000","last_active":"2024-04-25T17:35:54.416407+0000","last_peered":"2024-04-25T17:35:54.416407+0000","last_clean":"2024-04-25T17:35:54.416407+0000","last_became_active":"2024-04-25T17:32:35.065504+0000","last_became_peered":"2024-04-25T17:32:35.065504+0000","last_unstale":"2024-04-25T17:35:54.416407+0000","last_undegraded":"2024-04-25T17:35:54.416407+0000","last_fullsized":"2024-04-25T17:35:54.416407+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:33.438281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"92'101","reported_seq":222,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:56.480578+0000","last_change":"2024-04-25T17:32:32.045357+0000","last_active":"2024-04-25T17:35:56.480578+0000","last_peered":"2024-04-25T17:35:56.480578+0000","last_clean":"2024-04-25T17:35:56.480578+0000","last_became_active":"2024-04-25T17:32:32.045020+0000","last_became_peered":"2024-04-25T17:32:32.045020+0000","last_unstale":"2024-04-25T17:35:56.480578+0000","last_undegraded":"2024-04-25T17:35:56.480578+0000","last_fullsized":"2024-04-25T17:35:56.480578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:36.113130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":186,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395786+0000","last_change":"2024-04-25T17:32:35.077889+0000","last_active":"2024-04-25T17:35:54.395786+0000","last_peered":"2024-04-25T17:35:54.395786+0000","last_clean":"2024-04-25T17:35:54.395786+0000","last_became_active":"2024-04-25T17:32:35.076470+0000","last_became_peered":"2024-04-25T17:32:35.076470+0000","last_unstale":"2024-04-25T17:35:54.395786+0000","last_undegraded":"2024-04-25T17:35:54.395786+0000","last_fullsized":"2024-04-25T17:35:54.395786+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:45.457146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.310626+0000","last_change":"2024-04-25T17:32:32.406826+0000","last_active":"2024-04-25T17:35:54.310626+0000","last_peered":"2024-04-25T17:35:54.310626+0000","last_clean":"2024-04-25T17:35:54.310626+0000","last_became_active":"2024-04-25T17:32:32.406454+0000","last_became_peered":"2024-04-25T17:32:32.406454+0000","last_unstale":"2024-04-25T17:35:54.310626+0000","last_undegraded":"2024-04-25T17:35:54.310626+0000","last_fullsized":"2024-04-25T17:35:54.310626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:21:11.730942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.543116+0000","last_change":"2024-04-25T17:32:35.075961+0000","last_active":"2024-04-25T17:35:54.543116+0000","last_peered":"2024-04-25T17:35:54.543116+0000","last_clean":"2024-04-25T17:35:54.543116+0000","last_became_active":"2024-04-25T17:32:35.075595+0000","last_became_peered":"2024-04-25T17:32:35.075595+0000","last_unstale":"2024-04-25T17:35:54.543116+0000","last_undegraded":"2024-04-25T17:35:54.543116+0000","last_fullsized":"2024-04-25T17:35:54.543116+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:25.513096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352364+0000","last_change":"2024-04-25T17:32:32.289123+0000","last_active":"2024-04-25T17:35:54.352364+0000","last_peered":"2024-04-25T17:35:54.352364+0000","last_clean":"2024-04-25T17:35:54.352364+0000","last_became_active":"2024-04-25T17:32:32.288783+0000","last_became_peered":"2024-04-25T17:32:32.288783+0000","last_unstale":"2024-04-25T17:35:54.352364+0000","last_undegraded":"2024-04-25T17:35:54.352364+0000","last_fullsized":"2024-04-25T17:35:54.352364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:52.222440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.309346+0000","last_change":"2024-04-25T17:32:35.066364+0000","last_active":"2024-04-25T17:35:54.309346+0000","last_peered":"2024-04-25T17:35:54.309346+0000","last_clean":"2024-04-25T17:35:54.309346+0000","last_became_active":"2024-04-25T17:32:35.065489+0000","last_became_peered":"2024-04-25T17:32:35.065489+0000","last_unstale":"2024-04-25T17:35:54.309346+0000","last_undegraded":"2024-04-25T17:35:54.309346+0000","last_fullsized":"2024-04-25T17:35:54.309346+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:16.817796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395705+0000","last_change":"2024-04-25T17:32:32.040662+0000","last_active":"2024-04-25T17:35:54.395705+0000","last_peered":"2024-04-25T17:35:54.395705+0000","last_clean":"2024-04-25T17:35:54.395705+0000","last_became_active":"2024-04-25T17:32:32.040334+0000","last_became_peered":"2024-04-25T17:32:32.040334+0000","last_unstale":"2024-04-25T17:35:54.395705+0000","last_undegraded":"2024-04-25T17:35:54.395705+0000","last_fullsized":"2024-04-25T17:35:54.395705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:32:03.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549074+0000","last_change":"2024-04-25T17:32:35.079183+0000","last_active":"2024-04-25T17:35:54.549074+0000","last_peered":"2024-04-25T17:35:54.549074+0000","last_clean":"2024-04-25T17:35:54.549074+0000","last_became_active":"2024-04-25T17:32:35.078472+0000","last_became_peered":"2024-04-25T17:32:35.078472+0000","last_unstale":"2024-04-25T17:35:54.549074+0000","last_undegraded":"2024-04-25T17:35:54.549074+0000","last_fullsized":"2024-04-25T17:35:54.549074+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:39:59.120020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'2","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549072+0000","last_change":"2024-04-25T17:33:10.201283+0000","last_active":"2024-04-25T17:35:54.549072+0000","last_peered":"2024-04-25T17:35:54.549072+0000","last_clean":"2024-04-25T17:35:54.549072+0000","last_became_active":"2024-04-25T17:33:10.196192+0000","last_became_peered":"2024-04-25T17:33:10.196192+0000","last_unstale":"2024-04-25T17:35:54.549072+0000","last_undegraded":"2024-04-25T17:35:54.549072+0000","last_fullsized":"2024-04-25T17:35:54.549072+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:03.716366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395607+0000","last_change":"2024-04-25T17:32:35.081593+0000","last_active":"2024-04-25T17:35:54.395607+0000","last_peered":"2024-04-25T17:35:54.395607+0000","last_clean":"2024-04-25T17:35:54.395607+0000","last_became_active":"2024-04-25T17:32:35.081248+0000","last_became_peered":"2024-04-25T17:32:35.081248+0000","last_unstale":"2024-04-25T17:35:54.395607+0000","last_undegraded":"2024-04-25T17:35:54.395607+0000","last_fullsized":"2024-04-25T17:35:54.395607+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:41:01.177899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549097+0000","last_change":"2024-04-25T17:33:10.198901+0000","last_active":"2024-04-25T17:35:54.549097+0000","last_peered":"2024-04-25T17:35:54.549097+0000","last_clean":"2024-04-25T17:35:54.549097+0000","last_became_active":"2024-04-25T17:33:10.192191+0000","last_became_peered":"2024-04-25T17:33:10.192191+0000","last_unstale":"2024-04-25T17:35:54.549097+0000","last_undegraded":"2024-04-25T17:35:54.549097+0000","last_fullsized":"2024-04-25T17:35:54.549097+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:39.760642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542586+0000","last_change":"2024-04-25T17:32:35.068592+0000","last_active":"2024-04-25T17:35:54.542586+0000","last_peered":"2024-04-25T17:35:54.542586+0000","last_clean":"2024-04-25T17:35:54.542586+0000","last_became_active":"2024-04-25T17:32:35.068240+0000","last_became_peered":"2024-04-25T17:32:35.068240+0000","last_unstale":"2024-04-25T17:35:54.542586+0000","last_undegraded":"2024-04-25T17:35:54.542586+0000","last_fullsized":"2024-04-25T17:35:54.542586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:48.628281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352804+0000","last_change":"2024-04-25T17:32:32.406702+0000","last_active":"2024-04-25T17:35:54.352804+0000","last_peered":"2024-04-25T17:35:54.352804+0000","last_clean":"2024-04-25T17:35:54.352804+0000","last_became_active":"2024-04-25T17:32:32.406263+0000","last_became_peered":"2024-04-25T17:32:32.406263+0000","last_unstale":"2024-04-25T17:35:54.352804+0000","last_undegraded":"2024-04-25T17:35:54.352804+0000","last_fullsized":"2024-04-25T17:35:54.352804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:27.627002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542741+0000","last_change":"2024-04-25T17:32:35.073585+0000","last_active":"2024-04-25T17:35:54.542741+0000","last_peered":"2024-04-25T17:35:54.542741+0000","last_clean":"2024-04-25T17:35:54.542741+0000","last_became_active":"2024-04-25T17:32:35.072902+0000","last_became_peered":"2024-04-25T17:32:35.072902+0000","last_unstale":"2024-04-25T17:35:54.542741+0000","last_undegraded":"2024-04-25T17:35:54.542741+0000","last_fullsized":"2024-04-25T17:35:54.542741+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:17.277129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'65","reported_seq":125,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:56.109820+0000","last_change":"2024-04-25T17:32:32.042546+0000","last_active":"2024-04-25T17:35:56.109820+0000","last_peered":"2024-04-25T17:35:56.109820+0000","last_clean":"2024-04-25T17:35:56.109820+0000","last_became_active":"2024-04-25T17:32:32.042191+0000","last_became_peered":"2024-04-25T17:32:32.042191+0000","last_unstale":"2024-04-25T17:35:56.109820+0000","last_undegraded":"2024-04-25T17:35:56.109820+0000","last_fullsized":"2024-04-25T17:35:56.109820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:46:58.126260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542396+0000","last_change":"2024-04-25T17:32:35.077067+0000","last_active":"2024-04-25T17:35:54.542396+0000","last_peered":"2024-04-25T17:35:54.542396+0000","last_clean":"2024-04-25T17:35:54.542396+0000","last_became_active":"2024-04-25T17:32:35.076701+0000","last_became_peered":"2024-04-25T17:32:35.076701+0000","last_unstale":"2024-04-25T17:35:54.542396+0000","last_undegraded":"2024-04-25T17:35:54.542396+0000","last_fullsized":"2024-04-25T17:35:54.542396+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:06.646419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416631+0000","last_change":"2024-04-25T17:33:10.610032+0000","last_active":"2024-04-25T17:35:54.416631+0000","last_peered":"2024-04-25T17:35:54.416631+0000","last_clean":"2024-04-25T17:35:54.416631+0000","last_became_active":"2024-04-25T17:33:10.607260+0000","last_became_peered":"2024-04-25T17:33:10.607260+0000","last_unstale":"2024-04-25T17:35:54.416631+0000","last_undegraded":"2024-04-25T17:35:54.416631+0000","last_fullsized":"2024-04-25T17:35:54.416631+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:31.018402+0000","last_clean_scrub_stamp":"2024-04-25T17:32:31.018402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:42:41.509257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542556+0000","last_change":"2024-04-25T17:32:35.071351+0000","last_active":"2024-04-25T17:35:54.542556+0000","last_peered":"2024-04-25T17:35:54.542556+0000","last_clean":"2024-04-25T17:35:54.542556+0000","last_became_active":"2024-04-25T17:32:35.070559+0000","last_became_peered":"2024-04-25T17:32:35.070559+0000","last_unstale":"2024-04-25T17:35:54.542556+0000","last_undegraded":"2024-04-25T17:35:54.542556+0000","last_fullsized":"2024-04-25T17:35:54.542556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:23.864826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549145+0000","last_change":"2024-04-25T17:32:35.073326+0000","last_active":"2024-04-25T17:35:54.549145+0000","last_peered":"2024-04-25T17:35:54.549145+0000","last_clean":"2024-04-25T17:35:54.549145+0000","last_became_active":"2024-04-25T17:32:35.072904+0000","last_became_peered":"2024-04-25T17:32:35.072904+0000","last_unstale":"2024-04-25T17:35:54.549145+0000","last_undegraded":"2024-04-25T17:35:54.549145+0000","last_fullsized":"2024-04-25T17:35:54.549145+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:50.586127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352580+0000","last_change":"2024-04-25T17:33:10.195444+0000","last_active":"2024-04-25T17:35:54.352580+0000","last_peered":"2024-04-25T17:35:54.352580+0000","last_clean":"2024-04-25T17:35:54.352580+0000","last_became_active":"2024-04-25T17:33:10.195048+0000","last_became_peered":"2024-04-25T17:33:10.195048+0000","last_unstale":"2024-04-25T17:35:54.352580+0000","last_undegraded":"2024-04-25T17:35:54.352580+0000","last_fullsized":"2024-04-25T17:35:54.352580+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:03.098618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352519+0000","last_change":"2024-04-25T17:32:35.072053+0000","last_active":"2024-04-25T17:35:54.352519+0000","last_peered":"2024-04-25T17:35:54.352519+0000","last_clean":"2024-04-25T17:35:54.352519+0000","last_became_active":"2024-04-25T17:32:35.071712+0000","last_became_peered":"2024-04-25T17:32:35.071712+0000","last_unstale":"2024-04-25T17:35:54.352519+0000","last_undegraded":"2024-04-25T17:35:54.352519+0000","last_fullsized":"2024-04-25T17:35:54.352519+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:01:42.928854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.542632+0000","last_change":"2024-04-25T17:32:35.061408+0000","last_active":"2024-04-25T17:35:54.542632+0000","last_peered":"2024-04-25T17:35:54.542632+0000","last_clean":"2024-04-25T17:35:54.542632+0000","last_became_active":"2024-04-25T17:32:35.061105+0000","last_became_peered":"2024-04-25T17:32:35.061105+0000","last_unstale":"2024-04-25T17:35:54.542632+0000","last_undegraded":"2024-04-25T17:35:54.542632+0000","last_fullsized":"2024-04-25T17:35:54.542632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:48.242157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416347+0000","last_change":"2024-04-25T17:33:10.687122+0000","last_active":"2024-04-25T17:35:54.416347+0000","last_peered":"2024-04-25T17:35:54.416347+0000","last_clean":"2024-04-25T17:35:54.416347+0000","last_became_active":"2024-04-25T17:33:10.686755+0000","last_became_peered":"2024-04-25T17:33:10.686755+0000","last_unstale":"2024-04-25T17:35:54.416347+0000","last_undegraded":"2024-04-25T17:35:54.416347+0000","last_fullsized":"2024-04-25T17:35:54.416347+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:02.999655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549326+0000","last_change":"2024-04-25T17:32:35.077433+0000","last_active":"2024-04-25T17:35:54.549326+0000","last_peered":"2024-04-25T17:35:54.549326+0000","last_clean":"2024-04-25T17:35:54.549326+0000","last_became_active":"2024-04-25T17:32:35.076201+0000","last_became_peered":"2024-04-25T17:32:35.076201+0000","last_unstale":"2024-04-25T17:35:54.549326+0000","last_undegraded":"2024-04-25T17:35:54.549326+0000","last_fullsized":"2024-04-25T17:35:54.549326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:55:42.677022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352836+0000","last_change":"2024-04-25T17:32:35.068017+0000","last_active":"2024-04-25T17:35:54.352836+0000","last_peered":"2024-04-25T17:35:54.352836+0000","last_clean":"2024-04-25T17:35:54.352836+0000","last_became_active":"2024-04-25T17:32:35.067647+0000","last_became_peered":"2024-04-25T17:32:35.067647+0000","last_unstale":"2024-04-25T17:35:54.352836+0000","last_undegraded":"2024-04-25T17:35:54.352836+0000","last_fullsized":"2024-04-25T17:35:54.352836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:28:11.339402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395781+0000","last_change":"2024-04-25T17:32:35.075192+0000","last_active":"2024-04-25T17:35:54.395781+0000","last_peered":"2024-04-25T17:35:54.395781+0000","last_clean":"2024-04-25T17:35:54.395781+0000","last_became_active":"2024-04-25T17:32:35.074850+0000","last_became_peered":"2024-04-25T17:32:35.074850+0000","last_unstale":"2024-04-25T17:35:54.395781+0000","last_undegraded":"2024-04-25T17:35:54.395781+0000","last_fullsized":"2024-04-25T17:35:54.395781+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:04:41.343312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.402408+0000","last_change":"2024-04-25T17:32:35.078324+0000","last_active":"2024-04-25T17:35:54.402408+0000","last_peered":"2024-04-25T17:35:54.402408+0000","last_clean":"2024-04-25T17:35:54.402408+0000","last_became_active":"2024-04-25T17:32:35.077958+0000","last_became_peered":"2024-04-25T17:32:35.077958+0000","last_unstale":"2024-04-25T17:35:54.402408+0000","last_undegraded":"2024-04-25T17:35:54.402408+0000","last_fullsized":"2024-04-25T17:35:54.402408+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:46:10.148060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.395538+0000","last_change":"2024-04-25T17:32:35.078023+0000","last_active":"2024-04-25T17:35:54.395538+0000","last_peered":"2024-04-25T17:35:54.395538+0000","last_clean":"2024-04-25T17:35:54.395538+0000","last_became_active":"2024-04-25T17:32:35.077702+0000","last_became_peered":"2024-04-25T17:32:35.077702+0000","last_unstale":"2024-04-25T17:35:54.395538+0000","last_undegraded":"2024-04-25T17:35:54.395538+0000","last_fullsized":"2024-04-25T17:35:54.395538+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:44:37.933179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.352546+0000","last_change":"2024-04-25T17:32:35.078388+0000","last_active":"2024-04-25T17:35:54.352546+0000","last_peered":"2024-04-25T17:35:54.352546+0000","last_clean":"2024-04-25T17:35:54.352546+0000","last_became_active":"2024-04-25T17:32:35.078058+0000","last_became_peered":"2024-04-25T17:32:35.078058+0000","last_unstale":"2024-04-25T17:35:54.352546+0000","last_undegraded":"2024-04-25T17:35:54.352546+0000","last_fullsized":"2024-04-25T17:35:54.352546+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:05.699542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.416109+0000","last_change":"2024-04-25T17:32:35.068532+0000","last_active":"2024-04-25T17:35:54.416109+0000","last_peered":"2024-04-25T17:35:54.416109+0000","last_clean":"2024-04-25T17:35:54.416109+0000","last_became_active":"2024-04-25T17:32:35.068225+0000","last_became_peered":"2024-04-25T17:32:35.068225+0000","last_unstale":"2024-04-25T17:35:54.416109+0000","last_undegraded":"2024-04-25T17:35:54.416109+0000","last_fullsized":"2024-04-25T17:35:54.416109+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:09.911999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-25T17:35:54.549091+0000","last_change":"2024-04-25T17:32:35.065526+0000","last_active":"2024-04-25T17:35:54.549091+0000","last_peered":"2024-04-25T17:35:54.549091+0000","last_clean":"2024-04-25T17:35:54.549091+0000","last_became_active":"2024-04-25T17:32:35.065146+0000","last_became_peered":"2024-04-25T17:32:35.065146+0000","last_unstale":"2024-04-25T17:35:54.549091+0000","last_undegraded":"2024-04-25T17:35:54.549091+0000","last_fullsized":"2024-04-25T17:35:54.549091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T17:32:34.036220+0000","last_clean_scrub_stamp":"2024-04-25T17:32:34.036220+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:33.662055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":106941,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":794668,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1484,"num_read_kb":956,"num_write":423,"num_write_kb":2355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2384004,"data_compressed":103473,"data_compressed_allocated":135168,"data_compressed_original":2101248,"omap_allocated":0,"internal_metadata":0},"log_size":310,"ondisk_log_size":310,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351031,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":491851,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612670,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":1480475,"data_compressed":53748,"data_compressed_allocated":69632,"data_compressed_original":905216,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776196,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":885303,"data_compressed":22188,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037831,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":885223,"data_compressed":22188,"data_compressed_allocated":28672,"data_compressed_original":401408,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299470,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1085559,"data_compressed":38546,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462995,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":491476,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":558866,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":492747,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":560915,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":1213235,"data_compressed":37390,"data_compressed_allocated":49152,"data_compressed_original":708608,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888157,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":493227,"data_compressed":2899,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724633,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1122304,"data_stored":1412161,"data_compressed":53748,"data_compressed_allocated":69632,"data_compressed_original":905216,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35647,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330407,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":393954,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":398721,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T17:36:05.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:36:05.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:05 smithi173 ceph-mon[32856]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-25T17:36:05.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:05 smithi173 ceph-mon[32856]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:36:05.531 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:05 smithi098 ceph-mon[29192]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-25T17:36:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:05 smithi098 ceph-mon[29192]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:36:05.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:05 smithi203 ceph-mon[32834]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-25T17:36:05.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:05 smithi203 ceph-mon[32834]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T17:36:07.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 -- 172.21.15.98:0/4141560096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d10107990 msgr2=0x7f5d10107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:07.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 --2- 172.21.15.98:0/4141560096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d10107990 0x7f5d10107d70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009990 tx=0x7f5d00009ca0 comp rx=0 tx=0).stop 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 -- 172.21.15.98:0/4141560096 shutdown_connections 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 --2- 172.21.15.98:0/4141560096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d10102300 0x7f5d1010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 --2- 172.21.15.98:0/4141560096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d10107990 0x7f5d10107d70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 --2- 172.21.15.98:0/4141560096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d10101940 0x7f5d10101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 -- 172.21.15.98:0/4141560096 >> 172.21.15.98:0/4141560096 conn(0x7f5d100788f0 msgr2=0x7f5d10078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 -- 172.21.15.98:0/4141560096 shutdown_connections 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 -- 172.21.15.98:0/4141560096 wait complete. 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 Processor -- start 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.278+0000 7f5d1572d700 1 -- start start 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d1572d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d10101940 0x7f5d10199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d1572d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d10102300 0x7f5d10199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d1572d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d10107990 0x7f5d1019d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:07.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d1572d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d10112c50 con 0x7f5d10107990 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d1572d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5d10112ad0 con 0x7f5d10101940 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d1572d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5d10112dd0 con 0x7f5d10102300 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d10107990 0x7f5d1019d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d10107990 0x7f5d1019d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33628/0 (socket says 172.21.15.98:33628) 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0f7fe700 1 -- 172.21.15.98:0/3138239790 learned_addr learned my addr 172.21.15.98:0/3138239790 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0e7fc700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d10102300 0x7f5d10199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0effd700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d10101940 0x7f5d10199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0f7fe700 1 -- 172.21.15.98:0/3138239790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d10102300 msgr2=0x7f5d10199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0f7fe700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d10102300 0x7f5d10199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0f7fe700 1 -- 172.21.15.98:0/3138239790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d10101940 msgr2=0x7f5d10199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0f7fe700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d10101940 0x7f5d10199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0f7fe700 1 -- 172.21.15.98:0/3138239790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1019e130 con 0x7f5d10107990 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d0f7fe700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d10107990 0x7f5d1019d9f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5d08007d70 tx=0x7f5d0800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:07.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5cf7fff700 1 -- 172.21.15.98:0/3138239790 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08016020 con 0x7f5d10107990 2024-04-25T17:36:07.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5cf7fff700 1 -- 172.21.15.98:0/3138239790 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d0800ac90 con 0x7f5d10107990 2024-04-25T17:36:07.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5d1572d700 1 -- 172.21.15.98:0/3138239790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d1019e420 con 0x7f5d10107990 2024-04-25T17:36:07.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.279+0000 7f5cf7fff700 1 -- 172.21.15.98:0/3138239790 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d080115e0 con 0x7f5d10107990 2024-04-25T17:36:07.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.280+0000 7f5d1572d700 1 -- 172.21.15.98:0/3138239790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d101a5ca0 con 0x7f5d10107990 2024-04-25T17:36:07.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.280+0000 7f5d1572d700 1 -- 172.21.15.98:0/3138239790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d10109eb0 con 0x7f5d10107990 2024-04-25T17:36:07.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.282+0000 7f5cf7fff700 1 -- 172.21.15.98:0/3138239790 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f5d08019020 con 0x7f5d10107990 2024-04-25T17:36:07.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.282+0000 7f5cf7fff700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5cf80a3ed0 0x7f5cf80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:07.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.282+0000 7f5d0effd700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5cf80a3ed0 0x7f5cf80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:07.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.282+0000 7f5cf7fff700 1 -- 172.21.15.98:0/3138239790 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f5d080d0d10 con 0x7f5d10107990 2024-04-25T17:36:07.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.283+0000 7f5d0effd700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5cf80a3ed0 0x7f5cf80a6390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000b5c0 tx=0x7f5d00005960 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:07.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.286+0000 7f5cf7fff700 1 -- 172.21.15.98:0/3138239790 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d08020020 con 0x7f5d10107990 2024-04-25T17:36:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.530+0000 7f5d1572d700 1 -- 172.21.15.98:0/3138239790 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d10069730 con 0x7f5d10107990 2024-04-25T17:36:07.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:07 smithi098 ceph-mon[29192]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 68 op/s 2024-04-25T17:36:07.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.531+0000 7f5cf7fff700 1 -- 172.21.15.98:0/3138239790 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11877 (secure 0 0 0) 0x7f5d0809d2a0 con 0x7f5d10107990 2024-04-25T17:36:07.532 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:07.532 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:35:53.134159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 -- 172.21.15.98:0/3138239790 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5cf80a3ed0 msgr2=0x7f5cf80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5cf80a3ed0 0x7f5cf80a6390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000b5c0 tx=0x7f5d00005960 comp rx=0 tx=0).stop 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 -- 172.21.15.98:0/3138239790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d10107990 msgr2=0x7f5d1019d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d10107990 0x7f5d1019d9f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5d08007d70 tx=0x7f5d0800a340 comp rx=0 tx=0).stop 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 -- 172.21.15.98:0/3138239790 shutdown_connections 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5cf80a3ed0 0x7f5cf80a6390 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d10107990 0x7f5d1019d9f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5d10101940 0x7f5d10199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 --2- 172.21.15.98:0/3138239790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d10102300 0x7f5d10199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 -- 172.21.15.98:0/3138239790 >> 172.21.15.98:0/3138239790 conn(0x7f5d100788f0 msgr2=0x7f5d10100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:07.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 -- 172.21.15.98:0/3138239790 shutdown_connections 2024-04-25T17:36:07.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:07.533+0000 7f5d1572d700 1 -- 172.21.15.98:0/3138239790 wait complete. 2024-04-25T17:36:07.535 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 26 2024-04-25T17:36:07.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:07 smithi203 ceph-mon[32834]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 68 op/s 2024-04-25T17:36:07.802 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:07 smithi173 ceph-mon[32856]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 68 op/s 2024-04-25T17:36:08.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:36:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:08.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:08 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3138239790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:08.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:08 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3138239790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:08.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:08 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3138239790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:09.684 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:09 smithi098 ceph-mon[29192]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 98 KiB/s wr, 62 op/s 2024-04-25T17:36:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.697+0000 7f93620bb640 1 -- 172.21.15.98:0/3909513756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f935c10c520 msgr2=0x7f935c10e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.697+0000 7f93620bb640 1 --2- 172.21.15.98:0/3909513756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f935c10c520 0x7f935c10e9c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9354009ca0 tx=0x7f935402a9e0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.697+0000 7f93620bb640 1 -- 172.21.15.98:0/3909513756 shutdown_connections 2024-04-25T17:36:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.697+0000 7f93620bb640 1 --2- 172.21.15.98:0/3909513756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f935c10c520 0x7f935c10e9c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.697+0000 7f93620bb640 1 --2- 172.21.15.98:0/3909513756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c103b00 0x7f935c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.697+0000 7f93620bb640 1 --2- 172.21.15.98:0/3909513756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f935c1031e0 0x7f935c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.697+0000 7f93620bb640 1 -- 172.21.15.98:0/3909513756 >> 172.21.15.98:0/3909513756 conn(0x7f935c077fd0 msgr2=0x7f935c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.697+0000 7f93620bb640 1 -- 172.21.15.98:0/3909513756 shutdown_connections 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.697+0000 7f93620bb640 1 -- 172.21.15.98:0/3909513756 wait complete. 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f93620bb640 1 Processor -- start 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f93620bb640 1 -- start start 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f93620bb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f935c1031e0 0x7f935c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f93620bb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f935c103b00 0x7f935c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f93620bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c10c520 0x7f935c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f93620bb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f935c112190 con 0x7f935c1031e0 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f93620bb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f935c112010 con 0x7f935c103b00 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f93620bb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f935c112310 con 0x7f935c10c520 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f935b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f935c1031e0 0x7f935c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f935b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f935c1031e0 0x7f935c198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33652/0 (socket says 172.21.15.98:33652) 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f935b7fe640 1 -- 172.21.15.98:0/4067795060 learned_addr learned my addr 172.21.15.98:0/4067795060 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f935affd640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f935c103b00 0x7f935c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.698+0000 7f935bfff640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c10c520 0x7f935c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f935b7fe640 1 -- 172.21.15.98:0/4067795060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c10c520 msgr2=0x7f935c19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f935b7fe640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c10c520 0x7f935c19ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f935b7fe640 1 -- 172.21.15.98:0/4067795060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f935c103b00 msgr2=0x7f935c198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f935b7fe640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f935c103b00 0x7f935c198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f935b7fe640 1 -- 172.21.15.98:0/4067795060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935c19d5b0 con 0x7f935c1031e0 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f935b7fe640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f935c1031e0 0x7f935c198610 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f934c00c920 tx=0x7f934c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f9358ff9640 1 -- 172.21.15.98:0/4067795060 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c007c70 con 0x7f935c1031e0 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f93620bb640 1 -- 172.21.15.98:0/4067795060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f935c19d8a0 con 0x7f935c1031e0 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f9358ff9640 1 -- 172.21.15.98:0/4067795060 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f934c007e10 con 0x7f935c1031e0 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f9358ff9640 1 -- 172.21.15.98:0/4067795060 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c018820 con 0x7f935c1031e0 2024-04-25T17:36:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.699+0000 7f93620bb640 1 -- 172.21.15.98:0/4067795060 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f935c1a5130 con 0x7f935c1031e0 2024-04-25T17:36:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f9364dce740 1 Processor -- start 2024-04-25T17:36:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f9358ff9640 1 -- 172.21.15.98:0/4067795060 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f934c00f450 con 0x7f935c1031e0 2024-04-25T17:36:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f9364dce740 1 -- start start 2024-04-25T17:36:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f9364dce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82500 0x55915ba828e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f9364dce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82e20 0x55915ba916d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f9364dce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba94000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f9364dce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55915b9e7970 con 0x55915ba91c10 2024-04-25T17:36:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f9364dce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55915b9844a0 con 0x55915ba82500 2024-04-25T17:36:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f9364dce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55915b9e3930 con 0x55915ba82e20 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f935affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba94000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f935affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba94000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33658/0 (socket says 172.21.15.98:33658) 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f935affd640 1 -- 172.21.15.98:0/894788552 learned_addr learned my addr 172.21.15.98:0/894788552 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.702+0000 7f935b7fe640 1 --2- 172.21.15.98:0/894788552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82e20 0x55915ba916d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f935bfff640 1 --2- 172.21.15.98:0/894788552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82500 0x55915ba828e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f935affd640 1 -- 172.21.15.98:0/894788552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82e20 msgr2=0x55915ba916d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f935affd640 1 --2- 172.21.15.98:0/894788552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82e20 0x55915ba916d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f935affd640 1 -- 172.21.15.98:0/894788552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82500 msgr2=0x55915ba828e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f935affd640 1 --2- 172.21.15.98:0/894788552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82500 0x55915ba828e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f935affd640 1 -- 172.21.15.98:0/894788552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55915b6fa210 con 0x55915ba91c10 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f9358ff9640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f93440a3ee0 0x7f93440a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f935bfff640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f93440a3ee0 0x7f93440a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f935affd640 1 --2- 172.21.15.98:0/894788552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba94000 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f935000bc60 tx=0x7f93500107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ab8341bfbfbf64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f9358ff9640 1 -- 172.21.15.98:0/4067795060 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f934c0d0bb0 con 0x7f935c1031e0 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f9341ffb640 1 -- 172.21.15.98:0/894788552 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935001c480 con 0x55915ba91c10 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f9341ffb640 1 -- 172.21.15.98:0/894788552 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f935001cad0 con 0x55915ba91c10 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.703+0000 7f9341ffb640 1 -- 172.21.15.98:0/894788552 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9350011ae0 con 0x55915ba91c10 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f9364dce740 1 -- 172.21.15.98:0/894788552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 msgr2=0x55915ba94000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f9364dce740 1 --2- 172.21.15.98:0/894788552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba94000 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f935000bc60 tx=0x7f93500107a0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f935bfff640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f93440a3ee0 0x7f93440a63a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9354006010 tx=0x7f9354009800 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f9364dce740 1 -- 172.21.15.98:0/894788552 shutdown_connections 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f9364dce740 1 --2- 172.21.15.98:0/894788552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba94000 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f9364dce740 1 --2- 172.21.15.98:0/894788552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82e20 0x55915ba916d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f9364dce740 1 --2- 172.21.15.98:0/894788552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82500 0x55915ba828e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f9364dce740 1 -- 172.21.15.98:0/894788552 >> 172.21.15.98:0/894788552 conn(0x55915ba7eb30 msgr2=0x55915ba7ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:09.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f9364dce740 1 -- 172.21.15.98:0/894788552 shutdown_connections 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f9364dce740 1 -- 172.21.15.98:0/894788552 wait complete. 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.704+0000 7f9364dce740 1 -- start start 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f9364dce740 1 Processor -- start 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f9364dce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba82500 0x55915ba91570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f9364dce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 0x55915ba8e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f9364dce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba91c10 0x55915ba8edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f9364dce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55915ba94840 con 0x55915ba82500 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f9364dce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55915b9820b0 con 0x55915ba82e20 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f9364dce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55915ba949c0 con 0x55915ba91c10 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f9364dce740 10 client.0 _ll_register_callbacks cb 0x55915b96fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba82500 0x55915ba91570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba82500 0x55915ba91570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33664/0 (socket says 172.21.15.98:33664) 2024-04-25T17:36:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935b7fe640 1 -- 172.21.15.98:0/2856023002 learned_addr learned my addr 172.21.15.98:0/2856023002 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935affd640 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 0x55915ba8e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935bfff640 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba91c10 0x55915ba8edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935b7fe640 1 -- 172.21.15.98:0/2856023002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba91c10 msgr2=0x55915ba8edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935b7fe640 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba91c10 0x55915ba8edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935b7fe640 1 -- 172.21.15.98:0/2856023002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 msgr2=0x55915ba8e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935b7fe640 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 0x55915ba8e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935b7fe640 1 -- 172.21.15.98:0/2856023002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55915b7c6620 con 0x55915ba82500 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f935b7fe640 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba82500 0x55915ba91570 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f934c000c00 tx=0x7f934c0d4900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b2ac72ebe5ba95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f933affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.705+0000 7f933affd640 1 -- 172.21.15.98:0/2856023002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c09c700 con 0x55915ba82500 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f933affd640 1 -- 172.21.15.98:0/2856023002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f934c0870e0 con 0x55915ba82500 2024-04-25T17:36:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f9364dce740 1 -- 172.21.15.98:0/2856023002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f935001cad0 con 0x55915ba82500 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f933affd640 1 -- 172.21.15.98:0/2856023002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f934c09c290 con 0x55915ba82500 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f9364dce740 10 client.15261 fetch_fsmap learned FSMap version 26 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f9364dce740 1 -- 172.21.15.98:0/2856023002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55915baa0820 con 0x55915ba82500 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f933affd640 1 -- 172.21.15.98:0/2856023002 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f934c0b9080 con 0x55915ba82500 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f933affd640 1 -- 172.21.15.98:0/2856023002 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f934c0a5110 con 0x55915ba82500 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f9364dce740 10 client.15261 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f9364dce740 10 client.15261 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f9364dce740 20 client.15261 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f9364dce740 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55915baa1c90 0x55915baa40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f9364dce740 4 client.15261 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f9364dce740 1 -- 172.21.15.98:0/2856023002 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f935001cad0 con 0x55915baa1c90 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.706+0000 7f935affd640 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55915baa1c90 0x55915baa40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.707+0000 7f933affd640 10 client.15261 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.707+0000 7f935affd640 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55915baa1c90 0x55915baa40a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fbc10b6e8ece94a server_cookie=b0c78c39367275ff in_seq=0 out_seq=0 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f933affd640 1 -- 172.21.15.98:0/2856023002 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f935001cad0 con 0x55915baa1c90 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f933affd640 10 client.15261 handle_command_reply: tid=0 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 client.15261 shutdown 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 -- 172.21.15.98:0/2856023002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba82500 msgr2=0x55915ba91570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba82500 0x55915ba91570 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f934c000c00 tx=0x7f934c0d4900 comp rx=0 tx=0).stop 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 -- 172.21.15.98:0/2856023002 shutdown_connections 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55915baa1c90 0x55915baa40a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba91c10 0x55915ba8edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f933affd640 0 client.15261 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 0x55915ba8e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 --2- 172.21.15.98:0/2856023002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba82500 0x55915ba91570 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 -- 172.21.15.98:0/2856023002 >> 172.21.15.98:0/2856023002 conn(0x55915ba7eb30 msgr2=0x55915ba849a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 -- 172.21.15.98:0/2856023002 shutdown_connections 2024-04-25T17:36:09.715 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 1 -- 172.21.15.98:0/2856023002 wait complete. 2024-04-25T17:36:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.708+0000 7f9364dce740 20 client.15261 trim_cache size 0 max 16384 2024-04-25T17:36:09.716 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:09 smithi203 ceph-mon[32834]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 98 KiB/s wr, 62 op/s 2024-04-25T17:36:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f9364dce740 1 Processor -- start 2024-04-25T17:36:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f9364dce740 1 -- start start 2024-04-25T17:36:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f9364dce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82500 0x55915ba9d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f9364dce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 0x55915ba910a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f9364dce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915baa9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f9364dce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1ae0e0 con 0x55915ba91c10 2024-04-25T17:36:09.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f9364dce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1ab9b0 con 0x55915ba82e20 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f9364dce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1a9820 con 0x55915ba82500 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915baa9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915baa9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33666/0 (socket says 172.21.15.98:33666) 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935bfff640 1 -- 172.21.15.98:0/1660071359 learned_addr learned my addr 172.21.15.98:0/1660071359 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935b7fe640 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82500 0x55915ba9d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935affd640 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 0x55915ba910a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935bfff640 1 -- 172.21.15.98:0/1660071359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82500 msgr2=0x55915ba9d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935bfff640 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82500 0x55915ba9d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935bfff640 1 -- 172.21.15.98:0/1660071359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 msgr2=0x55915ba910a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935bfff640 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 0x55915ba910a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935bfff640 1 -- 172.21.15.98:0/1660071359 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55915ba79090 con 0x55915ba91c10 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935b7fe640 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82500 0x55915ba9d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.723+0000 7f935affd640 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 0x55915ba910a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:36:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f935bfff640 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915baa9970 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f935402f010 tx=0x7f9354040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1093fa3883d527a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f93417fa640 1 -- 172.21.15.98:0/1660071359 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9354033040 con 0x55915ba91c10 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f93417fa640 1 -- 172.21.15.98:0/1660071359 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93540044f0 con 0x55915ba91c10 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f93417fa640 1 -- 172.21.15.98:0/1660071359 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935402fe30 con 0x55915ba91c10 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f9364dce740 1 -- 172.21.15.98:0/1660071359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 msgr2=0x55915baa9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f9364dce740 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915baa9970 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f935402f010 tx=0x7f9354040000 comp rx=0 tx=0).stop 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f9364dce740 1 -- 172.21.15.98:0/1660071359 shutdown_connections 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f9364dce740 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915baa9970 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f9364dce740 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82e20 0x55915ba910a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f9364dce740 1 --2- 172.21.15.98:0/1660071359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82500 0x55915ba9d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f9364dce740 1 -- 172.21.15.98:0/1660071359 >> 172.21.15.98:0/1660071359 conn(0x55915ba7eb30 msgr2=0x55915ba89f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f9364dce740 1 -- 172.21.15.98:0/1660071359 shutdown_connections 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.724+0000 7f9364dce740 1 -- 172.21.15.98:0/1660071359 wait complete. 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f9364dce740 1 -- start start 2024-04-25T17:36:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f9364dce740 1 Processor -- start 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f9364dce740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82500 0x55915ba8b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f9364dce740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82e20 0x55915ba8bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f9364dce740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba8c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f9364dce740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1a9e70 con 0x55915ba91c10 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f9364dce740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1ab2e0 con 0x55915ba82500 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f9364dce740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f935c1aa540 con 0x55915ba82e20 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f9364dce740 10 client.0 _ll_register_callbacks cb 0x7f934c0d43b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f935bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba8c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f935bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba8c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33672/0 (socket says 172.21.15.98:33672) 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f935bfff640 1 -- 172.21.15.98:0/3250910211 learned_addr learned my addr 172.21.15.98:0/3250910211 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f935b7fe640 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82500 0x55915ba8b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f935bfff640 1 -- 172.21.15.98:0/3250910211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82e20 msgr2=0x55915ba8bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f935affd640 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82e20 0x55915ba8bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.725+0000 7f935bfff640 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82e20 0x55915ba8bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f935bfff640 1 -- 172.21.15.98:0/3250910211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82500 msgr2=0x55915ba8b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f935bfff640 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82500 0x55915ba8b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f935bfff640 1 -- 172.21.15.98:0/3250910211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55915bb073b0 con 0x55915ba91c10 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f935bfff640 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba8c150 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f935402af00 tx=0x7f935402f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7d743b38c078ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f933affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f933affd640 1 -- 172.21.15.98:0/3250910211 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9354033030 con 0x55915ba91c10 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f9364dce740 1 -- 172.21.15.98:0/3250910211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93540044f0 con 0x55915ba91c10 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f933affd640 1 -- 172.21.15.98:0/3250910211 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f93540040c0 con 0x55915ba91c10 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f933affd640 1 -- 172.21.15.98:0/3250910211 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9354062050 con 0x55915ba91c10 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f9364dce740 10 client.15267 fetch_fsmap learned FSMap version 26 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f9364dce740 1 -- 172.21.15.98:0/3250910211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55915bb085f0 con 0x55915ba91c10 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f933affd640 1 -- 172.21.15.98:0/3250910211 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93540044f0 con 0x55915ba91c10 2024-04-25T17:36:09.732 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f933affd640 1 -- 172.21.15.98:0/3250910211 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9354047430 con 0x55915ba91c10 2024-04-25T17:36:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f9364dce740 10 client.15267 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T17:36:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f9364dce740 10 client.15267 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:36:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f9364dce740 20 client.15267 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f9364dce740 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55915bb08d10 0x55915bb0b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f9364dce740 4 client.15267 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:36:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.726+0000 7f9364dce740 1 -- 172.21.15.98:0/3250910211 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55915b9f5390 con 0x55915bb08d10 2024-04-25T17:36:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.727+0000 7f935b7fe640 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55915bb08d10 0x55915bb0b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.727+0000 7f935b7fe640 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55915bb08d10 0x55915bb0b130 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=653e3c3f1cd3574e server_cookie=108d30e588d5e578 in_seq=0 out_seq=0 2024-04-25T17:36:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.727+0000 7f933affd640 10 client.15267 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f933affd640 1 -- 172.21.15.98:0/3250910211 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55915b9f5390 con 0x55915bb08d10 2024-04-25T17:36:09.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f933affd640 10 client.15267 handle_command_reply: tid=0 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f9364dce740 1 client.15267 shutdown 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f9364dce740 1 -- 172.21.15.98:0/3250910211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 msgr2=0x55915ba8c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f9364dce740 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba8c150 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f935402af00 tx=0x7f935402f870 comp rx=0 tx=0).stop 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f9364dce740 1 -- 172.21.15.98:0/3250910211 shutdown_connections 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f9364dce740 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55915bb08d10 0x55915bb0b130 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f9364dce740 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55915ba91c10 0x55915ba8c150 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f933affd640 0 client.15267 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f9364dce740 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55915ba82e20 0x55915ba8bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f9364dce740 1 --2- 172.21.15.98:0/3250910211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55915ba82500 0x55915ba8b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.738+0000 7f9364dce740 1 -- 172.21.15.98:0/3250910211 >> 172.21.15.98:0/3250910211 conn(0x55915ba7eb30 msgr2=0x55915ba7d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f9364dce740 1 -- 172.21.15.98:0/3250910211 shutdown_connections 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f9364dce740 1 -- 172.21.15.98:0/3250910211 wait complete. 2024-04-25T17:36:09.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f9364dce740 20 client.15267 trim_cache size 0 max 16384 2024-04-25T17:36:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:36:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:36:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:36:09.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 -- 172.21.15.98:0/4067795060 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f93440a3ee0 msgr2=0x7f93440a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f93440a3ee0 0x7f93440a63a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9354006010 tx=0x7f9354009800 comp rx=0 tx=0).stop 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 -- 172.21.15.98:0/4067795060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f935c1031e0 msgr2=0x7f935c198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f935c1031e0 0x7f935c198610 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f934c00c920 tx=0x7f934c00cde0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 -- 172.21.15.98:0/4067795060 shutdown_connections 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f93440a3ee0 0x7f93440a63a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c10c520 0x7f935c19ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f935c103b00 0x7f935c198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 --2- 172.21.15.98:0/4067795060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f935c1031e0 0x7f935c198610 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 -- 172.21.15.98:0/4067795060 >> 172.21.15.98:0/4067795060 conn(0x7f935c077fd0 msgr2=0x7f935c0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 -- 172.21.15.98:0/4067795060 shutdown_connections 2024-04-25T17:36:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:09.739+0000 7f93620bb640 1 -- 172.21.15.98:0/4067795060 wait complete. 2024-04-25T17:36:09.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:09 smithi173 ceph-mon[32856]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 98 KiB/s wr, 62 op/s 2024-04-25T17:36:10.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.248+0000 7f5ace672700 1 -- 172.21.15.98:0/3388801844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac806b740 msgr2=0x7f5ac806bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:10.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.248+0000 7f5ace672700 1 --2- 172.21.15.98:0/3388801844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac806b740 0x7f5ac806bba0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009990 tx=0x7f5abc009ca0 comp rx=0 tx=0).stop 2024-04-25T17:36:10.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.248+0000 7f5ace672700 1 -- 172.21.15.98:0/3388801844 shutdown_connections 2024-04-25T17:36:10.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.248+0000 7f5ace672700 1 --2- 172.21.15.98:0/3388801844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac8105480 0x7f5ac8105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:10.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.248+0000 7f5ace672700 1 --2- 172.21.15.98:0/3388801844 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ac8107970 0x7f5ac806b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.248+0000 7f5ace672700 1 --2- 172.21.15.98:0/3388801844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac806b740 0x7f5ac806bba0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.248+0000 7f5ace672700 1 -- 172.21.15.98:0/3388801844 >> 172.21.15.98:0/3388801844 conn(0x7f5ac80788f0 msgr2=0x7f5ac8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.248+0000 7f5ace672700 1 -- 172.21.15.98:0/3388801844 shutdown_connections 2024-04-25T17:36:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.248+0000 7f5ace672700 1 -- 172.21.15.98:0/3388801844 wait complete. 2024-04-25T17:36:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5ace672700 1 Processor -- start 2024-04-25T17:36:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5ace672700 1 -- start start 2024-04-25T17:36:10.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5ace672700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ac806b740 0x7f5ac81a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:10.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5ace672700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac8105480 0x7f5ac81a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:10.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5ace672700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8107970 0x7f5ac819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:10.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5ace672700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac81192b0 con 0x7f5ac8107970 2024-04-25T17:36:10.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5ace672700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac8119130 con 0x7f5ac806b740 2024-04-25T17:36:10.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5ace672700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac8119430 con 0x7f5ac8105480 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5accc0f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8107970 0x7f5ac819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5accc0f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8107970 0x7f5ac819f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33698/0 (socket says 172.21.15.98:33698) 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5accc0f700 1 -- 172.21.15.98:0/1513342966 learned_addr learned my addr 172.21.15.98:0/1513342966 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5ac7fff700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ac806b740 0x7f5ac81a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5ac77fe700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac8105480 0x7f5ac81a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5accc0f700 1 -- 172.21.15.98:0/1513342966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac8105480 msgr2=0x7f5ac81a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5accc0f700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac8105480 0x7f5ac81a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5accc0f700 1 -- 172.21.15.98:0/1513342966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ac806b740 msgr2=0x7f5ac81a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.249+0000 7f5accc0f700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ac806b740 0x7f5ac81a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.250+0000 7f5accc0f700 1 -- 172.21.15.98:0/1513342966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac819ff30 con 0x7f5ac8107970 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.250+0000 7f5accc0f700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8107970 0x7f5ac819f650 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000b970 tx=0x7f5ac000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.250+0000 7f5ac57fa700 1 -- 172.21.15.98:0/1513342966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0019070 con 0x7f5ac8107970 2024-04-25T17:36:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.250+0000 7f5ace672700 1 -- 172.21.15.98:0/1513342966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac81a0220 con 0x7f5ac8107970 2024-04-25T17:36:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.250+0000 7f5ac57fa700 1 -- 172.21.15.98:0/1513342966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ac0014440 con 0x7f5ac8107970 2024-04-25T17:36:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.250+0000 7f5ace672700 1 -- 172.21.15.98:0/1513342966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac81ac500 con 0x7f5ac8107970 2024-04-25T17:36:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.250+0000 7f5ac57fa700 1 -- 172.21.15.98:0/1513342966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac000ce60 con 0x7f5ac8107970 2024-04-25T17:36:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.251+0000 7f5ace672700 1 -- 172.21.15.98:0/1513342966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac8101f60 con 0x7f5ac8107970 2024-04-25T17:36:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.252+0000 7f5ac57fa700 1 -- 172.21.15.98:0/1513342966 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f5ac001c070 con 0x7f5ac8107970 2024-04-25T17:36:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.253+0000 7f5ac57fa700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5ab00a3ed0 0x7f5ab00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.253+0000 7f5ac7fff700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5ab00a3ed0 0x7f5ab00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.253+0000 7f5ac57fa700 1 -- 172.21.15.98:0/1513342966 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f5ac00d02f0 con 0x7f5ac8107970 2024-04-25T17:36:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.253+0000 7f5ac7fff700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5ab00a3ed0 0x7f5ab00a6390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8006fd0 tx=0x7f5ab8008040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.257+0000 7f5ac57fa700 1 -- 172.21.15.98:0/1513342966 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ac009c880 con 0x7f5ac8107970 2024-04-25T17:36:10.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.501+0000 7f5ace672700 1 -- 172.21.15.98:0/1513342966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ac81ac6d0 con 0x7f5ac8107970 2024-04-25T17:36:10.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.502+0000 7f5ac57fa700 1 -- 172.21.15.98:0/1513342966 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11877 (secure 0 0 0) 0x7f5ac009c180 con 0x7f5ac8107970 2024-04-25T17:36:10.503 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:35:53.134159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 -- 172.21.15.98:0/1513342966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5ab00a3ed0 msgr2=0x7f5ab00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5ab00a3ed0 0x7f5ab00a6390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8006fd0 tx=0x7f5ab8008040 comp rx=0 tx=0).stop 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 -- 172.21.15.98:0/1513342966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8107970 msgr2=0x7f5ac819f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8107970 0x7f5ac819f650 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000b970 tx=0x7f5ac000bd30 comp rx=0 tx=0).stop 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 -- 172.21.15.98:0/1513342966 shutdown_connections 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5ab00a3ed0 0x7f5ab00a6390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ac8107970 0x7f5ac819f650 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5ac806b740 0x7f5ac81a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 --2- 172.21.15.98:0/1513342966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac8105480 0x7f5ac81a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 -- 172.21.15.98:0/1513342966 >> 172.21.15.98:0/1513342966 conn(0x7f5ac80788f0 msgr2=0x7f5ac8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 -- 172.21.15.98:0/1513342966 shutdown_connections 2024-04-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:10.504+0000 7f5ace672700 1 -- 172.21.15.98:0/1513342966 wait complete. 2024-04-25T17:36:10.506 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 26 2024-04-25T17:36:11.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T17:36:11.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:11 smithi098 ceph-mon[29192]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 63 op/s 2024-04-25T17:36:11.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:11 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1513342966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:11.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:11 smithi203 ceph-mon[32834]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 63 op/s 2024-04-25T17:36:11.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:11 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1513342966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:11.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:11 smithi173 ceph-mon[32856]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 99 KiB/s wr, 63 op/s 2024-04-25T17:36:11.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:11 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1513342966' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.297+0000 7f37d6860700 1 -- 172.21.15.98:0/1863712749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0075590 msgr2=0x7f37d01136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.297+0000 7f37d6860700 1 --2- 172.21.15.98:0/1863712749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0075590 0x7f37d01136d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009990 tx=0x7f37c4009ca0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.297+0000 7f37d6860700 1 -- 172.21.15.98:0/1863712749 shutdown_connections 2024-04-25T17:36:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.297+0000 7f37d6860700 1 --2- 172.21.15.98:0/1863712749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d0074c70 0x7f37d0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.297+0000 7f37d6860700 1 --2- 172.21.15.98:0/1863712749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37d0113c10 0x7f37d0116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.297+0000 7f37d6860700 1 --2- 172.21.15.98:0/1863712749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0075590 0x7f37d01136d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.297+0000 7f37d6860700 1 -- 172.21.15.98:0/1863712749 >> 172.21.15.98:0/1863712749 conn(0x7f37d0079ba0 msgr2=0x7f37d0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:13.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.297+0000 7f37d6860700 1 -- 172.21.15.98:0/1863712749 shutdown_connections 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.297+0000 7f37d6860700 1 -- 172.21.15.98:0/1863712749 wait complete. 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.297+0000 7f37d6860700 1 Processor -- start 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d6860700 1 -- start start 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d6860700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d0074c70 0x7f37d01a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d6860700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37d0075590 0x7f37d01a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d6860700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0113c10 0x7f37d019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d6860700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f37d0119210 con 0x7f37d0113c10 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d6860700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f37d0119090 con 0x7f37d0075590 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d6860700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f37d0119390 con 0x7f37d0074c70 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d4dfd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0113c10 0x7f37d019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d4dfd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0113c10 0x7f37d019f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33714/0 (socket says 172.21.15.98:33714) 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d4dfd700 1 -- 172.21.15.98:0/1478747487 learned_addr learned my addr 172.21.15.98:0/1478747487 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37cffff700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d0074c70 0x7f37d01a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37cf7fe700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37d0075590 0x7f37d01a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:13.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d4dfd700 1 -- 172.21.15.98:0/1478747487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d0074c70 msgr2=0x7f37d01a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d4dfd700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d0074c70 0x7f37d01a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d4dfd700 1 -- 172.21.15.98:0/1478747487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37d0075590 msgr2=0x7f37d01a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d4dfd700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37d0075590 0x7f37d01a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.298+0000 7f37d4dfd700 1 -- 172.21.15.98:0/1478747487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d019fd10 con 0x7f37d0113c10 2024-04-25T17:36:13.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.299+0000 7f37cf7fe700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37d0075590 0x7f37d01a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:36:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.299+0000 7f37d4dfd700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0113c10 0x7f37d019f5b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f37c80094f0 tx=0x7f37c800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.299+0000 7f37cd7fa700 1 -- 172.21.15.98:0/1478747487 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c800ddd0 con 0x7f37d0113c10 2024-04-25T17:36:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.299+0000 7f37d6860700 1 -- 172.21.15.98:0/1478747487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d01a0000 con 0x7f37d0113c10 2024-04-25T17:36:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.299+0000 7f37cd7fa700 1 -- 172.21.15.98:0/1478747487 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37c8009e20 con 0x7f37d0113c10 2024-04-25T17:36:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.299+0000 7f37cd7fa700 1 -- 172.21.15.98:0/1478747487 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c8015400 con 0x7f37d0113c10 2024-04-25T17:36:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.299+0000 7f37d6860700 1 -- 172.21.15.98:0/1478747487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d01ac2c0 con 0x7f37d0113c10 2024-04-25T17:36:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.300+0000 7f37d6860700 1 -- 172.21.15.98:0/1478747487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d0069730 con 0x7f37d0113c10 2024-04-25T17:36:13.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.301+0000 7f37cd7fa700 1 -- 172.21.15.98:0/1478747487 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f37c8005020 con 0x7f37d0113c10 2024-04-25T17:36:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.302+0000 7f37cd7fa700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f37b80a3f50 0x7f37b80a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.302+0000 7f37cffff700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f37b80a3f50 0x7f37b80a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.302+0000 7f37cd7fa700 1 -- 172.21.15.98:0/1478747487 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f37c80d0430 con 0x7f37d0113c10 2024-04-25T17:36:13.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.302+0000 7f37cffff700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f37b80a3f50 0x7f37b80a6410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f37c0004630 tx=0x7f37c000b3c0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:13.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.307+0000 7f37cd7fa700 1 -- 172.21.15.98:0/1478747487 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f37c8010070 con 0x7f37d0113c10 2024-04-25T17:36:13.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.506+0000 7f37d6860700 1 -- 172.21.15.98:0/1478747487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37d01ac490 con 0x7f37d0113c10 2024-04-25T17:36:13.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.508+0000 7f37cd7fa700 1 -- 172.21.15.98:0/1478747487 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+23063 (secure 0 0 0) 0x7f37c809c8b0 con 0x7f37d0113c10 2024-04-25T17:36:13.509 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:13.509 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":92,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:35:54.302685+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T17:32:30.267079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T17:32:33.956902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/4145487482":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/674729727":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/1530074164":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/2167465649":"2024-04-26T17:35:54.302654+0000","172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3206373024":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/4145487482":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:36:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 -- 172.21.15.98:0/1478747487 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f37b80a3f50 msgr2=0x7f37b80a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f37b80a3f50 0x7f37b80a6410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f37c0004630 tx=0x7f37c000b3c0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 -- 172.21.15.98:0/1478747487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0113c10 msgr2=0x7f37d019f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0113c10 0x7f37d019f5b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f37c80094f0 tx=0x7f37c800d3b0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 -- 172.21.15.98:0/1478747487 shutdown_connections 2024-04-25T17:36:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f37b80a3f50 0x7f37b80a6410 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37d0074c70 0x7f37d01a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37d0113c10 0x7f37d019f5b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 --2- 172.21.15.98:0/1478747487 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37d0075590 0x7f37d01a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 -- 172.21.15.98:0/1478747487 >> 172.21.15.98:0/1478747487 conn(0x7f37d0079ba0 msgr2=0x7f37d0112ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 -- 172.21.15.98:0/1478747487 shutdown_connections 2024-04-25T17:36:13.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:13.510+0000 7f37d6860700 1 -- 172.21.15.98:0/1478747487 wait complete. 2024-04-25T17:36:13.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:13 smithi098 ceph-mon[29192]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 63 KiB/s wr, 41 op/s 2024-04-25T17:36:13.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:13 smithi203 ceph-mon[32834]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 63 KiB/s wr, 41 op/s 2024-04-25T17:36:13.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:13 smithi173 ceph-mon[32856]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 63 KiB/s wr, 41 op/s 2024-04-25T17:36:14.232 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-04-25T17:36:14.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:14 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1478747487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:36:14.477 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:14.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:14 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1478747487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:36:14.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:14 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1478747487' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:36:16.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:15 smithi098 ceph-mon[29192]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T17:36:16.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:15 smithi173 ceph-mon[32856]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T17:36:16.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:15 smithi203 ceph-mon[32834]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T17:36:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:{"run_dir":"/var/run/ceph"} 2024-04-25T17:36:17.082 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-04-25T17:36:17.082 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 395146 2024-04-25T17:36:17.082 DEBUG:tasks.quiescer:Yielding 2024-04-25T17:36:17.083 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T17:36:17.092 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T17:36:17.093 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:36:17.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-04-25T17:36:17.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:36:17.128 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:17.139 INFO:teuthology.orchestra.run.smithi098.stdout:Filesystem Size Used Avail Use% Mounted on 2024-04-25T17:36:17.139 INFO:teuthology.orchestra.run.smithi098.stdout:0@79754014-0328-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/8b073cd4-d70f-47d4-966e-81c60974204b 340G 0 340G 0% /tmp/cephfs 2024-04-25T17:36:17.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-04-25T17:36:17.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-04-25T17:36:17.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-04-25T17:36:17.342 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:18.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:17 smithi098 ceph-mon[29192]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-25T17:36:18.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:17 smithi173 ceph-mon[32856]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-25T17:36:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:17 smithi203 ceph-mon[32834]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-25T17:36:19.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-04-25T17:36:19.253 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-04-25T17:36:19.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-04-25T17:36:19.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-25T17:36:19.404 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/: 2024-04-25T17:36:19.404 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:19.404 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 . 2024-04-25T17:36:19.404 INFO:teuthology.orchestra.run.smithi098.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 .. 2024-04-25T17:36:19.404 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 data 2024-04-25T17:36:19.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:19.404 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data: 2024-04-25T17:36:19.405 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:19.405 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:19.405 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 .. 2024-04-25T17:36:19.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-04-25T17:36:19.488 INFO:teuthology.orchestra.run.smithi098.stdout:[Service] 2024-04-25T17:36:19.488 INFO:teuthology.orchestra.run.smithi098.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-04-25T17:36:19.488 INFO:teuthology.orchestra.run.smithi098.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-04-25T17:36:19.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-04-25T17:36:19.654 INFO:teuthology.orchestra.run.smithi098.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-04-25T17:36:20.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.045+0000 7f53b2ad0700 1 -- 172.21.15.98:0/897024132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53ac102300 msgr2=0x7f53ac10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:20.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.045+0000 7f53b2ad0700 1 --2- 172.21.15.98:0/897024132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53ac102300 0x7f53ac10e7f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f53a400b0a0 tx=0x7f53a400b3b0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.045+0000 7f53b2ad0700 1 -- 172.21.15.98:0/897024132 shutdown_connections 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.045+0000 7f53b2ad0700 1 --2- 172.21.15.98:0/897024132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53ac102300 0x7f53ac10e7f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.045+0000 7f53b2ad0700 1 --2- 172.21.15.98:0/897024132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac107990 0x7f53ac107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.045+0000 7f53b2ad0700 1 --2- 172.21.15.98:0/897024132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac101940 0x7f53ac101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.045+0000 7f53b2ad0700 1 -- 172.21.15.98:0/897024132 >> 172.21.15.98:0/897024132 conn(0x7f53ac0788f0 msgr2=0x7f53ac078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.045+0000 7f53b2ad0700 1 -- 172.21.15.98:0/897024132 shutdown_connections 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.045+0000 7f53b2ad0700 1 -- 172.21.15.98:0/897024132 wait complete. 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b2ad0700 1 Processor -- start 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b2ad0700 1 -- start start 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b2ad0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53ac101940 0x7f53ac1034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b2ad0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac102300 0x7f53ac103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b2ad0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac107990 0x7f53ac103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b2ad0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac111af0 con 0x7f53ac101940 2024-04-25T17:36:20.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b2ad0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac111970 con 0x7f53ac102300 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b2ad0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac111c70 con 0x7f53ac107990 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b086c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53ac101940 0x7f53ac1034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b086c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53ac101940 0x7f53ac1034f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53898/0 (socket says 172.21.15.98:53898) 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b086c700 1 -- 172.21.15.98:0/2072153083 learned_addr learned my addr 172.21.15.98:0/2072153083 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53abfff700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac102300 0x7f53ac103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b106d700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac107990 0x7f53ac103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b086c700 1 -- 172.21.15.98:0/2072153083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac107990 msgr2=0x7f53ac103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b086c700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac107990 0x7f53ac103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b086c700 1 -- 172.21.15.98:0/2072153083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac102300 msgr2=0x7f53ac103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b086c700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac102300 0x7f53ac103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.046+0000 7f53b086c700 1 -- 172.21.15.98:0/2072153083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac1a7600 con 0x7f53ac101940 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.047+0000 7f53b086c700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53ac101940 0x7f53ac1034f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f53a000e9a0 tx=0x7f53a000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.047+0000 7f53a9ffb700 1 -- 172.21.15.98:0/2072153083 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a000cbd0 con 0x7f53ac101940 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.047+0000 7f53b2ad0700 1 -- 172.21.15.98:0/2072153083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ac1a78f0 con 0x7f53ac101940 2024-04-25T17:36:20.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.047+0000 7f53a9ffb700 1 -- 172.21.15.98:0/2072153083 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53a000cd70 con 0x7f53ac101940 2024-04-25T17:36:20.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.047+0000 7f53a9ffb700 1 -- 172.21.15.98:0/2072153083 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0018680 con 0x7f53ac101940 2024-04-25T17:36:20.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.047+0000 7f53b2ad0700 1 -- 172.21.15.98:0/2072153083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ac1a7e30 con 0x7f53ac101940 2024-04-25T17:36:20.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.048+0000 7f53b2ad0700 1 -- 172.21.15.98:0/2072153083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53ac069730 con 0x7f53ac101940 2024-04-25T17:36:20.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.049+0000 7f53a9ffb700 1 -- 172.21.15.98:0/2072153083 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f53a0010430 con 0x7f53ac101940 2024-04-25T17:36:20.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.050+0000 7f53a9ffb700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f539c0a3ed0 0x7f539c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:20.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.050+0000 7f53a9ffb700 1 -- 172.21.15.98:0/2072153083 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f53a00d0c20 con 0x7f53ac101940 2024-04-25T17:36:20.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.050+0000 7f53abfff700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f539c0a3ed0 0x7f539c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:20.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.050+0000 7f53abfff700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f539c0a3ed0 0x7f539c0a6390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f53ac079040 tx=0x7f5398009260 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:20.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.055+0000 7f53a9ffb700 1 -- 172.21.15.98:0/2072153083 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f53a009d0a0 con 0x7f53ac101940 2024-04-25T17:36:20.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:19 smithi098 ceph-mon[29192]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T17:36:20.265 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:19 smithi203 ceph-mon[32834]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T17:36:20.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:19 smithi173 ceph-mon[32856]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-25T17:36:20.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.326+0000 7f53b2ad0700 1 -- 172.21.15.98:0/2072153083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53ac02df80 con 0x7f53ac101940 2024-04-25T17:36:20.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.326+0000 7f53a9ffb700 1 -- 172.21.15.98:0/2072153083 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11877 (secure 0 0 0) 0x7f53a009c9a0 con 0x7f53ac101940 2024-04-25T17:36:20.327 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:20.327 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:35:53.134159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:36:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 -- 172.21.15.98:0/2072153083 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f539c0a3ed0 msgr2=0x7f539c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f539c0a3ed0 0x7f539c0a6390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f53ac079040 tx=0x7f5398009260 comp rx=0 tx=0).stop 2024-04-25T17:36:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 -- 172.21.15.98:0/2072153083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53ac101940 msgr2=0x7f53ac1034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53ac101940 0x7f53ac1034f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f53a000e9a0 tx=0x7f53a000ecb0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 -- 172.21.15.98:0/2072153083 shutdown_connections 2024-04-25T17:36:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f539c0a3ed0 0x7f539c0a6390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53ac107990 0x7f53ac103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53ac101940 0x7f53ac1034f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 --2- 172.21.15.98:0/2072153083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f53ac102300 0x7f53ac103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:20.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 -- 172.21.15.98:0/2072153083 >> 172.21.15.98:0/2072153083 conn(0x7f53ac0788f0 msgr2=0x7f53ac06aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:20.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 -- 172.21.15.98:0/2072153083 shutdown_connections 2024-04-25T17:36:20.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:20.329+0000 7f53b2ad0700 1 -- 172.21.15.98:0/2072153083 wait complete. 2024-04-25T17:36:20.331 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 26 2024-04-25T17:36:21.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T17:32:37.173463+0000', 'modified': '2024-04-25T17:35:53.134159+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24461, 'mds_1': 24445, 'mds_2': 24440}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24422': {'gid': 24422, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6835/2662617129', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 2662617129}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 2662617129}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/2803298626', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 2803298626}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 2803298626}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6837/1057173981', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 1057173981}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 1057173981}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.173:6837/1096095017', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 1096095017}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 1096095017}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.203:6839/1340539385', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 1340539385}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 1340539385}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24461': {'gid': 24461, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.173:6839/2342732992', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 2342732992}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 2342732992}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24461, 'qdb_cluster': [24445, 24461, 24440]} 2024-04-25T17:36:21.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T17:36:21.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell mds.h status 2024-04-25T17:36:21.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:20 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2072153083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:21.414 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:21.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:20 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2072153083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:21.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:20 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2072153083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:22.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:21 smithi098 ceph-mon[29192]: pgmap v16: 97 pgs: 97 active+clean; 1.8 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 52 KiB/s wr, 13 op/s 2024-04-25T17:36:22.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:21 smithi173 ceph-mon[32856]: pgmap v16: 97 pgs: 97 active+clean; 1.8 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 52 KiB/s wr, 13 op/s 2024-04-25T17:36:22.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:21 smithi203 ceph-mon[32834]: pgmap v16: 97 pgs: 97 active+clean; 1.8 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 52 KiB/s wr, 13 op/s 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.827+0000 7f971d330640 1 -- 172.21.15.98:0/383417015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97181006c0 msgr2=0x7f9718100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.827+0000 7f971d330640 1 --2- 172.21.15.98:0/383417015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97181006c0 0x7f9718100aa0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9708009900 tx=0x7f970802a410 comp rx=0 tx=0).stop 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 -- 172.21.15.98:0/383417015 shutdown_connections 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 --2- 172.21.15.98:0/383417015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9718107330 0x7f971810eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 --2- 172.21.15.98:0/383417015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9718101070 0x7f9718106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 --2- 172.21.15.98:0/383417015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97181006c0 0x7f9718100aa0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 -- 172.21.15.98:0/383417015 >> 172.21.15.98:0/383417015 conn(0x7f9718077390 msgr2=0x7f97180fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 -- 172.21.15.98:0/383417015 shutdown_connections 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 -- 172.21.15.98:0/383417015 wait complete. 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 Processor -- start 2024-04-25T17:36:22.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 -- start start 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97181006c0 0x7f9718198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9718107330 0x7f9718198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971819cfa0 0x7f971819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9718112250 con 0x7f97181006c0 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f97181120d0 con 0x7f9718107330 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f971d330640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f97181123d0 con 0x7f971819cfa0 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f9716ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97181006c0 0x7f9718198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f9716ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97181006c0 0x7f9718198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53912/0 (socket says 172.21.15.98:53912) 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f9716ffd640 1 -- 172.21.15.98:0/855831443 learned_addr learned my addr 172.21.15.98:0/855831443 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f97177fe640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971819cfa0 0x7f971819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.828+0000 7f970ffff640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9718107330 0x7f9718198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f9716ffd640 1 -- 172.21.15.98:0/855831443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971819cfa0 msgr2=0x7f971819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f9716ffd640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971819cfa0 0x7f971819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f9716ffd640 1 -- 172.21.15.98:0/855831443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9718107330 msgr2=0x7f9718198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f9716ffd640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9718107330 0x7f9718198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f9716ffd640 1 -- 172.21.15.98:0/855831443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971819da90 con 0x7f97181006c0 2024-04-25T17:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f9716ffd640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97181006c0 0x7f9718198730 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9708005940 tx=0x7f9708004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f9714ff9640 1 -- 172.21.15.98:0/855831443 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970803f030 con 0x7f97181006c0 2024-04-25T17:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f971d330640 1 -- 172.21.15.98:0/855831443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f971819dd20 con 0x7f97181006c0 2024-04-25T17:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f9714ff9640 1 -- 172.21.15.98:0/855831443 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9708003aa0 con 0x7f97181006c0 2024-04-25T17:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f9714ff9640 1 -- 172.21.15.98:0/855831443 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9708035070 con 0x7f97181006c0 2024-04-25T17:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.829+0000 7f971d330640 1 -- 172.21.15.98:0/855831443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9718101800 con 0x7f97181006c0 2024-04-25T17:36:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f9714ff9640 1 -- 172.21.15.98:0/855831443 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f9708042030 con 0x7f97181006c0 2024-04-25T17:36:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f9720049740 1 Processor -- start 2024-04-25T17:36:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f9720049740 1 -- start start 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f9720049740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e717a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f9720049740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e717f40 0x55843e7222c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f9720049740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 0x55843e724bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f9720049740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55843e67c970 con 0x55843e717620 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f9720049740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55843e6194a0 con 0x55843e722800 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f9720049740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55843e678930 con 0x55843e717f40 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f97177fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e717a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.832+0000 7f97177fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e717a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53920/0 (socket says 172.21.15.98:53920) 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f970ffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 0x55843e724bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f97177fe640 1 -- 172.21.15.98:0/1546977125 learned_addr learned my addr 172.21.15.98:0/1546977125 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f9714ff9640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f97040a3ee0 0x7f97040a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f9716ffd640 1 --2- 172.21.15.98:0/1546977125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e717f40 0x55843e7222c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f97177fe640 1 -- 172.21.15.98:0/1546977125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e717f40 msgr2=0x55843e7222c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f97177fe640 1 --2- 172.21.15.98:0/1546977125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e717f40 0x55843e7222c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f97177fe640 1 -- 172.21.15.98:0/1546977125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 msgr2=0x55843e724bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f97177fe640 1 --2- 172.21.15.98:0/1546977125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 0x55843e724bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f97177fe640 1 -- 172.21.15.98:0/1546977125 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55843e38f210 con 0x55843e717620 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f970ffff640 1 --2- 172.21.15.98:0/1546977125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 0x55843e724bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f9714ff9640 1 -- 172.21.15.98:0/855831443 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f97080f0ff0 con 0x7f97181006c0 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f97177fe640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f97040a3ee0 0x7f97040a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f97177fe640 1 --2- 172.21.15.98:0/1546977125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e717a00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9710010440 tx=0x7f9710010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ec77e5bfb82663 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f970d7fa640 1 -- 172.21.15.98:0/1546977125 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710022070 con 0x55843e717620 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f970d7fa640 1 -- 172.21.15.98:0/1546977125 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f971001c390 con 0x55843e717620 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.833+0000 7f970d7fa640 1 -- 172.21.15.98:0/1546977125 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97100253a0 con 0x55843e717620 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f9720049740 1 -- 172.21.15.98:0/1546977125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 msgr2=0x55843e717a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f97177fe640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f97040a3ee0 0x7f97040a63a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f971001f000 tx=0x7f971001d000 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f9720049740 1 --2- 172.21.15.98:0/1546977125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e717a00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9710010440 tx=0x7f9710010900 comp rx=0 tx=0).stop 2024-04-25T17:36:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f9720049740 1 -- 172.21.15.98:0/1546977125 shutdown_connections 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f9720049740 1 --2- 172.21.15.98:0/1546977125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 0x55843e724bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f9720049740 1 --2- 172.21.15.98:0/1546977125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e717f40 0x55843e7222c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f9720049740 1 --2- 172.21.15.98:0/1546977125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e717a00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f9720049740 1 -- 172.21.15.98:0/1546977125 >> 172.21.15.98:0/1546977125 conn(0x55843e713b30 msgr2=0x55843e713f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f9720049740 1 -- 172.21.15.98:0/1546977125 shutdown_connections 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f9720049740 1 -- 172.21.15.98:0/1546977125 wait complete. 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.834+0000 7f9720049740 1 -- start start 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9720049740 1 Processor -- start 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9720049740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e722130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9720049740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e717f40 0x55843e71f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9720049740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e722800 0x55843e71f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9720049740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55843e725430 con 0x55843e717620 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9720049740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55843e6170b0 con 0x55843e717f40 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9720049740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55843e7255b0 con 0x55843e722800 2024-04-25T17:36:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9720049740 10 client.0 _ll_register_callbacks cb 0x55843e604d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9716ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e722130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9716ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e722130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53936/0 (socket says 172.21.15.98:53936) 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9716ffd640 1 -- 172.21.15.98:0/1352681401 learned_addr learned my addr 172.21.15.98:0/1352681401 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f970ffff640 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e717f40 0x55843e71f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f97177fe640 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e722800 0x55843e71f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9716ffd640 1 -- 172.21.15.98:0/1352681401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e722800 msgr2=0x55843e71f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9716ffd640 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e722800 0x55843e71f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9716ffd640 1 -- 172.21.15.98:0/1352681401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e717f40 msgr2=0x55843e71f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9716ffd640 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e717f40 0x55843e71f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9716ffd640 1 -- 172.21.15.98:0/1352681401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55843e45b620 con 0x55843e717620 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f9716ffd640 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e722130 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f97080bce80 tx=0x7f97080bcd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5efcece5e2579484 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f96fe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1352681401 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9708049070 con 0x55843e717620 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.835+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1352681401 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f970803bcc0 con 0x55843e717620 2024-04-25T17:36:22.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f9720049740 1 -- 172.21.15.98:0/1352681401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f971001c390 con 0x55843e717620 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1352681401 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f97080c5ab0 con 0x55843e717620 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f9720049740 10 client.15297 fetch_fsmap learned FSMap version 26 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f9720049740 1 -- 172.21.15.98:0/1352681401 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55843e731340 con 0x55843e717620 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1352681401 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f971001c390 con 0x55843e717620 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1352681401 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f97080f3110 con 0x55843e717620 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f9720049740 10 client.15297 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f9720049740 10 client.15297 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f9720049740 20 client.15297 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f9720049740 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55843e7327a0 0x55843e734c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f9720049740 4 client.15297 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f9720049740 1 -- 172.21.15.98:0/1352681401 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55843e68a390 con 0x55843e7327a0 2024-04-25T17:36:22.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.836+0000 7f970ffff640 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55843e7327a0 0x55843e734c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.837+0000 7f96fe7fc640 10 client.15297 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:22.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.837+0000 7f970ffff640 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55843e7327a0 0x55843e734c10 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=136c7eaf22308129 server_cookie=65749fd50181e2e2 in_seq=0 out_seq=0 2024-04-25T17:36:22.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.838+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1352681401 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55843e68a390 con 0x55843e7327a0 2024-04-25T17:36:22.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.838+0000 7f96fe7fc640 10 client.15297 handle_command_reply: tid=0 2024-04-25T17:36:22.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.838+0000 7f9720049740 1 client.15297 shutdown 2024-04-25T17:36:22.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.838+0000 7f9720049740 1 -- 172.21.15.98:0/1352681401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 msgr2=0x55843e722130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:22.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f9720049740 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e722130 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f97080bce80 tx=0x7f97080bcd80 comp rx=0 tx=0).stop 2024-04-25T17:36:22.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f9720049740 1 -- 172.21.15.98:0/1352681401 shutdown_connections 2024-04-25T17:36:22.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f9720049740 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55843e7327a0 0x55843e734c10 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f9720049740 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e722800 0x55843e71f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f9720049740 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e717f40 0x55843e71f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f96fe7fc640 0 client.15297 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f9720049740 1 --2- 172.21.15.98:0/1352681401 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e722130 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f9720049740 1 -- 172.21.15.98:0/1352681401 >> 172.21.15.98:0/1352681401 conn(0x55843e713b30 msgr2=0x55843e719860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f9720049740 1 -- 172.21.15.98:0/1352681401 shutdown_connections 2024-04-25T17:36:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f9720049740 1 -- 172.21.15.98:0/1352681401 wait complete. 2024-04-25T17:36:22.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.839+0000 7f9720049740 20 client.15297 trim_cache size 0 max 16384 2024-04-25T17:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f9720049740 1 Processor -- start 2024-04-25T17:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f9720049740 1 -- start start 2024-04-25T17:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f9720049740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e717620 0x55843e72e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f9720049740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717f40 0x55843e721c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f9720049740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e722800 0x55843e73a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f9720049740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97181b25c0 con 0x55843e717f40 2024-04-25T17:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f9720049740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f97181afe90 con 0x55843e717620 2024-04-25T17:36:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f9720049740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f97181add00 con 0x55843e722800 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f970ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717f40 0x55843e721c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f970ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717f40 0x55843e721c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53952/0 (socket says 172.21.15.98:53952) 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f970ffff640 1 -- 172.21.15.98:0/587939631 learned_addr learned my addr 172.21.15.98:0/587939631 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.854+0000 7f9716ffd640 1 --2- 172.21.15.98:0/587939631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e717620 0x55843e72e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.855+0000 7f97177fe640 1 --2- 172.21.15.98:0/587939631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e722800 0x55843e73a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.855+0000 7f970ffff640 1 -- 172.21.15.98:0/587939631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e722800 msgr2=0x55843e73a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.855+0000 7f970ffff640 1 --2- 172.21.15.98:0/587939631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e722800 0x55843e73a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.855+0000 7f970ffff640 1 -- 172.21.15.98:0/587939631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e717620 msgr2=0x55843e72e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.855+0000 7f970ffff640 1 --2- 172.21.15.98:0/587939631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e717620 0x55843e72e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.855+0000 7f970ffff640 1 -- 172.21.15.98:0/587939631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55843e718af0 con 0x55843e717f40 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.855+0000 7f970ffff640 1 --2- 172.21.15.98:0/587939631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717f40 0x55843e721c60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f971806a110 tx=0x7f9700005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb47045659c347fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.855+0000 7f970cff9640 1 -- 172.21.15.98:0/587939631 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700003e60 con 0x55843e717f40 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.855+0000 7f970cff9640 1 -- 172.21.15.98:0/587939631 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97000205a0 con 0x55843e717f40 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.855+0000 7f970cff9640 1 -- 172.21.15.98:0/587939631 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700010c00 con 0x55843e717f40 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 -- 172.21.15.98:0/587939631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717f40 msgr2=0x55843e721c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 --2- 172.21.15.98:0/587939631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717f40 0x55843e721c60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f971806a110 tx=0x7f9700005f90 comp rx=0 tx=0).stop 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 -- 172.21.15.98:0/587939631 shutdown_connections 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 --2- 172.21.15.98:0/587939631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e722800 0x55843e73a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 --2- 172.21.15.98:0/587939631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717f40 0x55843e721c60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 --2- 172.21.15.98:0/587939631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e717620 0x55843e72e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 -- 172.21.15.98:0/587939631 >> 172.21.15.98:0/587939631 conn(0x55843e713b30 msgr2=0x55843e719860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 -- 172.21.15.98:0/587939631 shutdown_connections 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 -- 172.21.15.98:0/587939631 wait complete. 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 -- start start 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.856+0000 7f9720049740 1 Processor -- start 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9720049740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e72ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9720049740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e717f40 0x55843e725ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9720049740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 0x55843e7263e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9720049740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f97181ae350 con 0x55843e717620 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9720049740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f97181af7c0 con 0x55843e722800 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9720049740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f97181aea20 con 0x55843e717f40 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9720049740 10 client.0 _ll_register_callbacks cb 0x7f970803b4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9716ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e72ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9716ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e72ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53968/0 (socket says 172.21.15.98:53968) 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9716ffd640 1 -- 172.21.15.98:0/1577347010 learned_addr learned my addr 172.21.15.98:0/1577347010 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f970ffff640 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e717f40 0x55843e725ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f97177fe640 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 0x55843e7263e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9716ffd640 1 -- 172.21.15.98:0/1577347010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e717f40 msgr2=0x55843e725ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9716ffd640 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e717f40 0x55843e725ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9716ffd640 1 -- 172.21.15.98:0/1577347010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 msgr2=0x55843e7263e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9716ffd640 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 0x55843e7263e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:22.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9716ffd640 1 -- 172.21.15.98:0/1577347010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55843e726be0 con 0x55843e717620 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f97177fe640 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 0x55843e7263e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9716ffd640 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e72ab00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f97080eed70 tx=0x7f97080b9190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a306494f577073d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f96fe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1577347010 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9708049040 con 0x55843e717620 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1577347010 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f97080c5670 con 0x55843e717620 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9720049740 1 -- 172.21.15.98:0/1577347010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97000205a0 con 0x55843e717620 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1577347010 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f97080bf080 con 0x55843e717620 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9720049740 10 client.15303 fetch_fsmap learned FSMap version 26 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.857+0000 7f9720049740 1 -- 172.21.15.98:0/1577347010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55843e798c80 con 0x55843e717620 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1577347010 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97000205a0 con 0x55843e717620 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1577347010 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f97080d6240 con 0x55843e717620 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f9720049740 10 client.15303 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f9720049740 10 client.15303 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f9720049740 20 client.15303 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:22.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f9720049740 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55843e799210 0x55843e79b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f9720049740 4 client.15303 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:36:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f9720049740 1 -- 172.21.15.98:0/1577347010 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55843e684200 con 0x55843e799210 2024-04-25T17:36:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f970ffff640 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55843e799210 0x55843e79b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f970ffff640 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55843e799210 0x55843e79b600 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26dde24c46f0f922 server_cookie=7af5d5475735038a in_seq=0 out_seq=0 2024-04-25T17:36:22.865 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:22.858+0000 7f96fe7fc640 10 client.15303 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:23.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f96fe7fc640 1 -- 172.21.15.98:0/1577347010 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55843e684200 con 0x55843e799210 2024-04-25T17:36:23.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f96fe7fc640 10 client.15303 handle_command_reply: tid=0 2024-04-25T17:36:23.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f9720049740 1 client.15303 shutdown 2024-04-25T17:36:23.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f9720049740 1 -- 172.21.15.98:0/1577347010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 msgr2=0x55843e72ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f9720049740 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e72ab00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f97080eed70 tx=0x7f97080b9190 comp rx=0 tx=0).stop 2024-04-25T17:36:23.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f9720049740 1 -- 172.21.15.98:0/1577347010 shutdown_connections 2024-04-25T17:36:23.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f9720049740 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55843e799210 0x55843e79b600 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f9720049740 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55843e722800 0x55843e7263e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f96fe7fc640 0 client.15303 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:23.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f9720049740 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55843e717f40 0x55843e725ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f9720049740 1 --2- 172.21.15.98:0/1577347010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55843e717620 0x55843e72ab00 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.086+0000 7f9720049740 1 -- 172.21.15.98:0/1577347010 >> 172.21.15.98:0/1577347010 conn(0x55843e713b30 msgr2=0x55843e712420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:23.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f9720049740 1 -- 172.21.15.98:0/1577347010 shutdown_connections 2024-04-25T17:36:23.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f9720049740 1 -- 172.21.15.98:0/1577347010 wait complete. 2024-04-25T17:36:23.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f9720049740 20 client.15303 trim_cache size 0 max 16384 2024-04-25T17:36:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:36:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:36:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:36:23.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:36:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 -- 172.21.15.98:0/855831443 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f97040a3ee0 msgr2=0x7f97040a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f97040a3ee0 0x7f97040a63a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f971001f000 tx=0x7f971001d000 comp rx=0 tx=0).stop 2024-04-25T17:36:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 -- 172.21.15.98:0/855831443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97181006c0 msgr2=0x7f9718198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97181006c0 0x7f9718198730 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9708005940 tx=0x7f9708004270 comp rx=0 tx=0).stop 2024-04-25T17:36:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 -- 172.21.15.98:0/855831443 shutdown_connections 2024-04-25T17:36:23.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f97040a3ee0 0x7f97040a63a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f971819cfa0 0x7f971819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9718107330 0x7f9718198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 --2- 172.21.15.98:0/855831443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f97181006c0 0x7f9718198730 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 -- 172.21.15.98:0/855831443 >> 172.21.15.98:0/855831443 conn(0x7f9718077390 msgr2=0x7f97180fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 -- 172.21.15.98:0/855831443 shutdown_connections 2024-04-25T17:36:23.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:23.087+0000 7f971d330640 1 -- 172.21.15.98:0/855831443 wait complete. 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.223+0000 7fec2176a700 1 -- 172.21.15.98:0/907140838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec1c102300 msgr2=0x7fec1c10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.223+0000 7fec2176a700 1 --2- 172.21.15.98:0/907140838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec1c102300 0x7fec1c10e7f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fec1400b0a0 tx=0x7fec1400b3b0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.223+0000 7fec2176a700 1 -- 172.21.15.98:0/907140838 shutdown_connections 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.223+0000 7fec2176a700 1 --2- 172.21.15.98:0/907140838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec1c107990 0x7fec1c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.223+0000 7fec2176a700 1 --2- 172.21.15.98:0/907140838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec1c102300 0x7fec1c10e7f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.223+0000 7fec2176a700 1 --2- 172.21.15.98:0/907140838 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec1c101940 0x7fec1c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.223+0000 7fec2176a700 1 -- 172.21.15.98:0/907140838 >> 172.21.15.98:0/907140838 conn(0x7fec1c0788f0 msgr2=0x7fec1c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.223+0000 7fec2176a700 1 -- 172.21.15.98:0/907140838 shutdown_connections 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.223+0000 7fec2176a700 1 -- 172.21.15.98:0/907140838 wait complete. 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.223+0000 7fec2176a700 1 Processor -- start 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec2176a700 1 -- start start 2024-04-25T17:36:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec2176a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec1c101940 0x7fec1c199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec2176a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec1c102300 0x7fec1c1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec2176a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec1c107990 0x7fec1c19d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec2176a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c111b30 con 0x7fec1c101940 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec2176a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c1119b0 con 0x7fec1c107990 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec2176a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c111cb0 con 0x7fec1c102300 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec1c101940 0x7fec1c199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec1c101940 0x7fec1c199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53986/0 (socket says 172.21.15.98:53986) 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1affd700 1 -- 172.21.15.98:0/1884089981 learned_addr learned my addr 172.21.15.98:0/1884089981 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1b7fe700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec1c107990 0x7fec1c19d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1a7fc700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec1c102300 0x7fec1c1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1affd700 1 -- 172.21.15.98:0/1884089981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec1c102300 msgr2=0x7fec1c1996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1affd700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec1c102300 0x7fec1c1996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1affd700 1 -- 172.21.15.98:0/1884089981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec1c107990 msgr2=0x7fec1c19d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1affd700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec1c107990 0x7fec1c19d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1affd700 1 -- 172.21.15.98:0/1884089981 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1c19e130 con 0x7fec1c101940 2024-04-25T17:36:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1a7fc700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec1c102300 0x7fec1c1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:36:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.224+0000 7fec1affd700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec1c101940 0x7fec1c199170 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c9b0 tx=0x7fec0c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.225+0000 7fec03fff700 1 -- 172.21.15.98:0/1884089981 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c007b00 con 0x7fec1c101940 2024-04-25T17:36:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.225+0000 7fec2176a700 1 -- 172.21.15.98:0/1884089981 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec1c19e420 con 0x7fec1c101940 2024-04-25T17:36:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.225+0000 7fec2176a700 1 -- 172.21.15.98:0/1884089981 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec1c1a5ca0 con 0x7fec1c101940 2024-04-25T17:36:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.225+0000 7fec03fff700 1 -- 172.21.15.98:0/1884089981 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec0c004ce0 con 0x7fec1c101940 2024-04-25T17:36:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.225+0000 7fec03fff700 1 -- 172.21.15.98:0/1884089981 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c005530 con 0x7fec1c101940 2024-04-25T17:36:23.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.227+0000 7fec03fff700 1 -- 172.21.15.98:0/1884089981 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fec0c004750 con 0x7fec1c101940 2024-04-25T17:36:23.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.228+0000 7fec03fff700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fec040a3ed0 0x7fec040a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.228+0000 7fec1a7fc700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fec040a3ed0 0x7fec040a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.228+0000 7fec03fff700 1 -- 172.21.15.98:0/1884089981 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fec0c0d0af0 con 0x7fec1c101940 2024-04-25T17:36:23.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.228+0000 7fec2bff3b80 1 Processor -- start 2024-04-25T17:36:23.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.228+0000 7fec1a7fc700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fec040a3ed0 0x7fec040a6390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec1c079080 tx=0x7fec10009260 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:23.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec2bff3b80 1 -- start start 2024-04-25T17:36:23.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a00b0 0x5606493a0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 0x5606493ab340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493ab880 0x5606493adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5606492f3c40 con 0x5606493ab880 2024-04-25T17:36:23.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56064929a8e0 con 0x5606493a09d0 2024-04-25T17:36:23.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5606492efc00 con 0x5606493a00b0 2024-04-25T17:36:23.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493ab880 0x5606493adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1a7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 0x5606493ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1affd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a00b0 0x5606493a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493ab880 0x5606493adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53994/0 (socket says 172.21.15.98:53994) 2024-04-25T17:36:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1b7fe700 1 -- 172.21.15.98:0/2256290572 learned_addr learned my addr 172.21.15.98:0/2256290572 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1b7fe700 1 -- 172.21.15.98:0/2256290572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a00b0 msgr2=0x5606493a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1b7fe700 1 --2- 172.21.15.98:0/2256290572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a00b0 0x5606493a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1b7fe700 1 -- 172.21.15.98:0/2256290572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 msgr2=0x5606493ab340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1b7fe700 1 --2- 172.21.15.98:0/2256290572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 0x5606493ab340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1b7fe700 1 -- 172.21.15.98:0/2256290572 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56064925e6f0 con 0x5606493ab880 2024-04-25T17:36:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1a7fc700 1 --2- 172.21.15.98:0/2256290572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 0x5606493ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1affd700 1 --2- 172.21.15.98:0/2256290572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a00b0 0x5606493a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.229+0000 7fec1b7fe700 1 --2- 172.21.15.98:0/2256290572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493ab880 0x5606493adc70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fec1400b0a0 tx=0x7fec14009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea1bfda7ff0a282 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.230+0000 7fec017fa700 1 -- 172.21.15.98:0/2256290572 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec1400bd30 con 0x5606493ab880 2024-04-25T17:36:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.230+0000 7fec017fa700 1 -- 172.21.15.98:0/2256290572 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec140074e0 con 0x5606493ab880 2024-04-25T17:36:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.230+0000 7fec017fa700 1 -- 172.21.15.98:0/2256290572 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14005280 con 0x5606493ab880 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.230+0000 7fec2bff3b80 1 -- 172.21.15.98:0/2256290572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493ab880 msgr2=0x5606493adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.230+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/2256290572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493ab880 0x5606493adc70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fec1400b0a0 tx=0x7fec14009520 comp rx=0 tx=0).stop 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.230+0000 7fec2bff3b80 1 -- 172.21.15.98:0/2256290572 shutdown_connections 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.230+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/2256290572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a00b0 0x5606493a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.230+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/2256290572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493ab880 0x5606493adc70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.230+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/2256290572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 0x5606493ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.230+0000 7fec2bff3b80 1 -- 172.21.15.98:0/2256290572 >> 172.21.15.98:0/2256290572 conn(0x5606492cd840 msgr2=0x56064939c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 1 -- 172.21.15.98:0/2256290572 shutdown_connections 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 1 -- 172.21.15.98:0/2256290572 wait complete. 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 1 -- start start 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 1 Processor -- start 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a09d0 0x5606493a81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493ab880 0x5606493a8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5606493ae310 con 0x5606493a00b0 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5606492ad560 con 0x5606493ab880 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5606493ae490 con 0x5606493a09d0 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec2bff3b80 10 client.0 _ll_register_callbacks cb 0x56064928c8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec1affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec1affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493aaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54004/0 (socket says 172.21.15.98:54004) 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec1affd700 1 -- 172.21.15.98:0/189072800 learned_addr learned my addr 172.21.15.98:0/189072800 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.231+0000 7fec1b7fe700 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493ab880 0x5606493a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7fec1a7fc700 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a09d0 0x5606493a81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7fec1b7fe700 1 -- 172.21.15.98:0/189072800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a09d0 msgr2=0x5606493a81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7fec1b7fe700 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a09d0 0x5606493a81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7fec1b7fe700 1 -- 172.21.15.98:0/189072800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 msgr2=0x5606493aaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:23.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7fec1b7fe700 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493aaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7fec1b7fe700 1 -- 172.21.15.98:0/189072800 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560649241880 con 0x5606493ab880 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7fec1b7fe700 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493ab880 0x5606493a8710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fec1400b070 tx=0x7fec140078c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=122ff6c74d25c07a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7febfeff5700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7febfeff5700 1 -- 172.21.15.98:0/189072800 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14003e10 con 0x5606493ab880 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7fec2bff3b80 1 -- 172.21.15.98:0/189072800 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec140074e0 con 0x5606493ab880 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7febfeff5700 1 -- 172.21.15.98:0/189072800 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fec140050b0 con 0x5606493ab880 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7febfeff5700 1 -- 172.21.15.98:0/189072800 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fec14007bb0 con 0x5606493ab880 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7fec2bff3b80 10 client.24944 fetch_fsmap learned FSMap version 26 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7fec2bff3b80 1 -- 172.21.15.98:0/189072800 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5606493b8b20 con 0x5606493ab880 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.232+0000 7febfeff5700 1 -- 172.21.15.98:0/189072800 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec140074e0 con 0x5606493ab880 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.233+0000 7febfeff5700 1 -- 172.21.15.98:0/189072800 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fec14011070 con 0x5606493ab880 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.233+0000 7fec2bff3b80 10 client.24944 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.233+0000 7fec2bff3b80 10 client.24944 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.233+0000 7fec2bff3b80 20 client.24944 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:23.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.233+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x5606493b9e70 0x5606493bc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.233+0000 7fec2bff3b80 4 client.24944 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.233+0000 7fec2bff3b80 1 -- 172.21.15.98:0/189072800 --> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560649301430 con 0x5606493b9e70 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.233+0000 7fec1affd700 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x5606493b9e70 0x5606493bc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.233+0000 7febfeff5700 10 client.24944 ms_handle_connect on v2:172.21.15.173:6836/1096095017 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.233+0000 7fec1affd700 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x5606493b9e70 0x5606493bc070 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=76fcb9cdb823e1d8 server_cookie=45f61ecaaae7d465 in_seq=0 out_seq=0 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7febfeff5700 1 -- 172.21.15.98:0/189072800 <== mds.2 v2:172.21.15.173:6836/1096095017 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560649301430 con 0x5606493b9e70 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7febfeff5700 10 client.24944 handle_command_reply: tid=0 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 client.24944 shutdown 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 -- 172.21.15.98:0/189072800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493ab880 msgr2=0x5606493a8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493ab880 0x5606493a8710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fec1400b070 tx=0x7fec140078c0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 -- 172.21.15.98:0/189072800 shutdown_connections 2024-04-25T17:36:23.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x5606493b9e70 0x5606493bc070 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493ab880 0x5606493a8710 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7febfeff5700 0 client.24944 ms_handle_reset on v2:172.21.15.173:6836/1096095017 2024-04-25T17:36:23.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493aaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/189072800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493a09d0 0x5606493a81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 -- 172.21.15.98:0/189072800 >> 172.21.15.98:0/189072800 conn(0x5606492cd840 msgr2=0x56064939c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:23.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 -- 172.21.15.98:0/189072800 shutdown_connections 2024-04-25T17:36:23.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 1 -- 172.21.15.98:0/189072800 wait complete. 2024-04-25T17:36:23.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.234+0000 7fec2bff3b80 20 client.24944 trim_cache size 0 max 16384 2024-04-25T17:36:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.256+0000 7fec2bff3b80 1 Processor -- start 2024-04-25T17:36:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec2bff3b80 1 -- start start 2024-04-25T17:36:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 0x5606493a3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493ab880 0x5606493acc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5606493ba3b0 con 0x5606493a00b0 2024-04-25T17:36:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5606493baa80 con 0x5606493a09d0 2024-04-25T17:36:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5606493bb150 con 0x5606493ab880 2024-04-25T17:36:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493a32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54012/0 (socket says 172.21.15.98:54012) 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1affd700 1 -- 172.21.15.98:0/3116227343 learned_addr learned my addr 172.21.15.98:0/3116227343 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1a7fc700 1 --2- 172.21.15.98:0/3116227343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 0x5606493a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1b7fe700 1 --2- 172.21.15.98:0/3116227343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493ab880 0x5606493acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1affd700 1 -- 172.21.15.98:0/3116227343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493ab880 msgr2=0x5606493acc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1affd700 1 --2- 172.21.15.98:0/3116227343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493ab880 0x5606493acc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1affd700 1 -- 172.21.15.98:0/3116227343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 msgr2=0x5606493a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1affd700 1 --2- 172.21.15.98:0/3116227343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 0x5606493a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1affd700 1 -- 172.21.15.98:0/3116227343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606493a1f20 con 0x5606493a00b0 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec1affd700 1 --2- 172.21.15.98:0/3116227343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493a32c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fec0c0991e0 tx=0x7fec0c09f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9d5fc1817e658c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.257+0000 7fec00ff9700 1 -- 172.21.15.98:0/3116227343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c09c810 con 0x5606493a00b0 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec00ff9700 1 -- 172.21.15.98:0/3116227343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec0c0b9070 con 0x5606493a00b0 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec00ff9700 1 -- 172.21.15.98:0/3116227343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c098e80 con 0x5606493a00b0 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec2bff3b80 1 -- 172.21.15.98:0/3116227343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 msgr2=0x5606493a32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/3116227343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493a32c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fec0c0991e0 tx=0x7fec0c09f040 comp rx=0 tx=0).stop 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec2bff3b80 1 -- 172.21.15.98:0/3116227343 shutdown_connections 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/3116227343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493ab880 0x5606493acc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/3116227343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a00b0 0x5606493a32c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/3116227343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a09d0 0x5606493a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec2bff3b80 1 -- 172.21.15.98:0/3116227343 >> 172.21.15.98:0/3116227343 conn(0x5606492cd840 msgr2=0x56064939c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec2bff3b80 1 -- 172.21.15.98:0/3116227343 shutdown_connections 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec2bff3b80 1 -- 172.21.15.98:0/3116227343 wait complete. 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.258+0000 7fec2bff3b80 1 -- start start 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec2bff3b80 1 Processor -- start 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a00b0 0x5606493a5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a09d0 0x5606493a5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec2bff3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493ab880 0x5606493c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5606493bbef0 con 0x5606493a09d0 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5606493bb820 con 0x5606493a00b0 2024-04-25T17:36:23.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec2bff3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5606493bc5c0 con 0x5606493ab880 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec2bff3b80 10 client.0 _ll_register_callbacks cb 0x56064928c8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a09d0 0x5606493a5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a00b0 0x5606493a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1b7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493ab880 0x5606493c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1affd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a00b0 0x5606493a5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48972/0 (socket says 172.21.15.98:48972) 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1affd700 1 -- 172.21.15.98:0/14721840 learned_addr learned my addr 172.21.15.98:0/14721840 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a09d0 0x5606493a5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54028/0 (socket says 172.21.15.98:54028) 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1a7fc700 1 -- 172.21.15.98:0/14721840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493ab880 msgr2=0x5606493c1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1a7fc700 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493ab880 0x5606493c1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1a7fc700 1 -- 172.21.15.98:0/14721840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a00b0 msgr2=0x5606493a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:23.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1a7fc700 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a00b0 0x5606493a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1a7fc700 1 -- 172.21.15.98:0/14721840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606493c2090 con 0x5606493a09d0 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1affd700 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a00b0 0x5606493a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7fec1a7fc700 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a09d0 0x5606493a5a50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x5606493bceb0 tx=0x7fec100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9f6fa62638d9ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.259+0000 7febfeff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7febfeff5700 1 -- 172.21.15.98:0/14721840 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec10004570 con 0x5606493a09d0 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7febfeff5700 1 -- 172.21.15.98:0/14721840 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fec10012450 con 0x5606493a09d0 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7fec2bff3b80 1 -- 172.21.15.98:0/14721840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec0c0b9070 con 0x5606493a09d0 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7febfeff5700 1 -- 172.21.15.98:0/14721840 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fec10012650 con 0x5606493a09d0 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7fec2bff3b80 10 client.15321 fetch_fsmap learned FSMap version 26 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7fec2bff3b80 1 -- 172.21.15.98:0/14721840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560649422530 con 0x5606493a09d0 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7febfeff5700 1 -- 172.21.15.98:0/14721840 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec0c0b9070 con 0x5606493a09d0 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7febfeff5700 1 -- 172.21.15.98:0/14721840 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fec10029020 con 0x5606493a09d0 2024-04-25T17:36:23.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7fec2bff3b80 10 client.15321 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7fec2bff3b80 10 client.15321 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7fec2bff3b80 20 client.15321 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x5606493b9e70 0x560649425e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7fec2bff3b80 4 client.15321 mds_command: new command op to 24440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7fec2bff3b80 1 -- 172.21.15.98:0/14721840 --> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] -- command(tid 0: {"prefix": "status"}) -- 0x5606492fb4d0 con 0x5606493b9e70 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.260+0000 7fec1affd700 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x5606493b9e70 0x560649425e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec1affd700 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x5606493b9e70 0x560649425e60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d425cf22abfb7bc server_cookie=d35dfcc53187493e in_seq=0 out_seq=0 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7febfeff5700 10 client.15321 ms_handle_connect on v2:172.21.15.173:6836/1096095017 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7febfeff5700 1 -- 172.21.15.98:0/14721840 <== mds.2 v2:172.21.15.173:6836/1096095017 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5606492fb4d0 con 0x5606493b9e70 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7febfeff5700 10 client.15321 handle_command_reply: tid=0 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 client.15321 shutdown 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 -- 172.21.15.98:0/14721840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a09d0 msgr2=0x5606493a5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a09d0 0x5606493a5a50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x5606493bceb0 tx=0x7fec100047c0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 -- 172.21.15.98:0/14721840 shutdown_connections 2024-04-25T17:36:23.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.173:6836/1096095017,v1:172.21.15.173:6837/1096095017] conn(0x5606493b9e70 0x560649425e60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7febfeff5700 0 client.15321 ms_handle_reset on v2:172.21.15.173:6836/1096095017 2024-04-25T17:36:23.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5606493ab880 0x5606493c1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5606493a00b0 0x5606493a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 --2- 172.21.15.98:0/14721840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5606493a09d0 0x5606493a5a50 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 -- 172.21.15.98:0/14721840 >> 172.21.15.98:0/14721840 conn(0x5606492cd840 msgr2=0x5606493a3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:23.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 -- 172.21.15.98:0/14721840 shutdown_connections 2024-04-25T17:36:23.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 1 -- 172.21.15.98:0/14721840 wait complete. 2024-04-25T17:36:23.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.261+0000 7fec2bff3b80 20 client.15321 trim_cache size 0 max 16384 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24440, 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 210.13872886999999, 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 26, 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 92, 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T17:36:23.269 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 258.37412461399998 2024-04-25T17:36:23.270 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:36:23.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 -- 172.21.15.98:0/1884089981 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fec040a3ed0 msgr2=0x7fec040a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fec040a3ed0 0x7fec040a6390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec1c079080 tx=0x7fec10009260 comp rx=0 tx=0).stop 2024-04-25T17:36:23.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 -- 172.21.15.98:0/1884089981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec1c101940 msgr2=0x7fec1c199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:23.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec1c101940 0x7fec1c199170 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c9b0 tx=0x7fec0c00ccc0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 -- 172.21.15.98:0/1884089981 shutdown_connections 2024-04-25T17:36:23.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fec040a3ed0 0x7fec040a6390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec1c107990 0x7fec1c19d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec1c101940 0x7fec1c199170 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 --2- 172.21.15.98:0/1884089981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec1c102300 0x7fec1c1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:23.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 -- 172.21.15.98:0/1884089981 >> 172.21.15.98:0/1884089981 conn(0x7fec1c0788f0 msgr2=0x7fec1c06ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:23.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 -- 172.21.15.98:0/1884089981 shutdown_connections 2024-04-25T17:36:23.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:23.262+0000 7fec2176a700 1 -- 172.21.15.98:0/1884089981 wait complete. 2024-04-25T17:36:23.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-25T17:36:24.145 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:24.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:24 smithi098 ceph-mon[29192]: pgmap v17: 97 pgs: 97 active+clean; 1.8 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-04-25T17:36:24.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:24 smithi173 ceph-mon[32856]: pgmap v17: 97 pgs: 97 active+clean; 1.8 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-04-25T17:36:24.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:24 smithi203 ceph-mon[32834]: pgmap v17: 97 pgs: 97 active+clean; 1.8 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 54 KiB/s wr, 13 op/s 2024-04-25T17:36:25.027 INFO:teuthology.orchestra.run.smithi098.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-04-25T17:36:25.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-25T17:36:25.068 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/: 2024-04-25T17:36:25.068 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.068 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 . 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 .. 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 data 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data: 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:total 35 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 . 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 .. 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 base 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 25 17:36 global 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 log 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_commit_ts 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_dynshmem 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 25 17:36 pg_hba.conf 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 25 17:36 pg_ident.conf 2024-04-25T17:36:25.069 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 pg_logical 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 pg_multixact 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_notify 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_replslot 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_serial 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_snapshots 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_stat 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_stat_tmp 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 pg_subtrans 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_tblspc 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_twophase 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 PG_VERSION 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 pg_wal 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 pg_xact 2024-04-25T17:36:25.070 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 25 17:36 postgresql.auto.conf 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 25 17:36 postgresql.conf 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/base: 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 . 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 17:36 1 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 17:36 13493 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 17:36 13494 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:total 7777 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 17:36 . 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 .. 2024-04-25T17:36:25.071 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 112 2024-04-25T17:36:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 113 2024-04-25T17:36:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 1247 2024-04-25T17:36:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1247_fsm 2024-04-25T17:36:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1247_vm 2024-04-25T17:36:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 17:36 1249 2024-04-25T17:36:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1249_fsm 2024-04-25T17:36:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1249_vm 2024-04-25T17:36:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 17:36 1255 2024-04-25T17:36:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1255_fsm 2024-04-25T17:36:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1255_vm 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 17:36 1259 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1259_fsm 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1259_vm 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 17:36 13344 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13344_fsm 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13344_vm 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13346 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13348 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13349_fsm 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349_vm 2024-04-25T17:36:25.073 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13351 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13353 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13354_fsm 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354_vm 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13356 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13358 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13359_fsm 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359_vm 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13361 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13363 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1417 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1418 2024-04-25T17:36:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 174 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 175 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2187 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2224 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2328 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2336 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2337 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2579 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2600 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2600_fsm 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2600_vm 2024-04-25T17:36:25.075 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601 2024-04-25T17:36:25.076 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2601_fsm 2024-04-25T17:36:25.076 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601_vm 2024-04-25T17:36:25.076 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2602 2024-04-25T17:36:25.076 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2602_fsm 2024-04-25T17:36:25.076 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2602_vm 2024-04-25T17:36:25.076 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2603 2024-04-25T17:36:25.076 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2603_fsm 2024-04-25T17:36:25.076 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2603_vm 2024-04-25T17:36:25.076 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2604 2024-04-25T17:36:25.076 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2605 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2605_fsm 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2605_vm 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2606_fsm 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606_vm 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2607 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2607_fsm 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2607_vm 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 17:36 2608 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2608_fsm 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2608_vm 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 17:36 2609 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2609_fsm 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2609_vm 2024-04-25T17:36:25.077 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2610 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2610_fsm 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2610_vm 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2611 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2612_fsm 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612_vm 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2613 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2615_fsm 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615_vm 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2616 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2616_fsm 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2616_vm 2024-04-25T17:36:25.078 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 17:36 2617 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2617_fsm 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2617_vm 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 17:36 2618 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2618_fsm 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2618_vm 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 17:36 2619 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2619_fsm 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2619_vm 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2620 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2650 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2651 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2652 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2653 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2654 2024-04-25T17:36:25.079 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2655 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2656 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2657 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 17:36 2658 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 2659 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2660 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2661 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2662 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2663 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2664 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2665 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2666 2024-04-25T17:36:25.080 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2667 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2668 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2669 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2670 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 17:36 2673 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 17:36 2674 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 17:36 2675 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2678 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2679 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2680 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2681 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2682 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2683 2024-04-25T17:36:25.081 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2684 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2685 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2686 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2687 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2688 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2689 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 17:36 2690 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 17:36 2691 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2692 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2693 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2696 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2699 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2701 2024-04-25T17:36:25.082 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2702 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2703 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2704 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2753 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2753_fsm 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2753_vm 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2754 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2755 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2756 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2757 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2830 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2831 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2832 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2833 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2834 2024-04-25T17:36:25.083 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2835 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2836 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2837 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 17:36 2838 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2838_fsm 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2838_vm 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2839 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840_fsm 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2840_vm 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2841 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2995 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2996 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3079_fsm 2024-04-25T17:36:25.084 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079_vm 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3080 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3081 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3085 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3118 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3119 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 17:36 3164 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3256 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3257 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3258 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3350 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3351 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3379 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3380 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3381 2024-04-25T17:36:25.085 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3394 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3394_fsm 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3394_vm 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3395 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3429 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3430 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3431 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3433 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3439 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3440 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3455 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 17:36 3456 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3456_fsm 2024-04-25T17:36:25.086 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3456_vm 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3466 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3467 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3468 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3501 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3502 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3503 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3534 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3541_fsm 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541_vm 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3542 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3574 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3575 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3576 2024-04-25T17:36:25.087 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3596 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3597 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3598 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3599 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3600_fsm 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600_vm 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3601_fsm 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601_vm 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3602_fsm 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602_vm 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603_fsm 2024-04-25T17:36:25.088 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3603_vm 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3604 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3605 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3606 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3607 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3608 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 3609 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3712 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3764_fsm 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764_vm 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3766 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3767 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3997 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4143 2024-04-25T17:36:25.089 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4144 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4145 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4146 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4147 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4148 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4149 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4150 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4151 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4152 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4153 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4154 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4155 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4156 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4157 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4158 2024-04-25T17:36:25.090 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4159 2024-04-25T17:36:25.091 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4160 2024-04-25T17:36:25.091 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4161 2024-04-25T17:36:25.091 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4162 2024-04-25T17:36:25.091 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4163 2024-04-25T17:36:25.091 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4164 2024-04-25T17:36:25.091 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4165 2024-04-25T17:36:25.094 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4166 2024-04-25T17:36:25.094 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4167 2024-04-25T17:36:25.094 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4168 2024-04-25T17:36:25.094 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4169 2024-04-25T17:36:25.094 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4170 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4171 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4172 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4173 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4174 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 5002 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 548 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 549 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6102 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6104 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6106 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6110 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6111 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6112 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6113 2024-04-25T17:36:25.095 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6117 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 826 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 827 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 828 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 17:36 pg_filenode.map 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 PG_VERSION 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:total 7777 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 17:36 . 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 .. 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 112 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 113 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 1247 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1247_fsm 2024-04-25T17:36:25.096 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1247_vm 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 17:36 1249 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1249_fsm 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1249_vm 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 17:36 1255 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1255_fsm 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1255_vm 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 17:36 1259 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1259_fsm 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1259_vm 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 17:36 13344 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13344_fsm 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13344_vm 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13346 2024-04-25T17:36:25.097 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13348 2024-04-25T17:36:25.098 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349 2024-04-25T17:36:25.098 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13349_fsm 2024-04-25T17:36:25.098 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349_vm 2024-04-25T17:36:25.098 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13351 2024-04-25T17:36:25.098 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13353 2024-04-25T17:36:25.098 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354 2024-04-25T17:36:25.098 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13354_fsm 2024-04-25T17:36:25.098 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354_vm 2024-04-25T17:36:25.098 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13356 2024-04-25T17:36:25.099 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13358 2024-04-25T17:36:25.099 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359 2024-04-25T17:36:25.099 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13359_fsm 2024-04-25T17:36:25.099 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359_vm 2024-04-25T17:36:25.099 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13361 2024-04-25T17:36:25.099 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13363 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1417 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1418 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 174 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 175 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2187 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2224 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2328 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2336 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2337 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2579 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2600 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2600_fsm 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2600_vm 2024-04-25T17:36:25.100 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2601_fsm 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601_vm 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2602 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2602_fsm 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2602_vm 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2603 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2603_fsm 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2603_vm 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2604 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2605 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2605_fsm 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2605_vm 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2606_fsm 2024-04-25T17:36:25.101 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606_vm 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2607 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2607_fsm 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2607_vm 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 17:36 2608 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2608_fsm 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2608_vm 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 17:36 2609 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2609_fsm 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2609_vm 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2610 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2610_fsm 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2610_vm 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2611 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612 2024-04-25T17:36:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2612_fsm 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612_vm 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2613 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2615_fsm 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615_vm 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2616 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2616_fsm 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2616_vm 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 17:36 2617 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2617_fsm 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2617_vm 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 17:36 2618 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2618_fsm 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2618_vm 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 17:36 2619 2024-04-25T17:36:25.103 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2619_fsm 2024-04-25T17:36:25.104 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2619_vm 2024-04-25T17:36:25.104 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2620 2024-04-25T17:36:25.104 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2650 2024-04-25T17:36:25.104 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2651 2024-04-25T17:36:25.104 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2652 2024-04-25T17:36:25.104 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2653 2024-04-25T17:36:25.104 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2654 2024-04-25T17:36:25.104 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2655 2024-04-25T17:36:25.104 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2656 2024-04-25T17:36:25.105 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2657 2024-04-25T17:36:25.105 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 17:36 2658 2024-04-25T17:36:25.105 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 2659 2024-04-25T17:36:25.105 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2660 2024-04-25T17:36:25.105 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2661 2024-04-25T17:36:25.105 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2662 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2663 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2664 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2665 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2666 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2667 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2668 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2669 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2670 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 17:36 2673 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 17:36 2674 2024-04-25T17:36:25.106 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 17:36 2675 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2678 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2679 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2680 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2681 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2682 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2683 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2684 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2685 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2686 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2687 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2688 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2689 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 17:36 2690 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 17:36 2691 2024-04-25T17:36:25.107 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2692 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2693 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2696 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2699 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2701 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2702 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2703 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2704 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2753 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2753_fsm 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2753_vm 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2754 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2755 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2756 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2757 2024-04-25T17:36:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2830 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2831 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2832 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2833 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2834 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2835 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2836 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2837 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 17:36 2838 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2838_fsm 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2838_vm 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2839 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840_fsm 2024-04-25T17:36:25.109 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2840_vm 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2841 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2995 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2996 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3079_fsm 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079_vm 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3080 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3081 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3085 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3118 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3119 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 17:36 3164 2024-04-25T17:36:25.110 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3256 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3257 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3258 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3350 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3351 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3379 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3380 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3381 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3394 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3394_fsm 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3394_vm 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3395 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3429 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3430 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3431 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3433 2024-04-25T17:36:25.111 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3439 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3440 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3455 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 17:36 3456 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3456_fsm 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3456_vm 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3466 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3467 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3468 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3501 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3502 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3503 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3534 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3541_fsm 2024-04-25T17:36:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541_vm 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3542 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3574 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3575 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3576 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3596 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3597 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3598 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3599 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3600_fsm 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600_vm 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3601_fsm 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601_vm 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602 2024-04-25T17:36:25.113 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3602_fsm 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602_vm 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603_fsm 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3603_vm 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3604 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3605 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3606 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3607 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3608 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 3609 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3712 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3764_fsm 2024-04-25T17:36:25.114 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764_vm 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3766 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3767 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3997 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4143 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4144 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4145 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4146 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4147 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4148 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4149 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4150 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4151 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4152 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4153 2024-04-25T17:36:25.115 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4154 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4155 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4156 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4157 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4158 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4159 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4160 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4161 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4162 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4163 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4164 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4165 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4166 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4167 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4168 2024-04-25T17:36:25.116 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4169 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4170 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4171 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4172 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4173 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4174 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 5002 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 548 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 549 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6102 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6104 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6106 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6110 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6111 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6112 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6113 2024-04-25T17:36:25.117 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6117 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 826 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 827 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 828 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 17:36 pg_filenode.map 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 PG_VERSION 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:total 7777 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 17:36 . 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 .. 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 112 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 113 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 1247 2024-04-25T17:36:25.118 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1247_fsm 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1247_vm 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 17:36 1249 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1249_fsm 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1249_vm 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 17:36 1255 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1255_fsm 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1255_vm 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 17:36 1259 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1259_fsm 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1259_vm 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 17:36 13344 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13344_fsm 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13344_vm 2024-04-25T17:36:25.119 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13346 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13348 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13349_fsm 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349_vm 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13351 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13353 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13354_fsm 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354_vm 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13356 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13358 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13359_fsm 2024-04-25T17:36:25.120 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359_vm 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13361 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13363 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1417 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1418 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 174 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 175 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2187 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2224 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2328 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2336 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2337 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2579 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2600 2024-04-25T17:36:25.121 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2600_fsm 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2600_vm 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2601_fsm 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601_vm 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2602 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2602_fsm 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2602_vm 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2603 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2603_fsm 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2603_vm 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2604 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2605 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2605_fsm 2024-04-25T17:36:25.122 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2605_vm 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2606_fsm 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606_vm 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2607 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2607_fsm 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2607_vm 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 17:36 2608 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2608_fsm 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2608_vm 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 17:36 2609 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2609_fsm 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2609_vm 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2610 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2610_fsm 2024-04-25T17:36:25.123 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2610_vm 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2611 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2612_fsm 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612_vm 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2613 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2615_fsm 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615_vm 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2616 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2616_fsm 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2616_vm 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 17:36 2617 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2617_fsm 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2617_vm 2024-04-25T17:36:25.124 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 17:36 2618 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2618_fsm 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2618_vm 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 17:36 2619 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2619_fsm 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2619_vm 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2620 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2650 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2651 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2652 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2653 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2654 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2655 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2656 2024-04-25T17:36:25.125 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2657 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 17:36 2658 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 2659 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2660 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2661 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2662 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2663 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2664 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2665 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2666 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2667 2024-04-25T17:36:25.126 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2668 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2669 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2670 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 17:36 2673 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 17:36 2674 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 17:36 2675 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2678 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2679 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2680 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2681 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2682 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2683 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2684 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2685 2024-04-25T17:36:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2686 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2687 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2688 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2689 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 17:36 2690 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 17:36 2691 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2692 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2693 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2696 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2699 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2701 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2702 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2703 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2704 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2753 2024-04-25T17:36:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2753_fsm 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2753_vm 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2754 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2755 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2756 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2757 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2830 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2831 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2832 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2833 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2834 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2835 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2836 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2837 2024-04-25T17:36:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 17:36 2838 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2838_fsm 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2838_vm 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2839 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840_fsm 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2840_vm 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2841 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2995 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2996 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3079_fsm 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079_vm 2024-04-25T17:36:25.130 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3080 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3081 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3085 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3118 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3119 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 17:36 3164 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3256 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3257 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3258 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3350 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3351 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3379 2024-04-25T17:36:25.131 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3380 2024-04-25T17:36:25.132 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3381 2024-04-25T17:36:25.132 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3394 2024-04-25T17:36:25.132 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3394_fsm 2024-04-25T17:36:25.132 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3394_vm 2024-04-25T17:36:25.132 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3395 2024-04-25T17:36:25.132 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3429 2024-04-25T17:36:25.133 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3430 2024-04-25T17:36:25.133 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3431 2024-04-25T17:36:25.133 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3433 2024-04-25T17:36:25.133 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3439 2024-04-25T17:36:25.133 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3440 2024-04-25T17:36:25.133 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3455 2024-04-25T17:36:25.133 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 17:36 3456 2024-04-25T17:36:25.134 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3456_fsm 2024-04-25T17:36:25.134 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3456_vm 2024-04-25T17:36:25.134 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3466 2024-04-25T17:36:25.134 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3467 2024-04-25T17:36:25.134 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3468 2024-04-25T17:36:25.135 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3501 2024-04-25T17:36:25.135 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3502 2024-04-25T17:36:25.135 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3503 2024-04-25T17:36:25.135 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3534 2024-04-25T17:36:25.135 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541 2024-04-25T17:36:25.135 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3541_fsm 2024-04-25T17:36:25.135 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541_vm 2024-04-25T17:36:25.136 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3542 2024-04-25T17:36:25.136 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3574 2024-04-25T17:36:25.136 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3575 2024-04-25T17:36:25.136 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3576 2024-04-25T17:36:25.136 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3596 2024-04-25T17:36:25.136 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3597 2024-04-25T17:36:25.136 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3598 2024-04-25T17:36:25.137 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3599 2024-04-25T17:36:25.137 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600 2024-04-25T17:36:25.137 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3600_fsm 2024-04-25T17:36:25.137 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600_vm 2024-04-25T17:36:25.137 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601 2024-04-25T17:36:25.137 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3601_fsm 2024-04-25T17:36:25.138 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601_vm 2024-04-25T17:36:25.138 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602 2024-04-25T17:36:25.138 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3602_fsm 2024-04-25T17:36:25.138 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602_vm 2024-04-25T17:36:25.138 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603 2024-04-25T17:36:25.141 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603_fsm 2024-04-25T17:36:25.141 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3603_vm 2024-04-25T17:36:25.142 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3604 2024-04-25T17:36:25.142 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3605 2024-04-25T17:36:25.142 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3606 2024-04-25T17:36:25.142 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3607 2024-04-25T17:36:25.142 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3608 2024-04-25T17:36:25.142 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 3609 2024-04-25T17:36:25.142 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3712 2024-04-25T17:36:25.142 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764 2024-04-25T17:36:25.142 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3764_fsm 2024-04-25T17:36:25.142 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764_vm 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3766 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3767 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3997 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4143 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4144 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4145 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4146 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4147 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4148 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4149 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4150 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4151 2024-04-25T17:36:25.143 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4152 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4153 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4154 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4155 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4156 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4157 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4158 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4159 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4160 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4161 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4162 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4163 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4164 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4165 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4166 2024-04-25T17:36:25.144 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4167 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4168 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4169 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4170 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4171 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4172 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4173 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4174 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 5002 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 548 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 549 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6102 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6104 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6106 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6110 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6111 2024-04-25T17:36:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6112 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6113 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6117 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 826 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 827 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 828 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 17:36 pg_filenode.map 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 PG_VERSION 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/global: 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:total 537 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 25 17:36 . 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1213 2024-04-25T17:36:25.146 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1213_fsm 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1213_vm 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1214 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1214_fsm 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1214_vm 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 1232 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 1233 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1260 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1260_fsm 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1260_vm 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1261 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1261_fsm 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1261_vm 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1262 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1262_fsm 2024-04-25T17:36:25.147 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1262_vm 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2396 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2396_fsm 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2396_vm 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2397 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2671 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2672 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2676 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2677 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2694 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2695 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2697 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2698 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2846 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2847 2024-04-25T17:36:25.148 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2964 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2965 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2966 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2967 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3592 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3593 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4060 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4061 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4175 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4176 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4177 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4178 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4181 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4182 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4183 2024-04-25T17:36:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4184 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4185 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4186 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6000 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6001 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6002 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6100 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6114 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6115 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 pg_control 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 17:36 pg_filenode.map 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/log: 2024-04-25T17:36:25.150 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.151 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:total 1 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 . 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 mappings 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 25 17:36 replorigin_checkpoint 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 snapshots 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 .. 2024-04-25T17:36:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 .. 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 . 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 members 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 offsets 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:total 8 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 . 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 .. 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 0000 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:total 8 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 . 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 .. 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 0000 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.155 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-25T17:36:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-25T17:36:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:total 8 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 . 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 0000 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:total 16384 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 . 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:36 000000010000000000000001 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 archive_status 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:36:25.158 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 .. 2024-04-25T17:36:25.159 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:25.159 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-25T17:36:25.159 INFO:teuthology.orchestra.run.smithi098.stdout:total 8 2024-04-25T17:36:25.159 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 . 2024-04-25T17:36:25.159 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 17:36 .. 2024-04-25T17:36:25.159 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 0000 2024-04-25T17:36:25.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-04-25T17:36:25.301 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-04-25T17:36:25.343 INFO:teuthology.orchestra.run.smithi098.stderr:dropping old tables... 2024-04-25T17:36:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-04-25T17:36:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-04-25T17:36:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-04-25T17:36:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-04-25T17:36:25.344 INFO:teuthology.orchestra.run.smithi098.stderr:creating tables... 2024-04-25T17:36:25.355 INFO:teuthology.orchestra.run.smithi098.stderr:generating data (client-side)... 2024-04-25T17:36:25.442 INFO:teuthology.orchestra.run.smithi098.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.07 s, remaining 2.17 s) 2024-04-25T17:36:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.23 s, remaining 3.50 s) 2024-04-25T17:36:25.697 INFO:teuthology.orchestra.run.smithi098.stderr:300000 of 3200000 tuples (9%) done (elapsed 0.32 s, remaining 3.14 s) 2024-04-25T17:36:25.834 INFO:teuthology.orchestra.run.smithi098.stderr:400000 of 3200000 tuples (12%) done (elapsed 0.46 s, remaining 3.24 s) 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.885+0000 7fa853786700 1 -- 172.21.15.98:0/1180563604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84c1022f0 msgr2=0x7fa84c10e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.885+0000 7fa853786700 1 --2- 172.21.15.98:0/1180563604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84c1022f0 0x7fa84c10e7e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009990 tx=0x7fa83c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.885+0000 7fa853786700 1 -- 172.21.15.98:0/1180563604 shutdown_connections 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.885+0000 7fa853786700 1 --2- 172.21.15.98:0/1180563604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84c1022f0 0x7fa84c10e7e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.885+0000 7fa853786700 1 --2- 172.21.15.98:0/1180563604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa84c107980 0x7fa84c107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.885+0000 7fa853786700 1 --2- 172.21.15.98:0/1180563604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c101930 0x7fa84c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.885+0000 7fa853786700 1 -- 172.21.15.98:0/1180563604 >> 172.21.15.98:0/1180563604 conn(0x7fa84c0788b0 msgr2=0x7fa84c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.886+0000 7fa853786700 1 -- 172.21.15.98:0/1180563604 shutdown_connections 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.886+0000 7fa853786700 1 -- 172.21.15.98:0/1180563604 wait complete. 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.886+0000 7fa853786700 1 Processor -- start 2024-04-25T17:36:25.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.886+0000 7fa853786700 1 -- start start 2024-04-25T17:36:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa853786700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa84c101930 0x7fa84c199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa853786700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84c1022f0 0x7fa84c19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa853786700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c107980 0x7fa84c1a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa853786700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c111ae0 con 0x7fa84c1022f0 2024-04-25T17:36:25.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa853786700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c111960 con 0x7fa84c101930 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa853786700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c111c60 con 0x7fa84c107980 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa851522700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa84c101930 0x7fa84c199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa850d21700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84c1022f0 0x7fa84c19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa851d23700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c107980 0x7fa84c1a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa851522700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa84c101930 0x7fa84c199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:37670/0 (socket says 172.21.15.98:37670) 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa850d21700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84c1022f0 0x7fa84c19a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55640/0 (socket says 172.21.15.98:55640) 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa851522700 1 -- 172.21.15.98:0/3900037326 learned_addr learned my addr 172.21.15.98:0/3900037326 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa851d23700 1 -- 172.21.15.98:0/3900037326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa84c101930 msgr2=0x7fa84c199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa851d23700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa84c101930 0x7fa84c199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa851d23700 1 -- 172.21.15.98:0/3900037326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84c1022f0 msgr2=0x7fa84c19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa851d23700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84c1022f0 0x7fa84c19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa851d23700 1 -- 172.21.15.98:0/3900037326 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84c1a5460 con 0x7fa84c107980 2024-04-25T17:36:25.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.887+0000 7fa850d21700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84c1022f0 0x7fa84c19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.888+0000 7fa851d23700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c107980 0x7fa84c1a4f20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0098e0 tx=0x7fa83c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.888+0000 7fa84a7fc700 1 -- 172.21.15.98:0/3900037326 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c027070 con 0x7fa84c107980 2024-04-25T17:36:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.888+0000 7fa84a7fc700 1 -- 172.21.15.98:0/3900037326 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c022470 con 0x7fa84c107980 2024-04-25T17:36:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.888+0000 7fa84a7fc700 1 -- 172.21.15.98:0/3900037326 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c017650 con 0x7fa84c107980 2024-04-25T17:36:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.888+0000 7fa853786700 1 -- 172.21.15.98:0/3900037326 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa84c197c90 con 0x7fa84c107980 2024-04-25T17:36:25.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.888+0000 7fa853786700 1 -- 172.21.15.98:0/3900037326 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa84c198120 con 0x7fa84c107980 2024-04-25T17:36:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.890+0000 7fa84a7fc700 1 -- 172.21.15.98:0/3900037326 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fa83c02a070 con 0x7fa84c107980 2024-04-25T17:36:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.891+0000 7fa84a7fc700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8340a3f50 0x7fa8340a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.891+0000 7fa851522700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8340a3f50 0x7fa8340a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.891+0000 7fa84a7fc700 1 -- 172.21.15.98:0/3900037326 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fa83c0d1760 con 0x7fa84c107980 2024-04-25T17:36:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.891+0000 7fa851522700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8340a3f50 0x7fa8340a6410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa844009c10 tx=0x7fa84400b040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.892+0000 7fa85e00fb80 1 Processor -- start 2024-04-25T17:36:25.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.892+0000 7fa85e00fb80 1 -- start start 2024-04-25T17:36:25.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.892+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b0b0 0x55af6568b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.892+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b9d0 0x55af65696340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.892+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 0x55af65698c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.892+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af655dec40 con 0x55af6568b9d0 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.892+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55af655858e0 con 0x55af65696880 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.892+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55af655dac00 con 0x55af6568b0b0 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa850d21700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b9d0 0x55af65696340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa850d21700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b9d0 0x55af65696340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55646/0 (socket says 172.21.15.98:55646) 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa850d21700 1 -- 172.21.15.98:0/2861965308 learned_addr learned my addr 172.21.15.98:0/2861965308 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa851d23700 1 --2- 172.21.15.98:0/2861965308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 0x55af65698c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa850d21700 1 -- 172.21.15.98:0/2861965308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b0b0 msgr2=0x55af6568b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa850d21700 1 --2- 172.21.15.98:0/2861965308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b0b0 0x55af6568b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa850d21700 1 -- 172.21.15.98:0/2861965308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 msgr2=0x55af65698c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa850d21700 1 --2- 172.21.15.98:0/2861965308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 0x55af65698c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa850d21700 1 -- 172.21.15.98:0/2861965308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af655496f0 con 0x55af6568b9d0 2024-04-25T17:36:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa851d23700 1 --2- 172.21.15.98:0/2861965308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 0x55af65698c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:36:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.893+0000 7fa850d21700 1 --2- 172.21.15.98:0/2861965308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b9d0 0x55af65696340 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa840009780 tx=0x7fa840009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d081175054425522 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.894+0000 7fa8337fe700 1 -- 172.21.15.98:0/2861965308 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84001e030 con 0x55af6568b9d0 2024-04-25T17:36:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.894+0000 7fa8337fe700 1 -- 172.21.15.98:0/2861965308 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa840004ce0 con 0x55af6568b9d0 2024-04-25T17:36:25.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.894+0000 7fa8337fe700 1 -- 172.21.15.98:0/2861965308 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84001a720 con 0x55af6568b9d0 2024-04-25T17:36:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.895+0000 7fa85e00fb80 1 -- 172.21.15.98:0/2861965308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b9d0 msgr2=0x55af65696340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.895+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/2861965308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b9d0 0x55af65696340 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa840009780 tx=0x7fa840009260 comp rx=0 tx=0).stop 2024-04-25T17:36:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.895+0000 7fa85e00fb80 1 -- 172.21.15.98:0/2861965308 shutdown_connections 2024-04-25T17:36:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.895+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/2861965308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b0b0 0x55af6568b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.895+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/2861965308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 0x55af65698c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.895+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/2861965308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b9d0 0x55af65696340 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.895+0000 7fa85e00fb80 1 -- 172.21.15.98:0/2861965308 >> 172.21.15.98:0/2861965308 conn(0x55af655b8840 msgr2=0x55af65687580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.895+0000 7fa85e00fb80 1 -- 172.21.15.98:0/2861965308 shutdown_connections 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.895+0000 7fa85e00fb80 1 -- 172.21.15.98:0/2861965308 wait complete. 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.895+0000 7fa85e00fb80 1 -- start start 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa85e00fb80 1 Processor -- start 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b0b0 0x55af65696050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b9d0 0x55af65693350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 0x55af65693890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af65699370 con 0x55af6568b0b0 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55af65598560 con 0x55af65696880 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55af656994f0 con 0x55af6568b9d0 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa85e00fb80 10 client.0 _ll_register_callbacks cb 0x55af655778d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa851522700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b0b0 0x55af65696050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa851522700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b0b0 0x55af65696050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55662/0 (socket says 172.21.15.98:55662) 2024-04-25T17:36:25.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.896+0000 7fa851522700 1 -- 172.21.15.98:0/753098817 learned_addr learned my addr 172.21.15.98:0/753098817 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa851d23700 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 0x55af65693890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa850d21700 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b9d0 0x55af65693350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa851d23700 1 -- 172.21.15.98:0/753098817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b9d0 msgr2=0x55af65693350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa851d23700 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b9d0 0x55af65693350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa851d23700 1 -- 172.21.15.98:0/753098817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b0b0 msgr2=0x55af65696050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa851d23700 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b0b0 0x55af65696050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa851d23700 1 -- 172.21.15.98:0/753098817 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af6552c880 con 0x55af65696880 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa851522700 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b0b0 0x55af65696050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa851d23700 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 0x55af65693890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0ced00 tx=0x7fa83c09da70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46760bea46397e83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa830ff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa830ff9700 1 -- 172.21.15.98:0/753098817 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c09cef0 con 0x55af65696880 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa830ff9700 1 -- 172.21.15.98:0/753098817 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fa83c0a2a50 con 0x55af65696880 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.897+0000 7fa85e00fb80 1 -- 172.21.15.98:0/753098817 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa840004ce0 con 0x55af65696880 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.898+0000 7fa830ff9700 1 -- 172.21.15.98:0/753098817 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa83c09d550 con 0x55af65696880 2024-04-25T17:36:25.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.898+0000 7fa85e00fb80 10 client.24962 fetch_fsmap learned FSMap version 26 2024-04-25T17:36:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.898+0000 7fa85e00fb80 1 -- 172.21.15.98:0/753098817 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55af656a3c70 con 0x55af65696880 2024-04-25T17:36:25.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.898+0000 7fa830ff9700 1 -- 172.21.15.98:0/753098817 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c09d0d0 con 0x55af65696880 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.898+0000 7fa830ff9700 1 -- 172.21.15.98:0/753098817 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa83c0a2c50 con 0x55af65696880 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.899+0000 7fa85e00fb80 10 client.24962 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.899+0000 7fa85e00fb80 10 client.24962 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.899+0000 7fa85e00fb80 20 client.24962 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.899+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55af656a6490 0x55af656a7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.899+0000 7fa85e00fb80 4 client.24962 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.899+0000 7fa85e00fb80 1 -- 172.21.15.98:0/753098817 --> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa840004ce0 con 0x55af656a6490 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.899+0000 7fa851522700 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55af656a6490 0x55af656a7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.899+0000 7fa851522700 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55af656a6490 0x55af656a7900 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b14e59f658b4ba0 server_cookie=617d05e3282888dc in_seq=0 out_seq=0 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.900+0000 7fa830ff9700 10 client.24962 ms_handle_connect on v2:172.21.15.203:6838/1340539385 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.900+0000 7fa830ff9700 1 -- 172.21.15.98:0/753098817 <== mds.1 v2:172.21.15.203:6838/1340539385 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fa840004ce0 con 0x55af656a6490 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.900+0000 7fa830ff9700 10 client.24962 handle_command_reply: tid=0 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.900+0000 7fa85e00fb80 1 client.24962 shutdown 2024-04-25T17:36:25.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.900+0000 7fa85e00fb80 1 -- 172.21.15.98:0/753098817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 msgr2=0x55af65693890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.900+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 0x55af65693890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0ced00 tx=0x7fa83c09da70 comp rx=0 tx=0).stop 2024-04-25T17:36:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.902+0000 7fa85e00fb80 1 -- 172.21.15.98:0/753098817 shutdown_connections 2024-04-25T17:36:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.902+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55af656a6490 0x55af656a7900 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.902+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af65696880 0x55af65693890 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.902+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af6568b0b0 0x55af65696050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.902+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/753098817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b9d0 0x55af65693350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.902+0000 7fa85e00fb80 1 -- 172.21.15.98:0/753098817 >> 172.21.15.98:0/753098817 conn(0x55af655b8840 msgr2=0x55af656871b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.903+0000 7fa830ff9700 0 client.24962 ms_handle_reset on v2:172.21.15.203:6838/1340539385 2024-04-25T17:36:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.904+0000 7fa85e00fb80 1 -- 172.21.15.98:0/753098817 shutdown_connections 2024-04-25T17:36:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.904+0000 7fa85e00fb80 1 -- 172.21.15.98:0/753098817 wait complete. 2024-04-25T17:36:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.904+0000 7fa85e00fb80 20 client.24962 trim_cache size 0 max 16384 2024-04-25T17:36:25.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa85e00fb80 1 Processor -- start 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa85e00fb80 1 -- start start 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b0b0 0x55af65698c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b9d0 0x55af65691c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af65692180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af656a5c40 con 0x55af65696880 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55af656a6310 con 0x55af6568b9d0 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55af656a69e0 con 0x55af6568b0b0 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa851d23700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af65692180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa851d23700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af65692180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55672/0 (socket says 172.21.15.98:55672) 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa851d23700 1 -- 172.21.15.98:0/2222343560 learned_addr learned my addr 172.21.15.98:0/2222343560 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa850d21700 1 --2- 172.21.15.98:0/2222343560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b9d0 0x55af65691c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.930+0000 7fa851522700 1 --2- 172.21.15.98:0/2222343560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b0b0 0x55af65698c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.931+0000 7fa851d23700 1 -- 172.21.15.98:0/2222343560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b0b0 msgr2=0x55af65698c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.931+0000 7fa851d23700 1 --2- 172.21.15.98:0/2222343560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b0b0 0x55af65698c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.931+0000 7fa851d23700 1 -- 172.21.15.98:0/2222343560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b9d0 msgr2=0x55af65691c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.931+0000 7fa851d23700 1 --2- 172.21.15.98:0/2222343560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b9d0 0x55af65691c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.931+0000 7fa851d23700 1 -- 172.21.15.98:0/2222343560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af6568cf20 con 0x55af65696880 2024-04-25T17:36:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.931+0000 7fa851d23700 1 --2- 172.21.15.98:0/2222343560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af65692180 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa83c022860 tx=0x7fa83c09af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3955e0f204ef9a10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.931+0000 7fa832ffd700 1 -- 172.21.15.98:0/2222343560 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c09cef0 con 0x55af65696880 2024-04-25T17:36:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.931+0000 7fa832ffd700 1 -- 172.21.15.98:0/2222343560 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c087b90 con 0x55af65696880 2024-04-25T17:36:25.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.931+0000 7fa832ffd700 1 -- 172.21.15.98:0/2222343560 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0a2c70 con 0x55af65696880 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 -- 172.21.15.98:0/2222343560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 msgr2=0x55af65692180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/2222343560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af65692180 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa83c022860 tx=0x7fa83c09af40 comp rx=0 tx=0).stop 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 -- 172.21.15.98:0/2222343560 shutdown_connections 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/2222343560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b0b0 0x55af65698c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/2222343560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af65692180 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/2222343560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b9d0 0x55af65691c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 -- 172.21.15.98:0/2222343560 >> 172.21.15.98:0/2222343560 conn(0x55af655b8840 msgr2=0x55af65687060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 -- 172.21.15.98:0/2222343560 shutdown_connections 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 -- 172.21.15.98:0/2222343560 wait complete. 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 -- start start 2024-04-25T17:36:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.932+0000 7fa85e00fb80 1 Processor -- start 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b0b0 0x55af656aa750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b9d0 0x55af656aac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa85e00fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af656ab1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af656a7780 con 0x55af65696880 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55af656a70b0 con 0x55af6568b0b0 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa85e00fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55af656a7e50 con 0x55af6568b9d0 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa85e00fb80 10 client.0 _ll_register_callbacks cb 0x55af655778d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851522700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b0b0 0x55af656aa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851d23700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af656ab1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851522700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b0b0 0x55af656aa750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:37702/0 (socket says 172.21.15.98:37702) 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851522700 1 -- 172.21.15.98:0/222012543 learned_addr learned my addr 172.21.15.98:0/222012543 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa850d21700 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b9d0 0x55af656aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851522700 1 -- 172.21.15.98:0/222012543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b9d0 msgr2=0x55af656aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851522700 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b9d0 0x55af656aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851522700 1 -- 172.21.15.98:0/222012543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 msgr2=0x55af656ab1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851522700 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af656ab1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851522700 1 -- 172.21.15.98:0/222012543 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af656882a0 con 0x55af6568b0b0 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851d23700 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af656ab1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa851522700 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b0b0 0x55af656aa750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa844010010 tx=0x7fa844004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb0790e36f581fe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa830ff9700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.933+0000 7fa830ff9700 1 -- 172.21.15.98:0/222012543 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8440175c0 con 0x55af6568b0b0 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa85e00fb80 1 -- 172.21.15.98:0/222012543 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55af655ec430 con 0x55af6568b0b0 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa830ff9700 1 -- 172.21.15.98:0/222012543 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fa844017760 con 0x55af6568b0b0 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa830ff9700 1 -- 172.21.15.98:0/222012543 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa844017960 con 0x55af6568b0b0 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa85e00fb80 10 client.24968 fetch_fsmap learned FSMap version 26 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa85e00fb80 1 -- 172.21.15.98:0/222012543 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55af6570d6e0 con 0x55af6568b0b0 2024-04-25T17:36:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa830ff9700 1 -- 172.21.15.98:0/222012543 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55af655ec430 con 0x55af6568b0b0 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa830ff9700 1 -- 172.21.15.98:0/222012543 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa844015410 con 0x55af6568b0b0 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa85e00fb80 10 client.24968 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa85e00fb80 10 client.24968 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa85e00fb80 20 client.24968 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55af65712350 0x55af65711060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa85e00fb80 4 client.24968 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa85e00fb80 1 -- 172.21.15.98:0/222012543 --> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] -- command(tid 0: {"prefix": "status"}) -- 0x55af655e64d0 con 0x55af65712350 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.934+0000 7fa850d21700 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55af65712350 0x55af65711060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa830ff9700 10 client.24968 ms_handle_connect on v2:172.21.15.203:6838/1340539385 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa850d21700 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55af65712350 0x55af65711060 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f63419c6d4441734 server_cookie=f3a85a5294e61f6a in_seq=0 out_seq=0 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa830ff9700 1 -- 172.21.15.98:0/222012543 <== mds.1 v2:172.21.15.203:6838/1340539385 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55af655e64d0 con 0x55af65712350 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa830ff9700 10 client.24968 handle_command_reply: tid=0 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 client.24968 shutdown 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 -- 172.21.15.98:0/222012543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b0b0 msgr2=0x55af656aa750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b0b0 0x55af656aa750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa844010010 tx=0x7fa844004ad0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 -- 172.21.15.98:0/222012543 shutdown_connections 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.203:6838/1340539385,v1:172.21.15.203:6839/1340539385] conn(0x55af65712350 0x55af65711060 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af65696880 0x55af656ab1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af6568b0b0 0x55af656aa750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa830ff9700 0 client.24968 ms_handle_reset on v2:172.21.15.203:6838/1340539385 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 --2- 172.21.15.98:0/222012543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af6568b9d0 0x55af656aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 -- 172.21.15.98:0/222012543 >> 172.21.15.98:0/222012543 conn(0x55af655b8840 msgr2=0x55af65693310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 -- 172.21.15.98:0/222012543 shutdown_connections 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 1 -- 172.21.15.98:0/222012543 wait complete. 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.935+0000 7fa85e00fb80 20 client.24968 trim_cache size 0 max 16384 2024-04-25T17:36:25.941 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24445, 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 213.82581852800001, 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 26, 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 92, 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 258.21865982700001 2024-04-25T17:36:25.942 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:36:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 -- 172.21.15.98:0/3900037326 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8340a3f50 msgr2=0x7fa8340a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8340a3f50 0x7fa8340a6410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa844009c10 tx=0x7fa84400b040 comp rx=0 tx=0).stop 2024-04-25T17:36:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 -- 172.21.15.98:0/3900037326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c107980 msgr2=0x7fa84c1a4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c107980 0x7fa84c1a4f20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0098e0 tx=0x7fa83c0047b0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 -- 172.21.15.98:0/3900037326 shutdown_connections 2024-04-25T17:36:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8340a3f50 0x7fa8340a6410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c107980 0x7fa84c1a4f20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa84c101930 0x7fa84c199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 --2- 172.21.15.98:0/3900037326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa84c1022f0 0x7fa84c19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 -- 172.21.15.98:0/3900037326 >> 172.21.15.98:0/3900037326 conn(0x7fa84c0788b0 msgr2=0x7fa84c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 -- 172.21.15.98:0/3900037326 shutdown_connections 2024-04-25T17:36:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:25.936+0000 7fa853786700 1 -- 172.21.15.98:0/3900037326 wait complete. 2024-04-25T17:36:26.004 INFO:teuthology.orchestra.run.smithi098.stderr:500000 of 3200000 tuples (15%) done (elapsed 0.63 s, remaining 3.42 s) 2024-04-25T17:36:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:600000 of 3200000 tuples (18%) done (elapsed 0.79 s, remaining 3.41 s) 2024-04-25T17:36:26.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:26 smithi098 ceph-mon[29192]: pgmap v18: 97 pgs: 97 active+clean; 1.8 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 52 KiB/s wr, 8 op/s 2024-04-25T17:36:26.246 INFO:teuthology.orchestra.run.smithi098.stderr:700000 of 3200000 tuples (21%) done (elapsed 0.87 s, remaining 3.12 s) 2024-04-25T17:36:26.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:26 smithi173 ceph-mon[32856]: pgmap v18: 97 pgs: 97 active+clean; 1.8 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 52 KiB/s wr, 8 op/s 2024-04-25T17:36:26.409 INFO:teuthology.orchestra.run.smithi098.stderr:800000 of 3200000 tuples (25%) done (elapsed 1.04 s, remaining 3.11 s) 2024-04-25T17:36:26.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:26 smithi203 ceph-mon[32834]: pgmap v18: 97 pgs: 97 active+clean; 1.8 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 52 KiB/s wr, 8 op/s 2024-04-25T17:36:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:900000 of 3200000 tuples (28%) done (elapsed 1.21 s, remaining 3.08 s) 2024-04-25T17:36:26.654 INFO:teuthology.orchestra.run.smithi098.stderr:1000000 of 3200000 tuples (31%) done (elapsed 1.28 s, remaining 2.82 s) 2024-04-25T17:36:26.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-25T17:36:26.808 INFO:teuthology.orchestra.run.smithi098.stderr:1100000 of 3200000 tuples (34%) done (elapsed 1.44 s, remaining 2.74 s) 2024-04-25T17:36:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:1200000 of 3200000 tuples (37%) done (elapsed 1.53 s, remaining 2.54 s) 2024-04-25T17:36:27.002 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:27.045 INFO:teuthology.orchestra.run.smithi098.stderr:1300000 of 3200000 tuples (40%) done (elapsed 1.67 s, remaining 2.45 s) 2024-04-25T17:36:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:1400000 of 3200000 tuples (43%) done (elapsed 1.79 s, remaining 2.30 s) 2024-04-25T17:36:27.316 INFO:teuthology.orchestra.run.smithi098.stderr:1500000 of 3200000 tuples (46%) done (elapsed 1.94 s, remaining 2.20 s) 2024-04-25T17:36:27.454 INFO:teuthology.orchestra.run.smithi098.stderr:1600000 of 3200000 tuples (50%) done (elapsed 2.08 s, remaining 2.08 s) 2024-04-25T17:36:27.545 INFO:teuthology.orchestra.run.smithi098.stderr:1700000 of 3200000 tuples (53%) done (elapsed 2.17 s, remaining 1.92 s) 2024-04-25T17:36:27.704 INFO:teuthology.orchestra.run.smithi098.stderr:1800000 of 3200000 tuples (56%) done (elapsed 2.33 s, remaining 1.81 s) 2024-04-25T17:36:27.835 INFO:teuthology.orchestra.run.smithi098.stderr:1900000 of 3200000 tuples (59%) done (elapsed 2.46 s, remaining 1.69 s) 2024-04-25T17:36:27.929 INFO:teuthology.orchestra.run.smithi098.stderr:2000000 of 3200000 tuples (62%) done (elapsed 2.56 s, remaining 1.53 s) 2024-04-25T17:36:28.079 INFO:teuthology.orchestra.run.smithi098.stderr:2100000 of 3200000 tuples (65%) done (elapsed 2.71 s, remaining 1.42 s) 2024-04-25T17:36:28.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:28 smithi098 ceph-mon[29192]: pgmap v19: 97 pgs: 97 active+clean; 44 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 3.7 MiB/s wr, 213 op/s 2024-04-25T17:36:28.242 INFO:teuthology.orchestra.run.smithi098.stderr:2200000 of 3200000 tuples (68%) done (elapsed 2.87 s, remaining 1.30 s) 2024-04-25T17:36:28.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:28 smithi173 ceph-mon[32856]: pgmap v19: 97 pgs: 97 active+clean; 44 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 3.7 MiB/s wr, 213 op/s 2024-04-25T17:36:28.332 INFO:teuthology.orchestra.run.smithi098.stderr:2300000 of 3200000 tuples (71%) done (elapsed 2.96 s, remaining 1.16 s) 2024-04-25T17:36:28.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:28 smithi203 ceph-mon[32834]: pgmap v19: 97 pgs: 97 active+clean; 44 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 3.7 MiB/s wr, 213 op/s 2024-04-25T17:36:28.488 INFO:teuthology.orchestra.run.smithi098.stderr:2400000 of 3200000 tuples (75%) done (elapsed 3.12 s, remaining 1.04 s) 2024-04-25T17:36:28.648 INFO:teuthology.orchestra.run.smithi098.stderr:2500000 of 3200000 tuples (78%) done (elapsed 3.28 s, remaining 0.92 s) 2024-04-25T17:36:28.739 INFO:teuthology.orchestra.run.smithi098.stderr:2600000 of 3200000 tuples (81%) done (elapsed 3.37 s, remaining 0.78 s) 2024-04-25T17:36:28.964 INFO:teuthology.orchestra.run.smithi098.stderr:2700000 of 3200000 tuples (84%) done (elapsed 3.59 s, remaining 0.67 s) 2024-04-25T17:36:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:2800000 of 3200000 tuples (87%) done (elapsed 3.69 s, remaining 0.53 s) 2024-04-25T17:36:29.240 INFO:teuthology.orchestra.run.smithi098.stderr:2900000 of 3200000 tuples (90%) done (elapsed 3.87 s, remaining 0.40 s) 2024-04-25T17:36:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 -- 172.21.15.98:0/3409404809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63241034b0 msgr2=0x7f632410f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 --2- 172.21.15.98:0/3409404809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63241034b0 0x7f632410f8e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f631c00b0a0 tx=0x7f631c00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 -- 172.21.15.98:0/3409404809 shutdown_connections 2024-04-25T17:36:29.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 --2- 172.21.15.98:0/3409404809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6324108a80 0x7f6324108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 --2- 172.21.15.98:0/3409404809 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63241034b0 0x7f632410f8e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 --2- 172.21.15.98:0/3409404809 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6324102af0 0x7f6324102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 -- 172.21.15.98:0/3409404809 >> 172.21.15.98:0/3409404809 conn(0x7f6324079ad0 msgr2=0x7f6324079ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 -- 172.21.15.98:0/3409404809 shutdown_connections 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 -- 172.21.15.98:0/3409404809 wait complete. 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 Processor -- start 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 -- start start 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6324102af0 0x7f6324199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.869+0000 7f632abaa700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63241034b0 0x7f63241995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f632abaa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6324108a80 0x7f632419d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f632abaa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6324112c10 con 0x7f6324108a80 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f632abaa700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6324112a90 con 0x7f6324102af0 2024-04-25T17:36:29.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f632abaa700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6324112d90 con 0x7f63241034b0 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6329147700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6324108a80 0x7f632419d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6329147700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6324108a80 0x7f632419d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55696/0 (socket says 172.21.15.98:55696) 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6329147700 1 -- 172.21.15.98:0/1950996350 learned_addr learned my addr 172.21.15.98:0/1950996350 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6323fff700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63241034b0 0x7f63241995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6328946700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6324102af0 0x7f6324199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6329147700 1 -- 172.21.15.98:0/1950996350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63241034b0 msgr2=0x7f63241995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6329147700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63241034b0 0x7f63241995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6329147700 1 -- 172.21.15.98:0/1950996350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6324102af0 msgr2=0x7f6324199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6329147700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6324102af0 0x7f6324199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6329147700 1 -- 172.21.15.98:0/1950996350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632419e090 con 0x7f6324108a80 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6329147700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6324108a80 0x7f632419d950 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f631c00b830 tx=0x7f631c004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6321ffb700 1 -- 172.21.15.98:0/1950996350 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c021070 con 0x7f6324108a80 2024-04-25T17:36:29.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6321ffb700 1 -- 172.21.15.98:0/1950996350 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f631c007e30 con 0x7f6324108a80 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f632abaa700 1 -- 172.21.15.98:0/1950996350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f632419e320 con 0x7f6324108a80 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.870+0000 7f6321ffb700 1 -- 172.21.15.98:0/1950996350 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c004700 con 0x7f6324108a80 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.871+0000 7f632abaa700 1 -- 172.21.15.98:0/1950996350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63241a5c00 con 0x7f6324108a80 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.872+0000 7f6321ffb700 1 -- 172.21.15.98:0/1950996350 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f631c007980 con 0x7f6324108a80 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.873+0000 7f6321ffb700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f63140a3ed0 0x7f63140a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.873+0000 7f6328946700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f63140a3ed0 0x7f63140a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.873+0000 7f6321ffb700 1 -- 172.21.15.98:0/1950996350 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f631c00f020 con 0x7f6324108a80 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6328946700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f63140a3ed0 0x7f63140a6390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f631800abd0 tx=0x7f631800a380 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6335433b80 1 Processor -- start 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6335433b80 1 -- start start 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6335433b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526770b0 0x55cc52677490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6335433b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc52682340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6335433b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc52684c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6335433b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc525cac40 con 0x55cc526779d0 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6335433b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cc525718e0 con 0x55cc52682880 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6335433b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cc525c6c00 con 0x55cc526770b0 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6323fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc52682340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6323fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc52682340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55712/0 (socket says 172.21.15.98:55712) 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.874+0000 7f6323fff700 1 -- 172.21.15.98:0/2346945834 learned_addr learned my addr 172.21.15.98:0/2346945834 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f6329147700 1 --2- 172.21.15.98:0/2346945834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc52684c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f6328946700 1 --2- 172.21.15.98:0/2346945834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526770b0 0x55cc52677490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f6323fff700 1 -- 172.21.15.98:0/2346945834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526770b0 msgr2=0x55cc52677490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f6323fff700 1 --2- 172.21.15.98:0/2346945834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526770b0 0x55cc52677490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f6323fff700 1 -- 172.21.15.98:0/2346945834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 msgr2=0x55cc52684c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f6323fff700 1 --2- 172.21.15.98:0/2346945834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc52684c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f6323fff700 1 -- 172.21.15.98:0/2346945834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc525356f0 con 0x55cc526779d0 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f6328946700 1 --2- 172.21.15.98:0/2346945834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526770b0 0x55cc52677490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:36:29.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f6323fff700 1 --2- 172.21.15.98:0/2346945834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc52682340 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f631000bd30 tx=0x7f63100106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c56040c94f0e603 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f630effd700 1 -- 172.21.15.98:0/2346945834 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310010db0 con 0x55cc526779d0 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f630effd700 1 -- 172.21.15.98:0/2346945834 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63100044e0 con 0x55cc526779d0 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.875+0000 7f630effd700 1 -- 172.21.15.98:0/2346945834 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310011440 con 0x55cc526779d0 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 -- 172.21.15.98:0/2346945834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 msgr2=0x55cc52682340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 --2- 172.21.15.98:0/2346945834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc52682340 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f631000bd30 tx=0x7f63100106b0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 -- 172.21.15.98:0/2346945834 shutdown_connections 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 --2- 172.21.15.98:0/2346945834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526770b0 0x55cc52677490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 --2- 172.21.15.98:0/2346945834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc52684c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 --2- 172.21.15.98:0/2346945834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc52682340 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 -- 172.21.15.98:0/2346945834 >> 172.21.15.98:0/2346945834 conn(0x55cc525a4840 msgr2=0x55cc52673580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 -- 172.21.15.98:0/2346945834 shutdown_connections 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 -- 172.21.15.98:0/2346945834 wait complete. 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 -- start start 2024-04-25T17:36:29.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.876+0000 7f6335433b80 1 Processor -- start 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6335433b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc52681fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6335433b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 0x55cc5267f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6335433b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc5267f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6335433b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc52685370 con 0x55cc526770b0 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6335433b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cc52584560 con 0x55cc52682880 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6335433b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cc526854f0 con 0x55cc526779d0 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6335433b80 10 client.0 _ll_register_callbacks cb 0x55cc525638d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6328946700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc52681fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6328946700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc52681fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55722/0 (socket says 172.21.15.98:55722) 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6328946700 1 -- 172.21.15.98:0/4031767779 learned_addr learned my addr 172.21.15.98:0/4031767779 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6323fff700 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 0x55cc5267f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6329147700 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc5267f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6328946700 1 -- 172.21.15.98:0/4031767779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 msgr2=0x55cc5267f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6328946700 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 0x55cc5267f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6328946700 1 -- 172.21.15.98:0/4031767779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 msgr2=0x55cc5267f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6328946700 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc5267f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6328946700 1 -- 172.21.15.98:0/4031767779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc52518880 con 0x55cc526770b0 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6328946700 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc52681fa0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6318004670 tx=0x7f63180047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b39203419ff90d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6303fff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6303fff700 1 -- 172.21.15.98:0/4031767779 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6318016070 con 0x55cc526770b0 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6335433b80 1 -- 172.21.15.98:0/4031767779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63100044e0 con 0x55cc526770b0 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.877+0000 7f6303fff700 1 -- 172.21.15.98:0/4031767779 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f6318010840 con 0x55cc526770b0 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6303fff700 1 -- 172.21.15.98:0/4031767779 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6318010a40 con 0x55cc526770b0 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6335433b80 10 client.15351 fetch_fsmap learned FSMap version 26 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6335433b80 1 -- 172.21.15.98:0/4031767779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cc5268fc80 con 0x55cc526770b0 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6303fff700 1 -- 172.21.15.98:0/4031767779 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63100044e0 con 0x55cc526770b0 2024-04-25T17:36:29.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6303fff700 1 -- 172.21.15.98:0/4031767779 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6318019070 con 0x55cc526770b0 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6335433b80 10 client.15351 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6335433b80 10 client.15351 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6335433b80 20 client.15351 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6335433b80 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc526922d0 0x55cc52693740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6335433b80 4 client.15351 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6335433b80 1 -- 172.21.15.98:0/4031767779 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc525d8430 con 0x55cc526922d0 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.878+0000 7f6323fff700 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc526922d0 0x55cc52693740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.879+0000 7f6303fff700 10 client.15351 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.879+0000 7f6323fff700 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc526922d0 0x55cc52693740 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21b56bb81490f110 server_cookie=fa7e552c8e1e3771 in_seq=0 out_seq=0 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.879+0000 7f6303fff700 1 -- 172.21.15.98:0/4031767779 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc525d8430 con 0x55cc526922d0 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.879+0000 7f6303fff700 10 client.15351 handle_command_reply: tid=0 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.879+0000 7f6335433b80 1 client.15351 shutdown 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 1 -- 172.21.15.98:0/4031767779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 msgr2=0x55cc52681fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:29.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc52681fa0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6318004670 tx=0x7f63180047c0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 1 -- 172.21.15.98:0/4031767779 shutdown_connections 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc526922d0 0x55cc52693740 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc5267f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6303fff700 0 client.15351 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc52681fa0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 1 --2- 172.21.15.98:0/4031767779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 0x55cc5267f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 1 -- 172.21.15.98:0/4031767779 >> 172.21.15.98:0/4031767779 conn(0x55cc525a4840 msgr2=0x55cc526731b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 1 -- 172.21.15.98:0/4031767779 shutdown_connections 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 1 -- 172.21.15.98:0/4031767779 wait complete. 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.880+0000 7f6335433b80 20 client.15351 trim_cache size 0 max 16384 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.903+0000 7f6335433b80 1 Processor -- start 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.903+0000 7f6335433b80 1 -- start start 2024-04-25T17:36:29.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.903+0000 7f6335433b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc526770b0 0x55cc52681ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.903+0000 7f6335433b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc5269cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.903+0000 7f6335433b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc52682880 0x55cc5269d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.903+0000 7f6335433b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc52691a80 con 0x55cc526779d0 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.903+0000 7f6335433b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cc52692150 con 0x55cc526770b0 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.903+0000 7f6335433b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cc52692820 con 0x55cc52682880 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.903+0000 7f6323fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc5269cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f6323fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc5269cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55728/0 (socket says 172.21.15.98:55728) 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f6323fff700 1 -- 172.21.15.98:0/1287598244 learned_addr learned my addr 172.21.15.98:0/1287598244 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f6328946700 1 --2- 172.21.15.98:0/1287598244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc526770b0 0x55cc52681ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f6329147700 1 --2- 172.21.15.98:0/1287598244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc52682880 0x55cc5269d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f6323fff700 1 -- 172.21.15.98:0/1287598244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc52682880 msgr2=0x55cc5269d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f6323fff700 1 --2- 172.21.15.98:0/1287598244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc52682880 0x55cc5269d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f6323fff700 1 -- 172.21.15.98:0/1287598244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc526770b0 msgr2=0x55cc52681ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:29.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f6323fff700 1 --2- 172.21.15.98:0/1287598244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc526770b0 0x55cc52681ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f6323fff700 1 -- 172.21.15.98:0/1287598244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc52678f20 con 0x55cc526779d0 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f6323fff700 1 --2- 172.21.15.98:0/1287598244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc5269cba0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f631000bc50 tx=0x7f6310005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2998218cdfcec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f630e7fc700 1 -- 172.21.15.98:0/1287598244 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631002e070 con 0x55cc526779d0 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f630e7fc700 1 -- 172.21.15.98:0/1287598244 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f631000b710 con 0x55cc526779d0 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.904+0000 7f630e7fc700 1 -- 172.21.15.98:0/1287598244 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310005690 con 0x55cc526779d0 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 -- 172.21.15.98:0/1287598244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 msgr2=0x55cc5269cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 --2- 172.21.15.98:0/1287598244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc5269cba0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f631000bc50 tx=0x7f6310005d00 comp rx=0 tx=0).stop 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 -- 172.21.15.98:0/1287598244 shutdown_connections 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 --2- 172.21.15.98:0/1287598244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc52682880 0x55cc5269d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 --2- 172.21.15.98:0/1287598244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc526770b0 0x55cc52681ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 --2- 172.21.15.98:0/1287598244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526779d0 0x55cc5269cba0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 -- 172.21.15.98:0/1287598244 >> 172.21.15.98:0/1287598244 conn(0x55cc525a4840 msgr2=0x55cc52691140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 -- 172.21.15.98:0/1287598244 shutdown_connections 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 -- 172.21.15.98:0/1287598244 wait complete. 2024-04-25T17:36:29.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 -- start start 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.905+0000 7f6335433b80 1 Processor -- start 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6335433b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc5269cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6335433b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 0x55cc5269d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6335433b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc52695b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6335433b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc526935c0 con 0x55cc526770b0 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6335433b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cc52692ef0 con 0x55cc52682880 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6335433b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cc52693c90 con 0x55cc526779d0 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6335433b80 10 client.0 _ll_register_callbacks cb 0x55cc525638d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6328946700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc5269cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6323fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 0x55cc5269d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6328946700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc5269cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55738/0 (socket says 172.21.15.98:55738) 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6328946700 1 -- 172.21.15.98:0/4076168678 learned_addr learned my addr 172.21.15.98:0/4076168678 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6329147700 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc52695b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6323fff700 1 -- 172.21.15.98:0/4076168678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 msgr2=0x55cc52695b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6323fff700 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc52695b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6323fff700 1 -- 172.21.15.98:0/4076168678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 msgr2=0x55cc5269cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6323fff700 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc5269cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6323fff700 1 -- 172.21.15.98:0/4076168678 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc52696360 con 0x55cc526779d0 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6328946700 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc5269cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6329147700 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc52695b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6323fff700 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 0x55cc5269d100 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f631000bd00 tx=0x7f6310010fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c86b2aa0c5cf1a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.906+0000 7f6303fff700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6303fff700 1 -- 172.21.15.98:0/4076168678 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631002e070 con 0x55cc526779d0 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6335433b80 1 -- 172.21.15.98:0/4076168678 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f631000b710 con 0x55cc526779d0 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6303fff700 1 -- 172.21.15.98:0/4076168678 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f6310005550 con 0x55cc526779d0 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6303fff700 1 -- 172.21.15.98:0/4076168678 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6310003680 con 0x55cc526779d0 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6335433b80 10 client.24937 fetch_fsmap learned FSMap version 26 2024-04-25T17:36:29.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6335433b80 1 -- 172.21.15.98:0/4076168678 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cc526f9f00 con 0x55cc526779d0 2024-04-25T17:36:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6303fff700 1 -- 172.21.15.98:0/4076168678 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f631000b710 con 0x55cc526779d0 2024-04-25T17:36:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6303fff700 1 -- 172.21.15.98:0/4076168678 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f63100294b0 con 0x55cc526779d0 2024-04-25T17:36:29.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6335433b80 10 client.24937 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T17:36:29.347 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:36:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "79754014-0328-11ef-bc93-c7b262605968", 2024-04-25T17:36:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-04-25T17:36:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24461, 2024-04-25T17:36:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T17:36:29.347 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T17:36:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T17:36:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 230.838964436, 2024-04-25T17:36:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 26, 2024-04-25T17:36:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 92, 2024-04-25T17:36:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T17:36:29.348 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 249.5664223 2024-04-25T17:36:29.348 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:36:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6335433b80 10 client.24937 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T17:36:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6335433b80 20 client.24937 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:29.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6335433b80 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc526fe3b0 0x55cc526fd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6335433b80 4 client.24937 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6335433b80 1 -- 172.21.15.98:0/4076168678 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "status"}) -- 0x55cc525d24d0 con 0x55cc526fe3b0 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.907+0000 7f6328946700 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc526fe3b0 0x55cc526fd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6303fff700 10 client.24937 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6328946700 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc526fe3b0 0x55cc526fd0c0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab36753d5bbf1917 server_cookie=da580222268bb5ee in_seq=0 out_seq=0 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6303fff700 1 -- 172.21.15.98:0/4076168678 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+320 (crc 0 0 0) 0x55cc525d24d0 con 0x55cc526fe3b0 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6303fff700 10 client.24937 handle_command_reply: tid=0 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 client.24937 shutdown 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 -- 172.21.15.98:0/4076168678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 msgr2=0x55cc5269d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 0x55cc5269d100 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f631000bd00 tx=0x7f6310010fb0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 -- 172.21.15.98:0/4076168678 shutdown_connections 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc526fe3b0 0x55cc526fd0c0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc52682880 0x55cc52695b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc526770b0 0x55cc5269cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 --2- 172.21.15.98:0/4076168678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc526779d0 0x55cc5269d100 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 -- 172.21.15.98:0/4076168678 >> 172.21.15.98:0/4076168678 conn(0x55cc525a4840 msgr2=0x55cc5267f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6303fff700 0 client.24937 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 -- 172.21.15.98:0/4076168678 shutdown_connections 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 1 -- 172.21.15.98:0/4076168678 wait complete. 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.908+0000 7f6335433b80 20 client.24937 trim_cache size 0 max 16384 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 -- 172.21.15.98:0/1950996350 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f63140a3ed0 msgr2=0x7f63140a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f63140a3ed0 0x7f63140a6390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f631800abd0 tx=0x7f631800a380 comp rx=0 tx=0).stop 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 -- 172.21.15.98:0/1950996350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6324108a80 msgr2=0x7f632419d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6324108a80 0x7f632419d950 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f631c00b830 tx=0x7f631c004ce0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 -- 172.21.15.98:0/1950996350 shutdown_connections 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f63140a3ed0 0x7f63140a6390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6324108a80 0x7f632419d950 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6324102af0 0x7f6324199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 --2- 172.21.15.98:0/1950996350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63241034b0 0x7f63241995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 -- 172.21.15.98:0/1950996350 >> 172.21.15.98:0/1950996350 conn(0x7f6324079ad0 msgr2=0x7f6324101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 -- 172.21.15.98:0/1950996350 shutdown_connections 2024-04-25T17:36:29.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:28.909+0000 7f632abaa700 1 -- 172.21.15.98:0/1950996350 wait complete. 2024-04-25T17:36:29.410 INFO:teuthology.orchestra.run.smithi098.stderr:3000000 of 3200000 tuples (93%) done (elapsed 4.04 s, remaining 0.27 s) 2024-04-25T17:36:29.500 INFO:teuthology.orchestra.run.smithi098.stderr:3100000 of 3200000 tuples (96%) done (elapsed 4.13 s, remaining 0.13 s) 2024-04-25T17:36:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:3200000 of 3200000 tuples (100%) done (elapsed 4.28 s, remaining 0.00 s) 2024-04-25T17:36:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:vacuuming... 2024-04-25T17:36:30.135 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-25T17:36:30.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:30 smithi173 ceph-mon[32856]: pgmap v20: 97 pgs: 97 active+clean; 44 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 3.7 MiB/s wr, 210 op/s 2024-04-25T17:36:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:creating primary keys... 2024-04-25T17:36:30.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:30 smithi098 ceph-mon[29192]: pgmap v20: 97 pgs: 97 active+clean; 44 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 3.7 MiB/s wr, 210 op/s 2024-04-25T17:36:30.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:30 smithi203 ceph-mon[32834]: pgmap v20: 97 pgs: 97 active+clean; 44 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 3.7 MiB/s wr, 210 op/s 2024-04-25T17:36:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:done in 6.18 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 4.33 s, vacuum 0.63 s, primary keys 1.21 s). 2024-04-25T17:36:31.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-04-25T17:36:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:starting vacuum...end. 2024-04-25T17:36:32.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:32 smithi173 ceph-mon[32856]: pgmap v21: 97 pgs: 97 active+clean; 237 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 36 MiB/s wr, 228 op/s 2024-04-25T17:36:32.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:32 smithi098 ceph-mon[29192]: pgmap v21: 97 pgs: 97 active+clean; 237 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 36 MiB/s wr, 228 op/s 2024-04-25T17:36:32.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:32 smithi203 ceph-mon[32834]: pgmap v21: 97 pgs: 97 active+clean; 237 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 36 MiB/s wr, 228 op/s 2024-04-25T17:36:34.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:34 smithi173 ceph-mon[32856]: pgmap v22: 97 pgs: 97 active+clean; 381 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 59 MiB/s wr, 242 op/s 2024-04-25T17:36:34.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:34 smithi098 ceph-mon[29192]: pgmap v22: 97 pgs: 97 active+clean; 381 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 59 MiB/s wr, 242 op/s 2024-04-25T17:36:34.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:34 smithi203 ceph-mon[32834]: pgmap v22: 97 pgs: 97 active+clean; 381 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 59 MiB/s wr, 242 op/s 2024-04-25T17:36:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.197+0000 7f17ef5e8640 1 -- 172.21.15.98:0/2188150063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e81010f0 msgr2=0x7f17e81014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.197+0000 7f17ef5e8640 1 --2- 172.21.15.98:0/2188150063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e81010f0 0x7f17e81014d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f17e400d910 tx=0x7f17e402c370 comp rx=0 tx=0).stop 2024-04-25T17:36:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.198+0000 7f17ef5e8640 1 -- 172.21.15.98:0/2188150063 shutdown_connections 2024-04-25T17:36:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.198+0000 7f17ef5e8640 1 --2- 172.21.15.98:0/2188150063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17e8106170 0x7f17e810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.198+0000 7f17ef5e8640 1 --2- 172.21.15.98:0/2188150063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e8101aa0 0x7f17e8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.198+0000 7f17ef5e8640 1 --2- 172.21.15.98:0/2188150063 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e81010f0 0x7f17e81014d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.198+0000 7f17ef5e8640 1 -- 172.21.15.98:0/2188150063 >> 172.21.15.98:0/2188150063 conn(0x7f17e8077fd0 msgr2=0x7f17e80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.198+0000 7f17ef5e8640 1 -- 172.21.15.98:0/2188150063 shutdown_connections 2024-04-25T17:36:36.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.198+0000 7f17ef5e8640 1 -- 172.21.15.98:0/2188150063 wait complete. 2024-04-25T17:36:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.199+0000 7f17ef5e8640 1 Processor -- start 2024-04-25T17:36:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.199+0000 7f17ef5e8640 1 -- start start 2024-04-25T17:36:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.199+0000 7f17ef5e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e81010f0 0x7f17e810d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ef5e8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e8106170 0x7f17e810dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ef5e8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17e810b6b0 0x7f17e810bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ef5e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17e810f690 con 0x7f17e81010f0 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ef5e8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f17e810f510 con 0x7f17e810b6b0 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ef5e8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17e810f810 con 0x7f17e8106170 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ecb83640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e8106170 0x7f17e810dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17edb85640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17e810b6b0 0x7f17e810bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ed384640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e81010f0 0x7f17e810d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ecb83640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e8106170 0x7f17e810dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:34980/0 (socket says 172.21.15.98:34980) 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ecb83640 1 -- 172.21.15.98:0/1831774798 learned_addr learned my addr 172.21.15.98:0/1831774798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ed384640 1 -- 172.21.15.98:0/1831774798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e8106170 msgr2=0x7f17e810dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ed384640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e8106170 0x7f17e810dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ed384640 1 -- 172.21.15.98:0/1831774798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17e810b6b0 msgr2=0x7f17e810bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ed384640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17e810b6b0 0x7f17e810bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ed384640 1 -- 172.21.15.98:0/1831774798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e810c3f0 con 0x7f17e81010f0 2024-04-25T17:36:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ecb83640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e8106170 0x7f17e810dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17edb85640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17e810b6b0 0x7f17e810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.200+0000 7f17ed384640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e81010f0 0x7f17e810d570 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f17e402c960 tx=0x7f17e40049e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.201+0000 7f17de7fc640 1 -- 172.21.15.98:0/1831774798 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4041070 con 0x7f17e81010f0 2024-04-25T17:36:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.201+0000 7f17de7fc640 1 -- 172.21.15.98:0/1831774798 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e40035f0 con 0x7f17e81010f0 2024-04-25T17:36:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.201+0000 7f17ef5e8640 1 -- 172.21.15.98:0/1831774798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e81ab710 con 0x7f17e81010f0 2024-04-25T17:36:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.201+0000 7f17de7fc640 1 -- 172.21.15.98:0/1831774798 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e403c6a0 con 0x7f17e81010f0 2024-04-25T17:36:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.201+0000 7f17ef5e8640 1 -- 172.21.15.98:0/1831774798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e81abb10 con 0x7f17e81010f0 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.204+0000 7f17f23d0740 1 Processor -- start 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17f23d0740 1 -- start start 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466b620 0x55855466ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 0x5585546762c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558554676800 0x558554678bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17f23d0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5585545d0970 con 0x55855466bf40 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17f23d0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55855456d4a0 con 0x55855466b620 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17f23d0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5585545cc930 con 0x558554676800 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17edb85640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466b620 0x55855466ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17edb85640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466b620 0x55855466ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:45038/0 (socket says 172.21.15.98:45038) 2024-04-25T17:36:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17edb85640 1 -- 172.21.15.98:0/2676425569 learned_addr learned my addr 172.21.15.98:0/2676425569 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17de7fc640 1 -- 172.21.15.98:0/1831774798 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f17e4044070 con 0x7f17e81010f0 2024-04-25T17:36:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17ecb83640 1 --2- 172.21.15.98:0/2676425569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558554676800 0x558554678bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.205+0000 7f17ed384640 1 --2- 172.21.15.98:0/2676425569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 0x5585546762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.206+0000 7f17edb85640 1 -- 172.21.15.98:0/2676425569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558554676800 msgr2=0x558554678bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.206+0000 7f17edb85640 1 --2- 172.21.15.98:0/2676425569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558554676800 0x558554678bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.206+0000 7f17edb85640 1 -- 172.21.15.98:0/2676425569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 msgr2=0x5585546762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.206+0000 7f17edb85640 1 --2- 172.21.15.98:0/2676425569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 0x5585546762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.206+0000 7f17edb85640 1 -- 172.21.15.98:0/2676425569 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585542e3210 con 0x55855466b620 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.206+0000 7f17edb85640 1 --2- 172.21.15.98:0/2676425569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466b620 0x55855466ba00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17e000a870 tx=0x7f17e0009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b6cb315ed1260eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.206+0000 7f17de7fc640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f17d40a3f60 0x7f17d40a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.207+0000 7f17de7fc640 1 -- 172.21.15.98:0/1831774798 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f17e40f0730 con 0x7f17e81010f0 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.207+0000 7f17d37fe640 1 -- 172.21.15.98:0/2676425569 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0018020 con 0x55855466b620 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.207+0000 7f17d37fe640 1 -- 172.21.15.98:0/2676425569 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e0014400 con 0x55855466b620 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.207+0000 7f17d37fe640 1 -- 172.21.15.98:0/2676425569 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e00135c0 con 0x55855466b620 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.207+0000 7f17ed384640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f17d40a3f60 0x7f17d40a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.207+0000 7f17f23d0740 1 -- 172.21.15.98:0/2676425569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466b620 msgr2=0x55855466ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.207+0000 7f17f23d0740 1 --2- 172.21.15.98:0/2676425569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466b620 0x55855466ba00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17e000a870 tx=0x7f17e0009650 comp rx=0 tx=0).stop 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.208+0000 7f17ed384640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f17d40a3f60 0x7f17d40a6420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f17e40ede40 tx=0x7f17e4005fd0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.208+0000 7f17f23d0740 1 -- 172.21.15.98:0/2676425569 shutdown_connections 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.208+0000 7f17f23d0740 1 --2- 172.21.15.98:0/2676425569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558554676800 0x558554678bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.208+0000 7f17f23d0740 1 --2- 172.21.15.98:0/2676425569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 0x5585546762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.208+0000 7f17f23d0740 1 --2- 172.21.15.98:0/2676425569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466b620 0x55855466ba00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.208+0000 7f17f23d0740 1 -- 172.21.15.98:0/2676425569 >> 172.21.15.98:0/2676425569 conn(0x558554667b30 msgr2=0x558554667f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.208+0000 7f17f23d0740 1 -- 172.21.15.98:0/2676425569 shutdown_connections 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.209+0000 7f17f23d0740 1 -- 172.21.15.98:0/2676425569 wait complete. 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.209+0000 7f17f23d0740 1 -- start start 2024-04-25T17:36:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.209+0000 7f17f23d0740 1 Processor -- start 2024-04-25T17:36:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 0x5585546761d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 0x5585546734c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 0x558554673ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17f23d0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558554679430 con 0x55855466bf40 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17f23d0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55855456b0b0 con 0x558554676800 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17f23d0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5585546795b0 con 0x55855466b620 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17ecb83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 0x5585546734c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17ecb83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 0x5585546734c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58774/0 (socket says 172.21.15.98:58774) 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17edb85640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 0x558554673ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17ecb83640 1 -- 172.21.15.98:0/704388097 learned_addr learned my addr 172.21.15.98:0/704388097 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17f23d0740 10 client.0 _ll_register_callbacks cb 0x558554558d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17edb85640 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 0x5585546761d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17edb85640 1 -- 172.21.15.98:0/704388097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 msgr2=0x5585546761d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17edb85640 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 0x5585546761d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17edb85640 1 -- 172.21.15.98:0/704388097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 msgr2=0x5585546734c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:36.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17edb85640 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 0x5585546734c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17edb85640 1 -- 172.21.15.98:0/704388097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585543af620 con 0x558554676800 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17ecb83640 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 0x5585546734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17edb85640 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 0x558554673ad0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f17e00093f0 tx=0x7f17e0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=633efa566a500ce7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.210+0000 7f17d0ff9640 1 -- 172.21.15.98:0/704388097 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0018030 con 0x558554676800 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17d0ff9640 1 -- 172.21.15.98:0/704388097 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f17e001e430 con 0x558554676800 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17f23d0740 1 -- 172.21.15.98:0/704388097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e0014400 con 0x558554676800 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17d0ff9640 1 -- 172.21.15.98:0/704388097 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f17e001e630 con 0x558554676800 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17f23d0740 10 client.24995 fetch_fsmap learned FSMap version 27 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17f23d0740 1 -- 172.21.15.98:0/704388097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558554685170 con 0x558554676800 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17d0ff9640 1 -- 172.21.15.98:0/704388097 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e0014400 con 0x558554676800 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17d0ff9640 1 -- 172.21.15.98:0/704388097 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7f17e001c3b0 con 0x558554676800 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17f23d0740 10 client.24995 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17f23d0740 10 client.24995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:36:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17f23d0740 20 client.24995 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17f23d0740 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558554686630 0x558554688a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17f23d0740 4 client.24995 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17f23d0740 1 -- 172.21.15.98:0/704388097 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585545de390 con 0x558554686630 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.211+0000 7f17ecb83640 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558554686630 0x558554688a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.212+0000 7f17d0ff9640 10 client.24995 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.212+0000 7f17ecb83640 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558554686630 0x558554688a20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=732f66be8d48ecf6 server_cookie=b5442fc04a281645 in_seq=0 out_seq=0 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17d0ff9640 1 -- 172.21.15.98:0/704388097 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5585545de390 con 0x558554686630 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17d0ff9640 10 client.24995 handle_command_reply: tid=0 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17f23d0740 1 client.24995 shutdown 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17f23d0740 1 -- 172.21.15.98:0/704388097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 msgr2=0x558554673ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17f23d0740 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 0x558554673ad0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f17e00093f0 tx=0x7f17e0004a40 comp rx=0 tx=0).stop 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17f23d0740 1 -- 172.21.15.98:0/704388097 shutdown_connections 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17f23d0740 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558554686630 0x558554688a20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17f23d0740 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 0x558554673ad0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17f23d0740 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466bf40 0x5585546734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17f23d0740 1 --2- 172.21.15.98:0/704388097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 0x5585546761d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17d0ff9640 0 client.24995 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.213+0000 7f17f23d0740 1 -- 172.21.15.98:0/704388097 >> 172.21.15.98:0/704388097 conn(0x558554667b30 msgr2=0x55855466d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.214+0000 7f17f23d0740 1 -- 172.21.15.98:0/704388097 shutdown_connections 2024-04-25T17:36:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.214+0000 7f17f23d0740 1 -- 172.21.15.98:0/704388097 wait complete. 2024-04-25T17:36:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.214+0000 7f17f23d0740 20 client.24995 trim_cache size 0 max 16384 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17f23d0740 1 Processor -- start 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17f23d0740 1 -- start start 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466b620 0x558554675d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466bf40 0x55855468e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 0x55855468e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17f23d0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81b6830 con 0x55855466b620 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17f23d0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81b4100 con 0x558554676800 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17f23d0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81b1f50 con 0x55855466bf40 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17edb85640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466b620 0x558554675d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17ecb83640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466bf40 0x55855468e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17edb85640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466b620 0x558554675d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58782/0 (socket says 172.21.15.98:58782) 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17edb85640 1 -- 172.21.15.98:0/2638024752 learned_addr learned my addr 172.21.15.98:0/2638024752 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17edb85640 1 --2- 172.21.15.98:0/2638024752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 0x55855468e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17ecb83640 1 -- 172.21.15.98:0/2638024752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 msgr2=0x55855468e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17ecb83640 1 --2- 172.21.15.98:0/2638024752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 0x55855468e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17ecb83640 1 -- 172.21.15.98:0/2638024752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466b620 msgr2=0x558554675d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17ecb83640 1 --2- 172.21.15.98:0/2638024752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466b620 0x558554675d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17ecb83640 1 -- 172.21.15.98:0/2638024752 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55855466b3e0 con 0x55855466bf40 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.233+0000 7f17edb85640 1 --2- 172.21.15.98:0/2638024752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 0x55855468e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.234+0000 7f17ecb83640 1 --2- 172.21.15.98:0/2638024752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466bf40 0x55855468e240 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x558554689180 tx=0x7f17d8009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78472a582f3f4ab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.234+0000 7f17d2ffd640 1 -- 172.21.15.98:0/2638024752 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8025070 con 0x55855466bf40 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.234+0000 7f17d2ffd640 1 -- 172.21.15.98:0/2638024752 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d8012d00 con 0x55855466bf40 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.234+0000 7f17d2ffd640 1 -- 172.21.15.98:0/2638024752 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d801e750 con 0x55855466bf40 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.234+0000 7f17f23d0740 1 -- 172.21.15.98:0/2638024752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466bf40 msgr2=0x55855468e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.235+0000 7f17f23d0740 1 --2- 172.21.15.98:0/2638024752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466bf40 0x55855468e240 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x558554689180 tx=0x7f17d8009190 comp rx=0 tx=0).stop 2024-04-25T17:36:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.235+0000 7f17f23d0740 1 -- 172.21.15.98:0/2638024752 shutdown_connections 2024-04-25T17:36:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.235+0000 7f17f23d0740 1 --2- 172.21.15.98:0/2638024752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558554676800 0x55855468e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.235+0000 7f17f23d0740 1 --2- 172.21.15.98:0/2638024752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466bf40 0x55855468e240 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.235+0000 7f17f23d0740 1 --2- 172.21.15.98:0/2638024752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55855466b620 0x558554675d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.235+0000 7f17f23d0740 1 -- 172.21.15.98:0/2638024752 >> 172.21.15.98:0/2638024752 conn(0x558554667b30 msgr2=0x558554673480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.235+0000 7f17f23d0740 1 -- 172.21.15.98:0/2638024752 shutdown_connections 2024-04-25T17:36:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.235+0000 7f17f23d0740 1 -- 172.21.15.98:0/2638024752 wait complete. 2024-04-25T17:36:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.235+0000 7f17f23d0740 1 -- start start 2024-04-25T17:36:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17f23d0740 1 Processor -- start 2024-04-25T17:36:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 0x55855467e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466bf40 0x55855467ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17f23d0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558554676800 0x55855467b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17f23d0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81b25c0 con 0x558554676800 2024-04-25T17:36:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17f23d0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81b3a30 con 0x55855466bf40 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17f23d0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81b2c90 con 0x55855466b620 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17f23d0740 10 client.0 _ll_register_callbacks cb 0x7f17e0014850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17edb85640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 0x55855467e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17edb85640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 0x55855467e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:35028/0 (socket says 172.21.15.98:35028) 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17edb85640 1 -- 172.21.15.98:0/771704250 learned_addr learned my addr 172.21.15.98:0/771704250 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17edb85640 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558554676800 0x55855467b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17ecb83640 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466bf40 0x55855467ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17edb85640 1 -- 172.21.15.98:0/771704250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 msgr2=0x55855467e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17edb85640 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 0x55855467e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.236+0000 7f17edb85640 1 -- 172.21.15.98:0/771704250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466bf40 msgr2=0x55855467ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17edb85640 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466bf40 0x55855467ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17edb85640 1 -- 172.21.15.98:0/771704250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55855467c190 con 0x558554676800 2024-04-25T17:36:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17edb85640 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558554676800 0x55855467b9b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f17e0000f80 tx=0x7f17e0007040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe00aa3ac3513cbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17d0ff9640 1 -- 172.21.15.98:0/771704250 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e003d060 con 0x558554676800 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17d0ff9640 1 -- 172.21.15.98:0/771704250 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f17e000ea50 con 0x558554676800 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17f23d0740 1 -- 172.21.15.98:0/771704250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17d8012d00 con 0x558554676800 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17d0ff9640 1 -- 172.21.15.98:0/771704250 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f17e0018040 con 0x558554676800 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17f23d0740 10 client.15375 fetch_fsmap learned FSMap version 27 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17f23d0740 1 -- 172.21.15.98:0/771704250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5585546ec550 con 0x558554676800 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17d0ff9640 1 -- 172.21.15.98:0/771704250 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d8012d00 con 0x558554676800 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17d0ff9640 1 -- 172.21.15.98:0/771704250 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 27) ==== 8342+0+0 (secure 0 0 0) 0x7f17e0037040 con 0x558554676800 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17f23d0740 10 client.15375 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17f23d0740 10 client.15375 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17f23d0740 20 client.15375 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17f23d0740 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5585546ecc10 0x5585546ef000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:36.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17f23d0740 4 client.15375 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:36:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.237+0000 7f17f23d0740 1 -- 172.21.15.98:0/771704250 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5585545d8200 con 0x5585546ecc10 2024-04-25T17:36:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.238+0000 7f17ecb83640 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5585546ecc10 0x5585546ef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.238+0000 7f17ecb83640 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5585546ecc10 0x5585546ef000 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de27426e4e5b33ce server_cookie=955343a345b09c7a in_seq=0 out_seq=0 2024-04-25T17:36:36.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.238+0000 7f17d0ff9640 10 client.15375 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17d0ff9640 1 -- 172.21.15.98:0/771704250 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5585545d8200 con 0x5585546ecc10 2024-04-25T17:36:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17d0ff9640 10 client.15375 handle_command_reply: tid=0 2024-04-25T17:36:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 client.15375 shutdown 2024-04-25T17:36:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 -- 172.21.15.98:0/771704250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558554676800 msgr2=0x55855467b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558554676800 0x55855467b9b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f17e0000f80 tx=0x7f17e0007040 comp rx=0 tx=0).stop 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 -- 172.21.15.98:0/771704250 shutdown_connections 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5585546ecc10 0x5585546ef000 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558554676800 0x55855467b9b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55855466bf40 0x55855467ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 --2- 172.21.15.98:0/771704250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55855466b620 0x55855467e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 -- 172.21.15.98:0/771704250 >> 172.21.15.98:0/771704250 conn(0x558554667b30 msgr2=0x558554666420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17d0ff9640 0 client.15375 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 -- 172.21.15.98:0/771704250 shutdown_connections 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.343+0000 7f17f23d0740 1 -- 172.21.15.98:0/771704250 wait complete. 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17f23d0740 20 client.15375 trim_cache size 0 max 16384 2024-04-25T17:36:36.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:36:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:36:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:36:36.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 -- 172.21.15.98:0/1831774798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f17d40a3f60 msgr2=0x7f17d40a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f17d40a3f60 0x7f17d40a6420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f17e40ede40 tx=0x7f17e4005fd0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 -- 172.21.15.98:0/1831774798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e81010f0 msgr2=0x7f17e810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e81010f0 0x7f17e810d570 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f17e402c960 tx=0x7f17e40049e0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 -- 172.21.15.98:0/1831774798 shutdown_connections 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f17d40a3f60 0x7f17d40a6420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17e810b6b0 0x7f17e810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e8106170 0x7f17e810dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 --2- 172.21.15.98:0/1831774798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e81010f0 0x7f17e810d570 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 -- 172.21.15.98:0/1831774798 >> 172.21.15.98:0/1831774798 conn(0x7f17e8077fd0 msgr2=0x7f17e80fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 -- 172.21.15.98:0/1831774798 shutdown_connections 2024-04-25T17:36:36.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:36.344+0000 7f17ef5e8640 1 -- 172.21.15.98:0/1831774798 wait complete. 2024-04-25T17:36:36.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:36 smithi098 ceph-mon[29192]: pgmap v23: 97 pgs: 97 active+clean; 381 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 59 MiB/s wr, 240 op/s 2024-04-25T17:36:36.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:36 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:36:36.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:36 smithi203 ceph-mon[32834]: pgmap v23: 97 pgs: 97 active+clean; 381 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 59 MiB/s wr, 240 op/s 2024-04-25T17:36:36.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:36 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:36:36.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:36 smithi173 ceph-mon[32856]: pgmap v23: 97 pgs: 97 active+clean; 381 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 59 MiB/s wr, 240 op/s 2024-04-25T17:36:36.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:36 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:36:36.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 5.0 s, 3584.2 tps, lat 27.435 ms stddev 27.472 2024-04-25T17:36:38.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:38 smithi098 ceph-mon[29192]: pgmap v24: 97 pgs: 97 active+clean; 808 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 77 MiB/s wr, 381 op/s 2024-04-25T17:36:38.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:38 smithi173 ceph-mon[32856]: pgmap v24: 97 pgs: 97 active+clean; 808 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 77 MiB/s wr, 381 op/s 2024-04-25T17:36:38.523 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:38 smithi203 ceph-mon[32834]: pgmap v24: 97 pgs: 97 active+clean; 808 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 77 MiB/s wr, 381 op/s 2024-04-25T17:36:39.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:39 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:36:39.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:39 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:36:39.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:39 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:36:40.356 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:40 smithi098 ceph-mon[29192]: pgmap v25: 97 pgs: 97 active+clean; 808 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 74 MiB/s wr, 176 op/s 2024-04-25T17:36:40.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:40 smithi203 ceph-mon[32834]: pgmap v25: 97 pgs: 97 active+clean; 808 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 74 MiB/s wr, 176 op/s 2024-04-25T17:36:40.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:40 smithi173 ceph-mon[32856]: pgmap v25: 97 pgs: 97 active+clean; 808 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 74 MiB/s wr, 176 op/s 2024-04-25T17:36:41.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:41 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:36:41.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:41 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:36:41.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:41 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:36:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 10.0 s, 3352.0 tps, lat 29.721 ms stddev 30.930 2024-04-25T17:36:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:42 smithi098 ceph-mon[29192]: pgmap v26: 97 pgs: 97 active+clean; 919 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 78 MiB/s wr, 570 op/s 2024-04-25T17:36:42.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:42 smithi203 ceph-mon[32834]: pgmap v26: 97 pgs: 97 active+clean; 919 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 78 MiB/s wr, 570 op/s 2024-04-25T17:36:42.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:42 smithi173 ceph-mon[32856]: pgmap v26: 97 pgs: 97 active+clean; 919 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 78 MiB/s wr, 570 op/s 2024-04-25T17:36:44.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:44 smithi098 ceph-mon[29192]: pgmap v27: 97 pgs: 97 active+clean; 919 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 46 MiB/s wr, 552 op/s 2024-04-25T17:36:44.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:44 smithi203 ceph-mon[32834]: pgmap v27: 97 pgs: 97 active+clean; 919 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 46 MiB/s wr, 552 op/s 2024-04-25T17:36:44.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:44 smithi173 ceph-mon[32856]: pgmap v27: 97 pgs: 97 active+clean; 919 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 46 MiB/s wr, 552 op/s 2024-04-25T17:36:45.136 INFO:tasks.quiescer.fs.[cephfs]:Total duration (55.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-25T17:36:45.136 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.55 2024-04-25T17:36:45.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:36:45.488 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:46.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:46 smithi098 ceph-mon[29192]: pgmap v28: 97 pgs: 97 active+clean; 919 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 23 MiB/s wr, 534 op/s 2024-04-25T17:36:46.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:46 smithi203 ceph-mon[32834]: pgmap v28: 97 pgs: 97 active+clean; 919 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 23 MiB/s wr, 534 op/s 2024-04-25T17:36:46.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:46 smithi173 ceph-mon[32856]: pgmap v28: 97 pgs: 97 active+clean; 919 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 23 MiB/s wr, 534 op/s 2024-04-25T17:36:46.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 15.0 s, 3621.0 tps, lat 27.695 ms stddev 29.131 2024-04-25T17:36:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.504+0000 7f274255b700 1 -- 172.21.15.98:0/3121942477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f273c11c220 msgr2=0x7f273c11e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.504+0000 7f274255b700 1 --2- 172.21.15.98:0/3121942477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f273c11c220 0x7f273c11e610 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2730009990 tx=0x7f2730009ca0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.504+0000 7f274255b700 1 -- 172.21.15.98:0/3121942477 shutdown_connections 2024-04-25T17:36:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.504+0000 7f274255b700 1 --2- 172.21.15.98:0/3121942477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f273c111220 0x7f273c109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.504+0000 7f274255b700 1 --2- 172.21.15.98:0/3121942477 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f273c11c220 0x7f273c11e610 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.504+0000 7f274255b700 1 --2- 172.21.15.98:0/3121942477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f273c109f80 0x7f273c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.504+0000 7f274255b700 1 -- 172.21.15.98:0/3121942477 >> 172.21.15.98:0/3121942477 conn(0x7f273c06dbb0 msgr2=0x7f273c06dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.504+0000 7f274255b700 1 -- 172.21.15.98:0/3121942477 shutdown_connections 2024-04-25T17:36:47.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.504+0000 7f274255b700 1 -- 172.21.15.98:0/3121942477 wait complete. 2024-04-25T17:36:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.505+0000 7f274255b700 1 Processor -- start 2024-04-25T17:36:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.505+0000 7f274255b700 1 -- start start 2024-04-25T17:36:47.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.505+0000 7f274255b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f273c109f80 0x7f273c1a7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.505+0000 7f274255b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f273c111220 0x7f273c1a8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f274255b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f273c11c220 0x7f273c1ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f274255b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f273c121a60 con 0x7f273c109f80 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f274255b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f273c1218e0 con 0x7f273c111220 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f274255b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f273c121be0 con 0x7f273c11c220 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f273b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f273c111220 0x7f273c1a8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f2740af8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f273c11c220 0x7f273c1ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f273b7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f273c111220 0x7f273c1a8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:34398/0 (socket says 172.21.15.98:34398) 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f273b7fe700 1 -- 172.21.15.98:0/4024095051 learned_addr learned my addr 172.21.15.98:0/4024095051 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f2740af8700 1 -- 172.21.15.98:0/4024095051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f273c111220 msgr2=0x7f273c1a8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f2740af8700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f273c111220 0x7f273c1a8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f2740af8700 1 -- 172.21.15.98:0/4024095051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f273c109f80 msgr2=0x7f273c1a7f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:36:47.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f2740af8700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f273c109f80 0x7f273c1a7f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.506+0000 7f2740af8700 1 -- 172.21.15.98:0/4024095051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273c1acf20 con 0x7f273c11c220 2024-04-25T17:36:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.507+0000 7f273b7fe700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f273c111220 0x7f273c1a8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:36:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.507+0000 7f2740af8700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f273c11c220 0x7f273c1ac7e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2730005b40 tx=0x7f273000bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:47.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.507+0000 7f27397fa700 1 -- 172.21.15.98:0/4024095051 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273000fa90 con 0x7f273c11c220 2024-04-25T17:36:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.507+0000 7f27397fa700 1 -- 172.21.15.98:0/4024095051 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2730004ce0 con 0x7f273c11c220 2024-04-25T17:36:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.508+0000 7f274255b700 1 -- 172.21.15.98:0/4024095051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f273c1ad1b0 con 0x7f273c11c220 2024-04-25T17:36:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.508+0000 7f274255b700 1 -- 172.21.15.98:0/4024095051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f273c1ad600 con 0x7f273c11c220 2024-04-25T17:36:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.508+0000 7f27397fa700 1 -- 172.21.15.98:0/4024095051 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27300054f0 con 0x7f273c11c220 2024-04-25T17:36:47.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.509+0000 7f274255b700 1 -- 172.21.15.98:0/4024095051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f273c069730 con 0x7f273c11c220 2024-04-25T17:36:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.511+0000 7f27397fa700 1 -- 172.21.15.98:0/4024095051 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f2730028060 con 0x7f273c11c220 2024-04-25T17:36:47.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.511+0000 7f27397fa700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27240a4020 0x7f27240a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.512+0000 7f27397fa700 1 -- 172.21.15.98:0/4024095051 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f27300d14a0 con 0x7f273c11c220 2024-04-25T17:36:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.512+0000 7f273bfff700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27240a4020 0x7f27240a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:47.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.512+0000 7f273bfff700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27240a4020 0x7f27240a64e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f272c009a70 tx=0x7f272c009260 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:47.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.518+0000 7f27397fa700 1 -- 172.21.15.98:0/4024095051 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2730013070 con 0x7f273c11c220 2024-04-25T17:36:47.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.874+0000 7f274255b700 1 -- 172.21.15.98:0/4024095051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f273c1b4bb0 con 0x7f273c11c220 2024-04-25T17:36:47.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.877+0000 7f27397fa700 1 -- 172.21.15.98:0/4024095051 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11881 (secure 0 0 0) 0x7f273009d950 con 0x7f273c11c220 2024-04-25T17:36:47.878 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:36:47.878 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:36:39.350510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:36:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.885+0000 7f274255b700 1 -- 172.21.15.98:0/4024095051 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27240a4020 msgr2=0x7f27240a64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.885+0000 7f274255b700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27240a4020 0x7f27240a64e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f272c009a70 tx=0x7f272c009260 comp rx=0 tx=0).stop 2024-04-25T17:36:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.885+0000 7f274255b700 1 -- 172.21.15.98:0/4024095051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f273c11c220 msgr2=0x7f273c1ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:47.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.885+0000 7f274255b700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f273c11c220 0x7f273c1ac7e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2730005b40 tx=0x7f273000bfd0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.888+0000 7f274255b700 1 -- 172.21.15.98:0/4024095051 shutdown_connections 2024-04-25T17:36:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.888+0000 7f274255b700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27240a4020 0x7f27240a64e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.888+0000 7f274255b700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f273c11c220 0x7f273c1ac7e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.888+0000 7f274255b700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f273c109f80 0x7f273c1a7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.888+0000 7f274255b700 1 --2- 172.21.15.98:0/4024095051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f273c111220 0x7f273c1a8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:47.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.888+0000 7f274255b700 1 -- 172.21.15.98:0/4024095051 >> 172.21.15.98:0/4024095051 conn(0x7f273c06dbb0 msgr2=0x7f273c11a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:47.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.889+0000 7f274255b700 1 -- 172.21.15.98:0/4024095051 shutdown_connections 2024-04-25T17:36:47.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:47.890+0000 7f274255b700 1 -- 172.21.15.98:0/4024095051 wait complete. 2024-04-25T17:36:47.892 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-04-25T17:36:48.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:48 smithi098 ceph-mon[29192]: pgmap v29: 97 pgs: 97 active+clean; 930 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 26 MiB/s wr, 849 op/s 2024-04-25T17:36:48.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:48 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4024095051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:48.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:48 smithi203 ceph-mon[32834]: pgmap v29: 97 pgs: 97 active+clean; 930 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 26 MiB/s wr, 849 op/s 2024-04-25T17:36:48.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:48 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/4024095051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:48.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:48 smithi173 ceph-mon[32856]: pgmap v29: 97 pgs: 97 active+clean; 930 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 26 MiB/s wr, 849 op/s 2024-04-25T17:36:48.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:48 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/4024095051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:36:48.693 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 87.55 --await-for 42.0' 2024-04-25T17:36:48.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 87.55 --await-for 42.0 2024-04-25T17:36:49.029 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.466+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/385940756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c4100450 msgr2=0x7fb4c41008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.466+0000 7fb4cc5f9640 1 --2- 172.21.15.98:0/385940756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c4100450 0x7fb4c41008d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009930 tx=0x7fb4b402a440 comp rx=0 tx=0).stop 2024-04-25T17:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.466+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/385940756 shutdown_connections 2024-04-25T17:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.466+0000 7fb4cc5f9640 1 --2- 172.21.15.98:0/385940756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c4100e10 0x7fb4c410ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.466+0000 7fb4cc5f9640 1 --2- 172.21.15.98:0/385940756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c4100450 0x7fb4c41008d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.466+0000 7fb4cc5f9640 1 --2- 172.21.15.98:0/385940756 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c4106f30 0x7fb4c4107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.466+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/385940756 >> 172.21.15.98:0/385940756 conn(0x7fb4c40773e0 msgr2=0x7fb4c40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.466+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/385940756 shutdown_connections 2024-04-25T17:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.466+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/385940756 wait complete. 2024-04-25T17:36:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.466+0000 7fb4cc5f9640 1 Processor -- start 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4cc5f9640 1 -- start start 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4cc5f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c4100450 0x7fb4c419e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4cc5f9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c4100e10 0x7fb4c419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4cc5f9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c4106f30 0x7fb4c4198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4cc5f9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c4112360 con 0x7fb4c4100450 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4cc5f9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c41121e0 con 0x7fb4c4106f30 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4cc5f9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c41124e0 con 0x7fb4c4100e10 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4ca395640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c4100450 0x7fb4c419e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4c9b94640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c4100e10 0x7fb4c419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4cab96640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c4106f30 0x7fb4c4198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4ca395640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c4100450 0x7fb4c419e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59914/0 (socket says 172.21.15.98:59914) 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4ca395640 1 -- 172.21.15.98:0/1614784813 learned_addr learned my addr 172.21.15.98:0/1614784813 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4c9b94640 1 -- 172.21.15.98:0/1614784813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c4106f30 msgr2=0x7fb4c4198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c4106f30 0x7fb4c4198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4c9b94640 1 -- 172.21.15.98:0/1614784813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c4100450 msgr2=0x7fb4c419e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c4100450 0x7fb4c419e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4c9b94640 1 -- 172.21.15.98:0/1614784813 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c4198de0 con 0x7fb4c4100e10 2024-04-25T17:36:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.467+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c4100e10 0x7fb4c419e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb4b402a960 tx=0x7fb4b4003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.468+0000 7fb4bb7fe640 1 -- 172.21.15.98:0/1614784813 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b403f070 con 0x7fb4c4100e10 2024-04-25T17:36:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.468+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/1614784813 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c4199070 con 0x7fb4c4100e10 2024-04-25T17:36:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.468+0000 7fb4bb7fe640 1 -- 172.21.15.98:0/1614784813 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4b4031b60 con 0x7fb4c4100e10 2024-04-25T17:36:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.468+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/1614784813 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c41a5320 con 0x7fb4c4100e10 2024-04-25T17:36:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.468+0000 7fb4bb7fe640 1 -- 172.21.15.98:0/1614784813 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b4035030 con 0x7fb4c4100e10 2024-04-25T17:36:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.471+0000 7fb4bb7fe640 1 -- 172.21.15.98:0/1614784813 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fb4b4042070 con 0x7fb4c4100e10 2024-04-25T17:36:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.472+0000 7fb4bb7fe640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb4b00a3f60 0x7fb4b00a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.472+0000 7fb4ca395640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb4b00a3f60 0x7fb4b00a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.472+0000 7fb4bb7fe640 1 -- 172.21.15.98:0/1614784813 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fb4b40f1360 con 0x7fb4c4100e10 2024-04-25T17:36:49.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.472+0000 7fb4cf3e0740 1 Processor -- start 2024-04-25T17:36:49.474 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.472+0000 7fb4ca395640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb4b00a3f60 0x7fb4b00a6420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009a30 tx=0x7fb4c0009190 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:49.474 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.472+0000 7fb4cf3e0740 1 -- start start 2024-04-25T17:36:49.474 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.473+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7e4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.473+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4f40 0x5595ef7ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.473+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 0x5595ef7f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.473+0000 7fb4cf3e0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5595ef749970 con 0x5595ef7ef800 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.473+0000 7fb4cf3e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5595ef6e64a0 con 0x5595ef7e4f40 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.473+0000 7fb4cf3e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5595ef745930 con 0x5595ef7e4620 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.473+0000 7fb4ca395640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7e4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.473+0000 7fb4ca395640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7e4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:45906/0 (socket says 172.21.15.98:45906) 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.473+0000 7fb4ca395640 1 -- 172.21.15.98:0/2495845849 learned_addr learned my addr 172.21.15.98:0/2495845849 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.473+0000 7fb4cab96640 1 --2- 172.21.15.98:0/2495845849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 0x5595ef7f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.474+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/2495845849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4f40 0x5595ef7ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.474+0000 7fb4ca395640 1 -- 172.21.15.98:0/2495845849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4f40 msgr2=0x5595ef7ef2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.474+0000 7fb4ca395640 1 --2- 172.21.15.98:0/2495845849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4f40 0x5595ef7ef2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.474+0000 7fb4ca395640 1 -- 172.21.15.98:0/2495845849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 msgr2=0x5595ef7f1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.474+0000 7fb4ca395640 1 --2- 172.21.15.98:0/2495845849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 0x5595ef7f1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.474+0000 7fb4ca395640 1 -- 172.21.15.98:0/2495845849 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595ef45c210 con 0x5595ef7e4620 2024-04-25T17:36:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.474+0000 7fb4ca395640 1 --2- 172.21.15.98:0/2495845849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7e4a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009fd0 tx=0x7fb4c0011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2de736b415d49e6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.475+0000 7fb4b8bfe640 1 -- 172.21.15.98:0/2495845849 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c001b040 con 0x5595ef7e4620 2024-04-25T17:36:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.475+0000 7fb4b8bfe640 1 -- 172.21.15.98:0/2495845849 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4c0024070 con 0x5595ef7e4620 2024-04-25T17:36:49.477 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.475+0000 7fb4b8bfe640 1 -- 172.21.15.98:0/2495845849 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c001f3f0 con 0x5595ef7e4620 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/2495845849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 msgr2=0x5595ef7e4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/2495845849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7e4a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009fd0 tx=0x7fb4c0011000 comp rx=0 tx=0).stop 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/2495845849 shutdown_connections 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/2495845849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 0x5595ef7f1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/2495845849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4f40 0x5595ef7ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/2495845849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7e4a00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/2495845849 >> 172.21.15.98:0/2495845849 conn(0x5595ef7e0b30 msgr2=0x5595ef7e0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/2495845849 shutdown_connections 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/2495845849 wait complete. 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 -- start start 2024-04-25T17:36:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.477+0000 7fb4cf3e0740 1 Processor -- start 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4620 0x5595ef7ef1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 0x5595ef7ec490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7ef800 0x5595ef7eca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4ca395640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4620 0x5595ef7ef1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4ca395640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4620 0x5595ef7ef1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:34436/0 (socket says 172.21.15.98:34436) 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4c9b94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 0x5595ef7ec490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4ca395640 1 -- 172.21.15.98:0/3373960325 learned_addr learned my addr 172.21.15.98:0/3373960325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5595ef7f2430 con 0x5595ef7e4f40 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5595ef6e40b0 con 0x5595ef7e4620 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5595ef7f25b0 con 0x5595ef7ef800 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4cf3e0740 10 client.0 _ll_register_callbacks cb 0x5595ef6d1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.478+0000 7fb4cab96640 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7ef800 0x5595ef7eca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.479+0000 7fb4ca395640 1 -- 172.21.15.98:0/3373960325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7ef800 msgr2=0x5595ef7eca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:49.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.479+0000 7fb4ca395640 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7ef800 0x5595ef7eca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.479+0000 7fb4ca395640 1 -- 172.21.15.98:0/3373960325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 msgr2=0x5595ef7ec490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.479+0000 7fb4ca395640 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 0x5595ef7ec490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.479+0000 7fb4ca395640 1 -- 172.21.15.98:0/3373960325 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595ef528620 con 0x5595ef7e4620 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.479+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 0x5595ef7ec490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.480+0000 7fb4ca395640 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4620 0x5595ef7ef1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0013ac0 tx=0x7fb4c000ea20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=496720927d7bb3bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.480+0000 7fb4adffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.480+0000 7fb4adffb640 1 -- 172.21.15.98:0/3373960325 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c002d070 con 0x5595ef7e4620 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.480+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4c0024070 con 0x5595ef7e4620 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.480+0000 7fb4adffb640 1 -- 172.21.15.98:0/3373960325 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fb4c0013650 con 0x5595ef7e4620 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.480+0000 7fb4adffb640 1 -- 172.21.15.98:0/3373960325 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb4c0013850 con 0x5595ef7e4620 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.480+0000 7fb4cf3e0740 10 client.25016 fetch_fsmap learned FSMap version 29 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.480+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5595ef802790 con 0x5595ef7e4620 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.480+0000 7fb4adffb640 1 -- 172.21.15.98:0/3373960325 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4c0024070 con 0x5595ef7e4620 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.480+0000 7fb4adffb640 1 -- 172.21.15.98:0/3373960325 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fb4c001e030 con 0x5595ef7e4620 2024-04-25T17:36:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.481+0000 7fb4cf3e0740 10 client.25016 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:36:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.481+0000 7fb4cf3e0740 10 client.25016 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:36:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.481+0000 7fb4cf3e0740 20 client.25016 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.481+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5595ef803ba0 0x5595ef805f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.481+0000 7fb4cf3e0740 4 client.25016 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:36:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.481+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595ef757390 con 0x5595ef803ba0 2024-04-25T17:36:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.481+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5595ef803ba0 0x5595ef805f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.482+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5595ef803ba0 0x5595ef805f90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9ac0c30f5fc08ee server_cookie=323e45fb63d0a409 in_seq=0 out_seq=0 2024-04-25T17:36:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.482+0000 7fb4adffb640 10 client.25016 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.483+0000 7fb4adffb640 1 -- 172.21.15.98:0/3373960325 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595ef757390 con 0x5595ef803ba0 2024-04-25T17:36:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.483+0000 7fb4adffb640 10 client.25016 handle_command_reply: tid=0 2024-04-25T17:36:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 client.25016 shutdown 2024-04-25T17:36:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4620 msgr2=0x5595ef7ef1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4620 0x5595ef7ef1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0013ac0 tx=0x7fb4c000ea20 comp rx=0 tx=0).stop 2024-04-25T17:36:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 shutdown_connections 2024-04-25T17:36:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5595ef803ba0 0x5595ef805f90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4adffb640 0 client.25016 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7ef800 0x5595ef7eca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 0x5595ef7ec490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/3373960325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4620 0x5595ef7ef1a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 >> 172.21.15.98:0/3373960325 conn(0x5595ef7e0b30 msgr2=0x5595ef7e6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 shutdown_connections 2024-04-25T17:36:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/3373960325 wait complete. 2024-04-25T17:36:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.484+0000 7fb4cf3e0740 20 client.25016 trim_cache size 0 max 16384 2024-04-25T17:36:49.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.503+0000 7fb4cf3e0740 1 Processor -- start 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cf3e0740 1 -- start start 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4f40 0x5595ef7e7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 0x5595ef7eecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cf3e0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c41ae230 con 0x5595ef7ef800 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cf3e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c41abad0 con 0x5595ef7e4f40 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cf3e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c41a9920 con 0x5595ef7e4620 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cab96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 0x5595ef7eecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cab96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 0x5595ef7eecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59934/0 (socket says 172.21.15.98:59934) 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cab96640 1 -- 172.21.15.98:0/4056233263 learned_addr learned my addr 172.21.15.98:0/4056233263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4ca395640 1 --2- 172.21.15.98:0/4056233263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/4056233263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4f40 0x5595ef7e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cab96640 1 -- 172.21.15.98:0/4056233263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 msgr2=0x5595ef7fb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cab96640 1 --2- 172.21.15.98:0/4056233263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7fb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cab96640 1 -- 172.21.15.98:0/4056233263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4f40 msgr2=0x5595ef7e7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cab96640 1 --2- 172.21.15.98:0/4056233263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4f40 0x5595ef7e7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cab96640 1 -- 172.21.15.98:0/4056233263 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595ef7eaa20 con 0x5595ef7ef800 2024-04-25T17:36:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.504+0000 7fb4cab96640 1 --2- 172.21.15.98:0/4056233263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 0x5595ef7eecd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc003b30 tx=0x7fb4bc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61fd24f4dd4f605 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.505+0000 7fb4affff640 1 -- 172.21.15.98:0/4056233263 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc020070 con 0x5595ef7ef800 2024-04-25T17:36:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.505+0000 7fb4affff640 1 -- 172.21.15.98:0/4056233263 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4bc0108e0 con 0x5595ef7ef800 2024-04-25T17:36:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.505+0000 7fb4affff640 1 -- 172.21.15.98:0/4056233263 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc01b920 con 0x5595ef7ef800 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.506+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4056233263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 msgr2=0x5595ef7eecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.506+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/4056233263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 0x5595ef7eecd0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc003b30 tx=0x7fb4bc004040 comp rx=0 tx=0).stop 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.506+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4056233263 shutdown_connections 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.506+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/4056233263 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7ef800 0x5595ef7eecd0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.506+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/4056233263 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7e4f40 0x5595ef7e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.506+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/4056233263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7fb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.506+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4056233263 >> 172.21.15.98:0/4056233263 conn(0x5595ef7e0b30 msgr2=0x5595ef7e7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.506+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4056233263 shutdown_connections 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.507+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4056233263 wait complete. 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.507+0000 7fb4cf3e0740 1 -- start start 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4cf3e0740 1 Processor -- start 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7f7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 0x5595ef7e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4cf3e0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7ef800 0x5595ef7e9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4cf3e0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c41a9f90 con 0x5595ef7e4f40 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4ca395640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7f7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4cf3e0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c41ab400 con 0x5595ef7ef800 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4cf3e0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c41aa660 con 0x5595ef7e4620 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4ca395640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7f7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:45934/0 (socket says 172.21.15.98:45934) 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4ca395640 1 -- 172.21.15.98:0/4225000614 learned_addr learned my addr 172.21.15.98:0/4225000614 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.508+0000 7fb4cf3e0740 10 client.0 _ll_register_callbacks cb 0x7fb4c0009730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 0x5595ef7e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4ca395640 1 -- 172.21.15.98:0/4225000614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7ef800 msgr2=0x5595ef7e9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4ca395640 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7ef800 0x5595ef7e9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4ca395640 1 -- 172.21.15.98:0/4225000614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 msgr2=0x5595ef7e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4ca395640 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 0x5595ef7e8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4ca395640 1 -- 172.21.15.98:0/4225000614 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595ef7e9920 con 0x5595ef7e4620 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 0x5595ef7e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:36:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4ca395640 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7f7a60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000e5a0 tx=0x7fb4c0009200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6cb091c9c5df69c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4adffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4adffb640 1 -- 172.21.15.98:0/4225000614 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c003f070 con 0x5595ef7e4620 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4adffb640 1 -- 172.21.15.98:0/4225000614 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fb4c0004220 con 0x5595ef7e4620 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.509+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4225000614 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4bc0108e0 con 0x5595ef7e4620 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4adffb640 1 -- 172.21.15.98:0/4225000614 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb4c002d040 con 0x5595ef7e4620 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4cf3e0740 10 client.24976 fetch_fsmap learned FSMap version 29 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4225000614 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5595ef869c90 con 0x5595ef7e4620 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4adffb640 1 -- 172.21.15.98:0/4225000614 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4bc0108e0 con 0x5595ef7e4620 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4adffb640 1 -- 172.21.15.98:0/4225000614 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fb4c001e040 con 0x5595ef7e4620 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4cf3e0740 10 client.24976 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4cf3e0740 10 client.24976 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4cf3e0740 20 client.24976 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5595ef86a410 0x5595ef86c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4cf3e0740 4 client.24976 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:36:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4225000614 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5595ef751200 con 0x5595ef86a410 2024-04-25T17:36:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.510+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5595ef86a410 0x5595ef86c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.512+0000 7fb4adffb640 10 client.24976 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.512+0000 7fb4c9b94640 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5595ef86a410 0x5595ef86c870 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a1e04d5a928ad5e server_cookie=d9c022bf435579b4 in_seq=0 out_seq=0 2024-04-25T17:36:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4adffb640 1 -- 172.21.15.98:0/4225000614 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5595ef751200 con 0x5595ef86a410 2024-04-25T17:36:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4adffb640 10 client.24976 handle_command_reply: tid=0 2024-04-25T17:36:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 client.24976 shutdown 2024-04-25T17:36:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4225000614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 msgr2=0x5595ef7f7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7f7a60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000e5a0 tx=0x7fb4c0009200 comp rx=0 tx=0).stop 2024-04-25T17:36:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4225000614 shutdown_connections 2024-04-25T17:36:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5595ef86a410 0x5595ef86c870 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5595ef7ef800 0x5595ef7e9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4adffb640 0 client.24976 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5595ef7e4f40 0x5595ef7e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 --2- 172.21.15.98:0/4225000614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5595ef7e4620 0x5595ef7f7a60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4225000614 >> 172.21.15.98:0/4225000614 conn(0x5595ef7e0b30 msgr2=0x5595ef7dfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4225000614 shutdown_connections 2024-04-25T17:36:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.526+0000 7fb4cf3e0740 1 -- 172.21.15.98:0/4225000614 wait complete. 2024-04-25T17:36:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.527+0000 7fb4cf3e0740 20 client.24976 trim_cache size 0 max 16384 2024-04-25T17:36:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:36:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:36:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:36:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.528+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/1614784813 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb4b00a3f60 msgr2=0x7fb4b00a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.528+0000 7fb4cc5f9640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb4b00a3f60 0x7fb4b00a6420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009a30 tx=0x7fb4c0009190 comp rx=0 tx=0).stop 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.528+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/1614784813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c4100e10 msgr2=0x7fb4c419e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.528+0000 7fb4cc5f9640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c4100e10 0x7fb4c419e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb4b402a960 tx=0x7fb4b4003c10 comp rx=0 tx=0).stop 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.529+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/1614784813 shutdown_connections 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.529+0000 7fb4cc5f9640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb4b00a3f60 0x7fb4b00a6420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.529+0000 7fb4cc5f9640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4c4106f30 0x7fb4c4198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.529+0000 7fb4cc5f9640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4c4100e10 0x7fb4c419e9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.529+0000 7fb4cc5f9640 1 --2- 172.21.15.98:0/1614784813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4c4100450 0x7fb4c419e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.529+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/1614784813 >> 172.21.15.98:0/1614784813 conn(0x7fb4c40773e0 msgr2=0x7fb4c40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.529+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/1614784813 shutdown_connections 2024-04-25T17:36:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:36:49.529+0000 7fb4cc5f9640 1 -- 172.21.15.98:0/1614784813 wait complete. 2024-04-25T17:36:50.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:50 smithi098 ceph-mon[29192]: pgmap v30: 97 pgs: 97 active+clean; 930 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 7.8 MiB/s wr, 708 op/s 2024-04-25T17:36:50.324 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:50 smithi173 ceph-mon[32856]: pgmap v30: 97 pgs: 97 active+clean; 930 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 7.8 MiB/s wr, 708 op/s 2024-04-25T17:36:50.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:50 smithi203 ceph-mon[32834]: pgmap v30: 97 pgs: 97 active+clean; 930 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 7.8 MiB/s wr, 708 op/s 2024-04-25T17:36:51.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 20.0 s, 3859.4 tps, lat 25.953 ms stddev 29.443 2024-04-25T17:36:52.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:52 smithi098 ceph-mon[29192]: pgmap v31: 97 pgs: 97 active+clean; 953 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 13 MiB/s wr, 1.05k op/s 2024-04-25T17:36:52.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:52 smithi203 ceph-mon[32834]: pgmap v31: 97 pgs: 97 active+clean; 953 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 13 MiB/s wr, 1.05k op/s 2024-04-25T17:36:52.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:52 smithi173 ceph-mon[32856]: pgmap v31: 97 pgs: 97 active+clean; 953 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 13 MiB/s wr, 1.05k op/s 2024-04-25T17:36:53.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.015+0000 7f73d2777700 1 -- 172.21.15.98:0/1509829197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73cc10f470 msgr2=0x7f73cc111860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:53.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.015+0000 7f73d2777700 1 --2- 172.21.15.98:0/1509829197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73cc10f470 0x7f73cc111860 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f73bc009990 tx=0x7f73bc009ca0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.016+0000 7f73d2777700 1 -- 172.21.15.98:0/1509829197 shutdown_connections 2024-04-25T17:36:53.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.016+0000 7f73d2777700 1 --2- 172.21.15.98:0/1509829197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73cc10f470 0x7f73cc111860 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.016+0000 7f73d2777700 1 --2- 172.21.15.98:0/1509829197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73cc074c70 0x7f73cc075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.016+0000 7f73d2777700 1 --2- 172.21.15.98:0/1509829197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73cc075590 0x7f73cc10ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.016+0000 7f73d2777700 1 -- 172.21.15.98:0/1509829197 >> 172.21.15.98:0/1509829197 conn(0x7f73cc079bf0 msgr2=0x7f73cc07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:53.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.016+0000 7f73d2777700 1 -- 172.21.15.98:0/1509829197 shutdown_connections 2024-04-25T17:36:53.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.016+0000 7f73d2777700 1 -- 172.21.15.98:0/1509829197 wait complete. 2024-04-25T17:36:53.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.016+0000 7f73d2777700 1 Processor -- start 2024-04-25T17:36:53.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.017+0000 7f73d2777700 1 -- start start 2024-04-25T17:36:53.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.017+0000 7f73d2777700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73cc074c70 0x7f73cc108480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.017+0000 7f73d2777700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73cc075590 0x7f73cc1089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.017+0000 7f73d2777700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73cc10f470 0x7f73cc1035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.017+0000 7f73d2777700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc114e70 con 0x7f73cc074c70 2024-04-25T17:36:53.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.017+0000 7f73d2777700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc114cf0 con 0x7f73cc10f470 2024-04-25T17:36:53.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.017+0000 7f73d2777700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc114ff0 con 0x7f73cc075590 2024-04-25T17:36:53.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.017+0000 7f73cb7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73cc075590 0x7f73cc1089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.018+0000 7f73cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73cc074c70 0x7f73cc108480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.018+0000 7f73cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73cc074c70 0x7f73cc108480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59960/0 (socket says 172.21.15.98:59960) 2024-04-25T17:36:53.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.018+0000 7f73cbfff700 1 -- 172.21.15.98:0/114504671 learned_addr learned my addr 172.21.15.98:0/114504671 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:53.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.018+0000 7f73d0d14700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73cc10f470 0x7f73cc1035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.018+0000 7f73cbfff700 1 -- 172.21.15.98:0/114504671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73cc075590 msgr2=0x7f73cc1089c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.018+0000 7f73cbfff700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73cc075590 0x7f73cc1089c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.018+0000 7f73cbfff700 1 -- 172.21.15.98:0/114504671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73cc10f470 msgr2=0x7f73cc1035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.018+0000 7f73cbfff700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73cc10f470 0x7f73cc1035a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.018+0000 7f73cbfff700 1 -- 172.21.15.98:0/114504671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73cc103ea0 con 0x7f73cc074c70 2024-04-25T17:36:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.018+0000 7f73cbfff700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73cc074c70 0x7f73cc108480 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f73c400ef10 tx=0x7f73c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.019+0000 7f73c97fa700 1 -- 172.21.15.98:0/114504671 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c400c9c0 con 0x7f73cc074c70 2024-04-25T17:36:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.019+0000 7f73d2777700 1 -- 172.21.15.98:0/114504671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73cc104130 con 0x7f73cc074c70 2024-04-25T17:36:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.019+0000 7f73c97fa700 1 -- 172.21.15.98:0/114504671 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c4019070 con 0x7f73cc074c70 2024-04-25T17:36:53.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.019+0000 7f73c97fa700 1 -- 172.21.15.98:0/114504671 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c4014480 con 0x7f73cc074c70 2024-04-25T17:36:53.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.019+0000 7f73d2777700 1 -- 172.21.15.98:0/114504671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73cc10e370 con 0x7f73cc074c70 2024-04-25T17:36:53.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.021+0000 7f73c97fa700 1 -- 172.21.15.98:0/114504671 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f73c4012050 con 0x7f73cc074c70 2024-04-25T17:36:53.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.023+0000 7f73dd000b80 1 Processor -- start 2024-04-25T17:36:53.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.023+0000 7f73dd000b80 1 -- start start 2024-04-25T17:36:53.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec078a80 0x55ffec078e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73c97fa700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f73b40a3f70 0x7f73b40a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 0x55ffec083d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec084270 0x55ffec086660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73dd000b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ffebfcbec0 con 0x55ffec078a80 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73dd000b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ffebf72af0 con 0x55ffec084270 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73dd000b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ffebfc7e80 con 0x55ffec0793a0 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73cbfff700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f73b40a3f70 0x7f73b40a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73d0d14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec078a80 0x55ffec078e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73cb7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 0x55ffec083d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73d0d14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec078a80 0x55ffec078e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59966/0 (socket says 172.21.15.98:59966) 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73d0d14700 1 -- 172.21.15.98:0/3158941236 learned_addr learned my addr 172.21.15.98:0/3158941236 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73d0d14700 1 --2- 172.21.15.98:0/3158941236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec084270 0x55ffec086660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73c97fa700 1 -- 172.21.15.98:0/114504671 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f73c40d0a10 con 0x7f73cc074c70 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73cb7fe700 1 -- 172.21.15.98:0/3158941236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec084270 msgr2=0x55ffec086660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/3158941236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec084270 0x55ffec086660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73cb7fe700 1 -- 172.21.15.98:0/3158941236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec078a80 msgr2=0x55ffec078e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/3158941236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec078a80 0x55ffec078e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73cb7fe700 1 -- 172.21.15.98:0/3158941236 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffebf19a60 con 0x55ffec0793a0 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73cbfff700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f73b40a3f70 0x7f73b40a6430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73c40ce070 tx=0x7f73c4098410 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.024+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/3158941236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 0x55ffec083d30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f73cc104df0 tx=0x7f73c0009f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e97ecfff633dfcf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.025+0000 7f73b27fc700 1 -- 172.21.15.98:0/3158941236 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0011cb0 con 0x55ffec0793a0 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.025+0000 7f73b27fc700 1 -- 172.21.15.98:0/3158941236 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c0011e50 con 0x55ffec0793a0 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.025+0000 7f73b27fc700 1 -- 172.21.15.98:0/3158941236 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0005280 con 0x55ffec0793a0 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.025+0000 7f73dd000b80 1 -- 172.21.15.98:0/3158941236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 msgr2=0x55ffec083d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:53.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.025+0000 7f73dd000b80 1 --2- 172.21.15.98:0/3158941236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 0x55ffec083d30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f73cc104df0 tx=0x7f73c0009f40 comp rx=0 tx=0).stop 2024-04-25T17:36:53.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.026+0000 7f73dd000b80 1 -- 172.21.15.98:0/3158941236 shutdown_connections 2024-04-25T17:36:53.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.026+0000 7f73dd000b80 1 --2- 172.21.15.98:0/3158941236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec084270 0x55ffec086660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.026+0000 7f73dd000b80 1 --2- 172.21.15.98:0/3158941236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec078a80 0x55ffec078e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.026+0000 7f73dd000b80 1 --2- 172.21.15.98:0/3158941236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 0x55ffec083d30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.026+0000 7f73dd000b80 1 -- 172.21.15.98:0/3158941236 >> 172.21.15.98:0/3158941236 conn(0x55ffebfa6820 msgr2=0x55ffec074f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:53.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.026+0000 7f73dd000b80 1 -- 172.21.15.98:0/3158941236 shutdown_connections 2024-04-25T17:36:53.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.026+0000 7f73dd000b80 1 -- 172.21.15.98:0/3158941236 wait complete. 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.027+0000 7f73dd000b80 1 -- start start 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.027+0000 7f73dd000b80 1 Processor -- start 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.027+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 0x55ffec083a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec0793a0 0x55ffec080d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec084270 0x55ffec081240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73d0d14700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 0x55ffec083a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73d0d14700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 0x55ffec083a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:45954/0 (socket says 172.21.15.98:45954) 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73d0d14700 1 -- 172.21.15.98:0/190861186 learned_addr learned my addr 172.21.15.98:0/190861186 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ffec086c60 con 0x55ffec0793a0 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ffebf6ad60 con 0x55ffec084270 2024-04-25T17:36:53.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ffec086de0 con 0x55ffec078a80 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec0793a0 0x55ffec080d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73d0d14700 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec084270 0x55ffec081240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73dd000b80 10 client.0 _ll_register_callbacks cb 0x55ffebf64ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73d0d14700 1 -- 172.21.15.98:0/190861186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec084270 msgr2=0x55ffec081240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73d0d14700 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec084270 0x55ffec081240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73d0d14700 1 -- 172.21.15.98:0/190861186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec0793a0 msgr2=0x55ffec080d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73d0d14700 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec0793a0 0x55ffec080d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73d0d14700 1 -- 172.21.15.98:0/190861186 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffebf83400 con 0x55ffec078a80 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73d0d14700 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 0x55ffec083a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0059d0 tx=0x7f73bc009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b2b7b783ac0c844 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.028+0000 7f73ab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.029+0000 7f73ab7fe700 1 -- 172.21.15.98:0/190861186 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc040060 con 0x55ffec078a80 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.029+0000 7f73ab7fe700 1 -- 172.21.15.98:0/190861186 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f73bc00bc10 con 0x55ffec078a80 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.029+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73c0011e50 con 0x55ffec078a80 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.029+0000 7f73ab7fe700 1 -- 172.21.15.98:0/190861186 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f73bc00be10 con 0x55ffec078a80 2024-04-25T17:36:53.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.029+0000 7f73dd000b80 10 client.24988 fetch_fsmap learned FSMap version 29 2024-04-25T17:36:53.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.029+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ffec0914b0 con 0x55ffec078a80 2024-04-25T17:36:53.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.029+0000 7f73ab7fe700 1 -- 172.21.15.98:0/190861186 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73bc003b30 con 0x55ffec078a80 2024-04-25T17:36:53.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.030+0000 7f73ab7fe700 1 -- 172.21.15.98:0/190861186 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f73bc00f600 con 0x55ffec078a80 2024-04-25T17:36:53.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.030+0000 7f73dd000b80 10 client.24988 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:36:53.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.030+0000 7f73dd000b80 10 client.24988 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:36:53.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.030+0000 7f73dd000b80 20 client.24988 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:53.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.030+0000 7f73dd000b80 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ffec093e30 0x55ffec0962b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.030+0000 7f73dd000b80 4 client.24988 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:36:53.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.030+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f73c0011e50 con 0x55ffec093e30 2024-04-25T17:36:53.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.030+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ffec093e30 0x55ffec0962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.031+0000 7f73ab7fe700 10 client.24988 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.031+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ffec093e30 0x55ffec0962b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95d40468cd8d6695 server_cookie=cb9e4e91ddfacd8e in_seq=0 out_seq=0 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73ab7fe700 1 -- 172.21.15.98:0/190861186 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f73c0011e50 con 0x55ffec093e30 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73ab7fe700 10 client.24988 handle_command_reply: tid=0 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 client.24988 shutdown 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 msgr2=0x55ffec083a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 0x55ffec083a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0059d0 tx=0x7f73bc009ca0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 shutdown_connections 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ffec093e30 0x55ffec0962b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 0x55ffec083a00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec084270 0x55ffec081240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 --2- 172.21.15.98:0/190861186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec0793a0 0x55ffec080d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 >> 172.21.15.98:0/190861186 conn(0x55ffebfa6820 msgr2=0x55ffec074b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73ab7fe700 0 client.24988 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:53.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 shutdown_connections 2024-04-25T17:36:53.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 1 -- 172.21.15.98:0/190861186 wait complete. 2024-04-25T17:36:53.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.032+0000 7f73dd000b80 20 client.24988 trim_cache size 0 max 16384 2024-04-25T17:36:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73dd000b80 1 Processor -- start 2024-04-25T17:36:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73dd000b80 1 -- start start 2024-04-25T17:36:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec078a80 0x55ffec07fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 0x55ffec07c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec08fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73dd000b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc1b61a0 con 0x55ffec084270 2024-04-25T17:36:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73dd000b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc072af0 con 0x55ffec078a80 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73dd000b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc070940 con 0x55ffec0793a0 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73d0d14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec08fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73d0d14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec08fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59984/0 (socket says 172.21.15.98:59984) 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73d0d14700 1 -- 172.21.15.98:0/3431306849 learned_addr learned my addr 172.21.15.98:0/3431306849 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/3431306849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 0x55ffec07c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.058+0000 7f73d0d14700 1 --2- 172.21.15.98:0/3431306849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec078a80 0x55ffec07fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.059+0000 7f73cb7fe700 1 -- 172.21.15.98:0/3431306849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec078a80 msgr2=0x55ffec07fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.059+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/3431306849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec078a80 0x55ffec07fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.059+0000 7f73cb7fe700 1 -- 172.21.15.98:0/3431306849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 msgr2=0x55ffec08fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.059+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/3431306849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec08fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.059+0000 7f73cb7fe700 1 -- 172.21.15.98:0/3431306849 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffec06f390 con 0x55ffec0793a0 2024-04-25T17:36:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.059+0000 7f73d0d14700 1 --2- 172.21.15.98:0/3431306849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec08fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:36:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.059+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/3431306849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 0x55ffec07c460 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f73c0004c40 tx=0x7f73c0005ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2c140be1000b578 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.059+0000 7f73b1ffb700 1 -- 172.21.15.98:0/3431306849 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0004870 con 0x55ffec0793a0 2024-04-25T17:36:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.060+0000 7f73b1ffb700 1 -- 172.21.15.98:0/3431306849 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c002a070 con 0x55ffec0793a0 2024-04-25T17:36:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.060+0000 7f73b1ffb700 1 -- 172.21.15.98:0/3431306849 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0005480 con 0x55ffec0793a0 2024-04-25T17:36:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.061+0000 7f73dd000b80 1 -- 172.21.15.98:0/3431306849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 msgr2=0x55ffec07c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.062+0000 7f73dd000b80 1 --2- 172.21.15.98:0/3431306849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 0x55ffec07c460 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f73c0004c40 tx=0x7f73c0005ce0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.062+0000 7f73dd000b80 1 -- 172.21.15.98:0/3431306849 shutdown_connections 2024-04-25T17:36:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.062+0000 7f73dd000b80 1 --2- 172.21.15.98:0/3431306849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec08fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.062+0000 7f73dd000b80 1 --2- 172.21.15.98:0/3431306849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec078a80 0x55ffec07fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.062+0000 7f73dd000b80 1 --2- 172.21.15.98:0/3431306849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec0793a0 0x55ffec07c460 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.062+0000 7f73dd000b80 1 -- 172.21.15.98:0/3431306849 >> 172.21.15.98:0/3431306849 conn(0x55ffebfa6820 msgr2=0x55ffec082530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.062+0000 7f73dd000b80 1 -- 172.21.15.98:0/3431306849 shutdown_connections 2024-04-25T17:36:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.062+0000 7f73dd000b80 1 -- 172.21.15.98:0/3431306849 wait complete. 2024-04-25T17:36:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.063+0000 7f73dd000b80 1 -- start start 2024-04-25T17:36:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.063+0000 7f73dd000b80 1 Processor -- start 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.063+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 0x55ffec08c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.063+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec0793a0 0x55ffec08c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.063+0000 7f73dd000b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec0890d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.063+0000 7f73dd000b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc070fb0 con 0x55ffec084270 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.063+0000 7f73dd000b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc072420 con 0x55ffec0793a0 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.063+0000 7f73dd000b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc071680 con 0x55ffec078a80 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.063+0000 7f73dd000b80 10 client.0 _ll_register_callbacks cb 0x55ffebf64ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73d0d14700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 0x55ffec08c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73d0d14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec0890d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73d0d14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec0890d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59998/0 (socket says 172.21.15.98:59998) 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73d0d14700 1 -- 172.21.15.98:0/1979664814 learned_addr learned my addr 172.21.15.98:0/1979664814 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec0793a0 0x55ffec08c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73d0d14700 1 -- 172.21.15.98:0/1979664814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 msgr2=0x55ffec08c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73d0d14700 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 0x55ffec08c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73d0d14700 1 -- 172.21.15.98:0/1979664814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec0793a0 msgr2=0x55ffec08c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:36:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73d0d14700 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec0793a0 0x55ffec08c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73d0d14700 1 -- 172.21.15.98:0/1979664814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffec089910 con 0x55ffec084270 2024-04-25T17:36:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73d0d14700 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec0890d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0059d0 tx=0x7f73bc005610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c955e28f25cf44b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:36:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73ab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:36:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73ab7fe700 1 -- 172.21.15.98:0/1979664814 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73bc040070 con 0x55ffec084270 2024-04-25T17:36:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.064+0000 7f73dd000b80 1 -- 172.21.15.98:0/1979664814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73c002a070 con 0x55ffec084270 2024-04-25T17:36:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.068+0000 7f73ab7fe700 1 -- 172.21.15.98:0/1979664814 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f73bc003680 con 0x55ffec084270 2024-04-25T17:36:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.069+0000 7f73ab7fe700 1 -- 172.21.15.98:0/1979664814 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f73bc04a080 con 0x55ffec084270 2024-04-25T17:36:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.069+0000 7f73dd000b80 10 client.15414 fetch_fsmap learned FSMap version 29 2024-04-25T17:36:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.069+0000 7f73dd000b80 1 -- 172.21.15.98:0/1979664814 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ffec0fafe0 con 0x55ffec084270 2024-04-25T17:36:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.069+0000 7f73ab7fe700 1 -- 172.21.15.98:0/1979664814 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73bc003d70 con 0x55ffec084270 2024-04-25T17:36:53.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.069+0000 7f73ab7fe700 1 -- 172.21.15.98:0/1979664814 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f73bc03aaa0 con 0x55ffec084270 2024-04-25T17:36:53.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.070+0000 7f73dd000b80 10 client.15414 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:36:53.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.070+0000 7f73dd000b80 10 client.15414 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:36:53.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.070+0000 7f73dd000b80 20 client.15414 populate_metadata read hostname 'smithi098' 2024-04-25T17:36:53.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.070+0000 7f73dd000b80 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ffec0fb790 0x55ffec0fdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:36:53.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.070+0000 7f73dd000b80 4 client.15414 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.55, "await_for": 42.0}] 2024-04-25T17:36:53.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.070+0000 7f73dd000b80 1 -- 172.21.15.98:0/1979664814 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.55, "await_for": 42.0}) -- 0x7f73c002a070 con 0x55ffec0fb790 2024-04-25T17:36:53.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.070+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ffec0fb790 0x55ffec0fdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:36:53.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.071+0000 7f73cb7fe700 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ffec0fb790 0x55ffec0fdbb0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9e8c5abda72f39b server_cookie=1dc9fc3af8757a43 in_seq=0 out_seq=0 2024-04-25T17:36:53.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:53.071+0000 7f73ab7fe700 10 client.15414 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73ab7fe700 1 -- 172.21.15.98:0/1979664814 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x7f73c002a070 con 0x55ffec0fb790 2024-04-25T17:36:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73ab7fe700 10 client.15414 handle_command_reply: tid=0 2024-04-25T17:36:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 client.15414 shutdown 2024-04-25T17:36:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 -- 172.21.15.98:0/1979664814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 msgr2=0x55ffec0890d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec0890d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0059d0 tx=0x7f73bc005610 comp rx=0 tx=0).stop 2024-04-25T17:36:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 -- 172.21.15.98:0/1979664814 shutdown_connections 2024-04-25T17:36:54.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ffec0fb790 0x55ffec0fdbb0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ffec078a80 0x55ffec08c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ffec084270 0x55ffec0890d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73ab7fe700 0 client.15414 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 --2- 172.21.15.98:0/1979664814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ffec0793a0 0x55ffec08c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 -- 172.21.15.98:0/1979664814 >> 172.21.15.98:0/1979664814 conn(0x55ffebfa6820 msgr2=0x55ffec082dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 -- 172.21.15.98:0/1979664814 shutdown_connections 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.028+0000 7f73dd000b80 1 -- 172.21.15.98:0/1979664814 wait complete. 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73dd000b80 20 client.15414 trim_cache size 0 max 16384 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 29, 2024-04-25T17:36:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 2, 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "caaeb197": { 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 2, 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 87.5, 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:36:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:36:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:36:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:36:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:36:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:36:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:36:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:36:54.032 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:36:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 -- 172.21.15.98:0/114504671 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f73b40a3f70 msgr2=0x7f73b40a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:54.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f73b40a3f70 0x7f73b40a6430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73c40ce070 tx=0x7f73c4098410 comp rx=0 tx=0).stop 2024-04-25T17:36:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 -- 172.21.15.98:0/114504671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73cc074c70 msgr2=0x7f73cc108480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:36:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73cc074c70 0x7f73cc108480 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f73c400ef10 tx=0x7f73c400c3a0 comp rx=0 tx=0).stop 2024-04-25T17:36:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 -- 172.21.15.98:0/114504671 shutdown_connections 2024-04-25T17:36:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f73b40a3f70 0x7f73b40a6430 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73cc10f470 0x7f73cc1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73cc074c70 0x7f73cc108480 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 --2- 172.21.15.98:0/114504671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73cc075590 0x7f73cc1089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:36:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 -- 172.21.15.98:0/114504671 >> 172.21.15.98:0/114504671 conn(0x7f73cc079bf0 msgr2=0x7f73cc19c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:36:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.029+0000 7f73d2777700 1 -- 172.21.15.98:0/114504671 shutdown_connections 2024-04-25T17:36:54.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:36:54.030+0000 7f73d2777700 1 -- 172.21.15.98:0/114504671 wait complete. 2024-04-25T17:36:54.381 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:54 smithi098 ceph-mon[29192]: pgmap v32: 97 pgs: 97 active+clean; 953 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.0 MiB/s wr, 654 op/s 2024-04-25T17:36:54.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:54 smithi203 ceph-mon[32834]: pgmap v32: 97 pgs: 97 active+clean; 953 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.0 MiB/s wr, 654 op/s 2024-04-25T17:36:54.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:54 smithi173 ceph-mon[32856]: pgmap v32: 97 pgs: 97 active+clean; 953 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.0 MiB/s wr, 654 op/s 2024-04-25T17:36:54.869 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'caaeb197', quiesce took 9.7 seconds. Will release after: 27.55 2024-04-25T17:36:56.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:56 smithi098 ceph-mon[29192]: pgmap v33: 97 pgs: 97 active+clean; 953 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.0 MiB/s wr, 654 op/s 2024-04-25T17:36:56.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:56 smithi203 ceph-mon[32834]: pgmap v33: 97 pgs: 97 active+clean; 953 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.0 MiB/s wr, 654 op/s 2024-04-25T17:36:56.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:56 smithi173 ceph-mon[32856]: pgmap v33: 97 pgs: 97 active+clean; 953 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.0 MiB/s wr, 654 op/s 2024-04-25T17:36:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 25.0 s, 966.3 tps, lat 30.882 ms stddev 33.624 2024-04-25T17:36:58.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:36:58 smithi098 ceph-mon[29192]: pgmap v34: 97 pgs: 97 active+clean; 955 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 9.8 MiB/s wr, 906 op/s 2024-04-25T17:36:58.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:36:58 smithi203 ceph-mon[32834]: pgmap v34: 97 pgs: 97 active+clean; 955 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 9.8 MiB/s wr, 906 op/s 2024-04-25T17:36:58.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:36:58 smithi173 ceph-mon[32856]: pgmap v34: 97 pgs: 97 active+clean; 955 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 9.8 MiB/s wr, 906 op/s 2024-04-25T17:37:00.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:00 smithi098 ceph-mon[29192]: pgmap v35: 97 pgs: 97 active+clean; 955 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 6.6 MiB/s wr, 591 op/s 2024-04-25T17:37:00.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:00 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:37:00.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:00 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:37:00.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:00 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:37:00.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:00 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:37:00.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:00 smithi203 ceph-mon[32834]: pgmap v35: 97 pgs: 97 active+clean; 955 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 6.6 MiB/s wr, 591 op/s 2024-04-25T17:37:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:00 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:37:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:00 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:37:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:00 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:37:00.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:00 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:37:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:00 smithi173 ceph-mon[32856]: pgmap v35: 97 pgs: 97 active+clean; 955 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 6.6 MiB/s wr, 591 op/s 2024-04-25T17:37:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:00 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:37:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:00 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:37:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:00 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:37:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:00 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:37:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 30.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:37:02.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:02 smithi098 ceph-mon[29192]: pgmap v36: 97 pgs: 97 active+clean; 956 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 7.4 MiB/s wr, 605 op/s 2024-04-25T17:37:02.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:02 smithi203 ceph-mon[32834]: pgmap v36: 97 pgs: 97 active+clean; 956 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 7.4 MiB/s wr, 605 op/s 2024-04-25T17:37:02.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:02 smithi173 ceph-mon[32856]: pgmap v36: 97 pgs: 97 active+clean; 956 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 7.4 MiB/s wr, 605 op/s 2024-04-25T17:37:02.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.627+0000 7f13b4b30640 1 -- 172.21.15.98:0/821289404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b01005c0 msgr2=0x7f13b0100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:02.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.627+0000 7f13b4b30640 1 --2- 172.21.15.98:0/821289404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b01005c0 0x7f13b0100a40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f13a400aa40 tx=0x7f13a402b750 comp rx=0 tx=0).stop 2024-04-25T17:37:02.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.627+0000 7f13b4b30640 1 -- 172.21.15.98:0/821289404 shutdown_connections 2024-04-25T17:37:02.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.627+0000 7f13b4b30640 1 --2- 172.21.15.98:0/821289404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13b0100f80 0x7f13b010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.627+0000 7f13b4b30640 1 --2- 172.21.15.98:0/821289404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b01005c0 0x7f13b0100a40 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.627+0000 7f13b4b30640 1 --2- 172.21.15.98:0/821289404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b0106660 0x7f13b0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.627+0000 7f13b4b30640 1 -- 172.21.15.98:0/821289404 >> 172.21.15.98:0/821289404 conn(0x7f13b0077390 msgr2=0x7f13b00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:02.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.627+0000 7f13b4b30640 1 -- 172.21.15.98:0/821289404 shutdown_connections 2024-04-25T17:37:02.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.627+0000 7f13b4b30640 1 -- 172.21.15.98:0/821289404 wait complete. 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13b4b30640 1 Processor -- start 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13b4b30640 1 -- start start 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13b4b30640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13b01005c0 0x7f13b0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13b4b30640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0100f80 0x7f13b0198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13b4b30640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b0106660 0x7f13b019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13b4b30640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13b0110010 con 0x7f13b0100f80 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13b4b30640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f13b010fe90 con 0x7f13b0106660 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13b4b30640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f13b0110190 con 0x7f13b01005c0 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13add9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0100f80 0x7f13b0198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13add9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0100f80 0x7f13b0198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33846/0 (socket says 172.21.15.98:33846) 2024-04-25T17:37:02.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13add9b640 1 -- 172.21.15.98:0/2361044995 learned_addr learned my addr 172.21.15.98:0/2361044995 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13ae59c640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13b01005c0 0x7f13b0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.628+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b0106660 0x7f13b019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f13add9b640 1 -- 172.21.15.98:0/2361044995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13b01005c0 msgr2=0x7f13b0198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f13add9b640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13b01005c0 0x7f13b0198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f13add9b640 1 -- 172.21.15.98:0/2361044995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b0106660 msgr2=0x7f13b019cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f13add9b640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b0106660 0x7f13b019cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f13add9b640 1 -- 172.21.15.98:0/2361044995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b019d670 con 0x7f13b0100f80 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f13add9b640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0100f80 0x7f13b0198bc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f13a4005940 tx=0x7f13a40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f139f7fe640 1 -- 172.21.15.98:0/2361044995 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a403f070 con 0x7f13b0100f80 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f13b4b30640 1 -- 172.21.15.98:0/2361044995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b019d900 con 0x7f13b0100f80 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f139f7fe640 1 -- 172.21.15.98:0/2361044995 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a4032400 con 0x7f13b0100f80 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f139f7fe640 1 -- 172.21.15.98:0/2361044995 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4036040 con 0x7f13b0100f80 2024-04-25T17:37:02.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.629+0000 7f13b4b30640 1 -- 172.21.15.98:0/2361044995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b01a51f0 con 0x7f13b0100f80 2024-04-25T17:37:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.631+0000 7f139f7fe640 1 -- 172.21.15.98:0/2361044995 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f13a4042070 con 0x7f13b0100f80 2024-04-25T17:37:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.631+0000 7f13b7846740 1 Processor -- start 2024-04-25T17:37:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13b7846740 1 -- start start 2024-04-25T17:37:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f139f7fe640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13980a3ee0 0x7f13980a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13b7846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a3311a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13b7846740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a331c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13b7846740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a331ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13b7846740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5646a3276970 con 0x5646a3311f40 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13b7846740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5646a32134a0 con 0x5646a331c800 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13b7846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5646a3272930 con 0x5646a3311620 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13980a3ee0 0x7f13980a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f139f7fe640 1 -- 172.21.15.98:0/2361044995 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f13a40f11d0 con 0x7f13b0100f80 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13add9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a331c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13ae59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a3311a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13add9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a331c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33858/0 (socket says 172.21.15.98:33858) 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13add9b640 1 -- 172.21.15.98:0/3360976033 learned_addr learned my addr 172.21.15.98:0/3360976033 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.632+0000 7f13aed9d640 1 --2- 172.21.15.98:0/3360976033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a331ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f13ae59c640 1 -- 172.21.15.98:0/3360976033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 msgr2=0x5646a331ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f13ae59c640 1 --2- 172.21.15.98:0/3360976033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a331ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f13ae59c640 1 -- 172.21.15.98:0/3360976033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 msgr2=0x5646a331c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f13ae59c640 1 --2- 172.21.15.98:0/3360976033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a331c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f13ae59c640 1 -- 172.21.15.98:0/3360976033 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646a2f89210 con 0x5646a3311620 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f13aed9d640 1 --2- 172.21.15.98:0/3360976033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a331ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f13add9b640 1 --2- 172.21.15.98:0/3360976033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a331c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f13aed9d640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13980a3ee0 0x7f13980a63a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009f20 tx=0x7f13a80137d0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f13ae59c640 1 --2- 172.21.15.98:0/3360976033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a3311a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f13a0005fd0 tx=0x7f13a0010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a19c0f2092dcb3db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f139cff9640 1 -- 172.21.15.98:0/3360976033 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0016070 con 0x5646a3311620 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f139cff9640 1 -- 172.21.15.98:0/3360976033 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a0003b70 con 0x5646a3311620 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.633+0000 7f139cff9640 1 -- 172.21.15.98:0/3360976033 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0012740 con 0x5646a3311620 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 -- 172.21.15.98:0/3360976033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 msgr2=0x5646a3311a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 --2- 172.21.15.98:0/3360976033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a3311a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f13a0005fd0 tx=0x7f13a0010740 comp rx=0 tx=0).stop 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 -- 172.21.15.98:0/3360976033 shutdown_connections 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 --2- 172.21.15.98:0/3360976033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a331ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 --2- 172.21.15.98:0/3360976033 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a331c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 --2- 172.21.15.98:0/3360976033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a3311a00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 -- 172.21.15.98:0/3360976033 >> 172.21.15.98:0/3360976033 conn(0x5646a330db30 msgr2=0x5646a330df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 -- 172.21.15.98:0/3360976033 shutdown_connections 2024-04-25T17:37:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 -- 172.21.15.98:0/3360976033 wait complete. 2024-04-25T17:37:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 -- start start 2024-04-25T17:37:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.634+0000 7f13b7846740 1 Processor -- start 2024-04-25T17:37:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13b7846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a331c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13b7846740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a3319480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13b7846740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a3319a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13b7846740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5646a331f430 con 0x5646a3311f40 2024-04-25T17:37:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13b7846740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5646a32110b0 con 0x5646a331c800 2024-04-25T17:37:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13b7846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5646a331f5b0 con 0x5646a3311620 2024-04-25T17:37:02.642 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13b7846740 10 client.0 _ll_register_callbacks cb 0x5646a31fed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13add9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a3319480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13add9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a3319480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33868/0 (socket says 172.21.15.98:33868) 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13add9b640 1 -- 172.21.15.98:0/81461919 learned_addr learned my addr 172.21.15.98:0/81461919 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13ae59c640 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a331c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13aed9d640 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a3319a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13add9b640 1 -- 172.21.15.98:0/81461919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 msgr2=0x5646a331c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13add9b640 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a331c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13add9b640 1 -- 172.21.15.98:0/81461919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 msgr2=0x5646a3319a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13add9b640 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a3319a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13add9b640 1 -- 172.21.15.98:0/81461919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646a3055620 con 0x5646a3311f40 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13add9b640 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a3319480 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f13a40bcd20 tx=0x7f13a40bcc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8db78791e3bd55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f1395ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:02.643 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f1395ffb640 1 -- 172.21.15.98:0/81461919 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4049040 con 0x5646a3311f40 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f1395ffb640 1 -- 172.21.15.98:0/81461919 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f13a40325b0 con 0x5646a3311f40 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.635+0000 7f13b7846740 1 -- 172.21.15.98:0/81461919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a0003b70 con 0x5646a3311f40 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f1395ffb640 1 -- 172.21.15.98:0/81461919 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f13a40bd720 con 0x5646a3311f40 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f13b7846740 10 client.15426 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f13b7846740 1 -- 172.21.15.98:0/81461919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5646a332b3c0 con 0x5646a3311f40 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f1395ffb640 1 -- 172.21.15.98:0/81461919 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a0003b70 con 0x5646a3311f40 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f1395ffb640 1 -- 172.21.15.98:0/81461919 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f13a400b040 con 0x5646a3311f40 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f13b7846740 10 client.15426 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f13b7846740 10 client.15426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f13b7846740 20 client.15426 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f13b7846740 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5646a332c8a0 0x5646a332ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f13b7846740 4 client.15426 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f13b7846740 1 -- 172.21.15.98:0/81461919 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646a3284390 con 0x5646a332c8a0 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.636+0000 7f13ae59c640 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5646a332c8a0 0x5646a332ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.644 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.637+0000 7f1395ffb640 10 client.15426 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.637+0000 7f13ae59c640 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5646a332c8a0 0x5646a332ed00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14a9cea55dd4c330 server_cookie=aab98497c87c969a in_seq=0 out_seq=0 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.637+0000 7f1395ffb640 1 -- 172.21.15.98:0/81461919 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646a3284390 con 0x5646a332c8a0 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.637+0000 7f1395ffb640 10 client.15426 handle_command_reply: tid=0 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.637+0000 7f13b7846740 1 client.15426 shutdown 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.637+0000 7f13b7846740 1 -- 172.21.15.98:0/81461919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 msgr2=0x5646a3319480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.637+0000 7f13b7846740 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a3319480 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f13a40bcd20 tx=0x7f13a40bcc20 comp rx=0 tx=0).stop 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.638+0000 7f13b7846740 1 -- 172.21.15.98:0/81461919 shutdown_connections 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.638+0000 7f13b7846740 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5646a332c8a0 0x5646a332ed00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.638+0000 7f13b7846740 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a3319a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.638+0000 7f13b7846740 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311f40 0x5646a3319480 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.645 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.638+0000 7f13b7846740 1 --2- 172.21.15.98:0/81461919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a331c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.638+0000 7f1395ffb640 0 client.15426 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.638+0000 7f13b7846740 1 -- 172.21.15.98:0/81461919 >> 172.21.15.98:0/81461919 conn(0x5646a330db30 msgr2=0x5646a33138b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.638+0000 7f13b7846740 1 -- 172.21.15.98:0/81461919 shutdown_connections 2024-04-25T17:37:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.638+0000 7f13b7846740 1 -- 172.21.15.98:0/81461919 wait complete. 2024-04-25T17:37:02.646 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.638+0000 7f13b7846740 20 client.15426 trim_cache size 0 max 16384 2024-04-25T17:37:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.652+0000 7f13b7846740 1 Processor -- start 2024-04-25T17:37:02.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.652+0000 7f13b7846740 1 -- start start 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.652+0000 7f13b7846740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311620 0x5646a33282f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.652+0000 7f13b7846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311f40 0x5646a330cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.652+0000 7f13b7846740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a3334610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.652+0000 7f13b7846740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01b0300 con 0x5646a3311620 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.652+0000 7f13b7846740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01adba0 con 0x5646a331c800 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.652+0000 7f13b7846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01ab9f0 con 0x5646a3311f40 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13ae59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311620 0x5646a33282f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13ae59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311620 0x5646a33282f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33882/0 (socket says 172.21.15.98:33882) 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13ae59c640 1 -- 172.21.15.98:0/1711577503 learned_addr learned my addr 172.21.15.98:0/1711577503 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13add9b640 1 --2- 172.21.15.98:0/1711577503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311f40 0x5646a330cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13ae59c640 1 -- 172.21.15.98:0/1711577503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311f40 msgr2=0x5646a330cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13ae59c640 1 --2- 172.21.15.98:0/1711577503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311f40 0x5646a330cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.658 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13ae59c640 1 -- 172.21.15.98:0/1711577503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 msgr2=0x5646a3334610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13aed9d640 1 --2- 172.21.15.98:0/1711577503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a3334610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13ae59c640 1 --2- 172.21.15.98:0/1711577503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a3334610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13ae59c640 1 -- 172.21.15.98:0/1711577503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646a330d960 con 0x5646a3311620 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13aed9d640 1 --2- 172.21.15.98:0/1711577503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a3334610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f13ae59c640 1 --2- 172.21.15.98:0/1711577503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311620 0x5646a33282f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f13a0005fd0 tx=0x7f13a0004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430d0e2910065e3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f1397fff640 1 -- 172.21.15.98:0/1711577503 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0016040 con 0x5646a3311620 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f1397fff640 1 -- 172.21.15.98:0/1711577503 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a0026860 con 0x5646a3311620 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.653+0000 7f1397fff640 1 -- 172.21.15.98:0/1711577503 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0005510 con 0x5646a3311620 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 -- 172.21.15.98:0/1711577503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311620 msgr2=0x5646a33282f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 --2- 172.21.15.98:0/1711577503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311620 0x5646a33282f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f13a0005fd0 tx=0x7f13a0004840 comp rx=0 tx=0).stop 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 -- 172.21.15.98:0/1711577503 shutdown_connections 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 --2- 172.21.15.98:0/1711577503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a331c800 0x5646a3334610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 --2- 172.21.15.98:0/1711577503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311f40 0x5646a330cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 --2- 172.21.15.98:0/1711577503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a3311620 0x5646a33282f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 -- 172.21.15.98:0/1711577503 >> 172.21.15.98:0/1711577503 conn(0x5646a330db30 msgr2=0x5646a330c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 -- 172.21.15.98:0/1711577503 shutdown_connections 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 -- 172.21.15.98:0/1711577503 wait complete. 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 -- start start 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 Processor -- start 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a33221d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a3311f40 0x5646a3322710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a331c800 0x5646a3322c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01ac060 con 0x5646a331c800 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01ad4d0 con 0x5646a3311f40 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01ac730 con 0x5646a3311620 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.654+0000 7f13b7846740 10 client.0 _ll_register_callbacks cb 0x7f13a40ee7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:02.660 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13aed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a331c800 0x5646a3322c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13aed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a331c800 0x5646a3322c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33894/0 (socket says 172.21.15.98:33894) 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13aed9d640 1 -- 172.21.15.98:0/605602465 learned_addr learned my addr 172.21.15.98:0/605602465 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13add9b640 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a3311f40 0x5646a3322710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13ae59c640 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a33221d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13aed9d640 1 -- 172.21.15.98:0/605602465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 msgr2=0x5646a33221d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13aed9d640 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a33221d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13aed9d640 1 -- 172.21.15.98:0/605602465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a3311f40 msgr2=0x5646a3322710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13aed9d640 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a3311f40 0x5646a3322710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13aed9d640 1 -- 172.21.15.98:0/605602465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646a339a6a0 con 0x5646a331c800 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13aed9d640 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a331c800 0x5646a3322c50 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f13a8013cd0 tx=0x7f13a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1fe28408246700 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f1395ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f1395ffb640 1 -- 172.21.15.98:0/605602465 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a802e060 con 0x5646a331c800 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f1395ffb640 1 -- 172.21.15.98:0/605602465 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f13a800f9a0 con 0x5646a331c800 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13b7846740 1 -- 172.21.15.98:0/605602465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a0026860 con 0x5646a331c800 2024-04-25T17:37:02.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f1395ffb640 1 -- 172.21.15.98:0/605602465 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f13a8009960 con 0x5646a331c800 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13b7846740 10 client.15432 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f13b7846740 1 -- 172.21.15.98:0/605602465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5646a339b8a0 con 0x5646a331c800 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.655+0000 7f1395ffb640 1 -- 172.21.15.98:0/605602465 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a0026860 con 0x5646a331c800 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.656+0000 7f1395ffb640 1 -- 172.21.15.98:0/605602465 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f13a801f070 con 0x5646a331c800 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.656+0000 7f13b7846740 10 client.15432 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.656+0000 7f13b7846740 10 client.15432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.656+0000 7f13b7846740 20 client.15432 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.656+0000 7f13b7846740 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5646a339c0b0 0x5646a339e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.656+0000 7f13b7846740 4 client.15432 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.656+0000 7f13b7846740 1 -- 172.21.15.98:0/605602465 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5646a327e200 con 0x5646a339c0b0 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.656+0000 7f13ae59c640 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5646a339c0b0 0x5646a339e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.656+0000 7f1395ffb640 10 client.15432 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.656+0000 7f13ae59c640 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5646a339c0b0 0x5646a339e4d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8efce844bff88f83 server_cookie=f88281041dc22833 in_seq=0 out_seq=0 2024-04-25T17:37:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f1395ffb640 1 -- 172.21.15.98:0/605602465 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5646a327e200 con 0x5646a339c0b0 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f1395ffb640 10 client.15432 handle_command_reply: tid=0 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f13b7846740 1 client.15432 shutdown 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f13b7846740 1 -- 172.21.15.98:0/605602465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a331c800 msgr2=0x5646a3322c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f13b7846740 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a331c800 0x5646a3322c50 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f13a8013cd0 tx=0x7f13a8004730 comp rx=0 tx=0).stop 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f13b7846740 1 -- 172.21.15.98:0/605602465 shutdown_connections 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f13b7846740 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5646a339c0b0 0x5646a339e4d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f13b7846740 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5646a331c800 0x5646a3322c50 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f13b7846740 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5646a3311f40 0x5646a3322710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f13b7846740 1 --2- 172.21.15.98:0/605602465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5646a3311620 0x5646a33221d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.692+0000 7f1395ffb640 0 client.15432 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b7846740 1 -- 172.21.15.98:0/605602465 >> 172.21.15.98:0/605602465 conn(0x5646a330db30 msgr2=0x5646a330ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b7846740 1 -- 172.21.15.98:0/605602465 shutdown_connections 2024-04-25T17:37:02.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b7846740 1 -- 172.21.15.98:0/605602465 wait complete. 2024-04-25T17:37:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b7846740 20 client.15432 trim_cache size 0 max 16384 2024-04-25T17:37:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:37:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:37:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:37:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:37:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 -- 172.21.15.98:0/2361044995 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13980a3ee0 msgr2=0x7f13980a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13980a3ee0 0x7f13980a63a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009f20 tx=0x7f13a80137d0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 -- 172.21.15.98:0/2361044995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0100f80 msgr2=0x7f13b0198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0100f80 0x7f13b0198bc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f13a4005940 tx=0x7f13a40042f0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 -- 172.21.15.98:0/2361044995 shutdown_connections 2024-04-25T17:37:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13980a3ee0 0x7f13980a63a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b0106660 0x7f13b019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b0100f80 0x7f13b0198bc0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 --2- 172.21.15.98:0/2361044995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13b01005c0 0x7f13b0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 -- 172.21.15.98:0/2361044995 >> 172.21.15.98:0/2361044995 conn(0x7f13b0077390 msgr2=0x7f13b00fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 -- 172.21.15.98:0/2361044995 shutdown_connections 2024-04-25T17:37:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:02.693+0000 7f13b4b30640 1 -- 172.21.15.98:0/2361044995 wait complete. 2024-04-25T17:37:04.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:04 smithi098 ceph-mon[29192]: pgmap v37: 97 pgs: 97 active+clean; 956 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 2.6 MiB/s wr, 265 op/s 2024-04-25T17:37:04.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:04 smithi203 ceph-mon[32834]: pgmap v37: 97 pgs: 97 active+clean; 956 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 2.6 MiB/s wr, 265 op/s 2024-04-25T17:37:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:04 smithi173 ceph-mon[32856]: pgmap v37: 97 pgs: 97 active+clean; 956 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 2.6 MiB/s wr, 265 op/s 2024-04-25T17:37:06.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:06 smithi098 ceph-mon[29192]: pgmap v38: 97 pgs: 97 active+clean; 956 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 2.6 MiB/s wr, 265 op/s 2024-04-25T17:37:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:06 smithi203 ceph-mon[32834]: pgmap v38: 97 pgs: 97 active+clean; 956 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 2.6 MiB/s wr, 265 op/s 2024-04-25T17:37:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:06 smithi173 ceph-mon[32856]: pgmap v38: 97 pgs: 97 active+clean; 956 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 2.6 MiB/s wr, 265 op/s 2024-04-25T17:37:06.599 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 35.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:37:08.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:08 smithi098 ceph-mon[29192]: pgmap v39: 97 pgs: 97 active+clean; 958 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 8.8 MiB/s wr, 316 op/s 2024-04-25T17:37:08.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:08 smithi203 ceph-mon[32834]: pgmap v39: 97 pgs: 97 active+clean; 958 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 8.8 MiB/s wr, 316 op/s 2024-04-25T17:37:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:08 smithi173 ceph-mon[32856]: pgmap v39: 97 pgs: 97 active+clean; 958 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 8.8 MiB/s wr, 316 op/s 2024-04-25T17:37:10.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:10 smithi098 ceph-mon[29192]: pgmap v40: 97 pgs: 97 active+clean; 958 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.0 MiB/s wr, 65 op/s 2024-04-25T17:37:10.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:10 smithi203 ceph-mon[32834]: pgmap v40: 97 pgs: 97 active+clean; 958 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.0 MiB/s wr, 65 op/s 2024-04-25T17:37:10.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:10 smithi173 ceph-mon[32856]: pgmap v40: 97 pgs: 97 active+clean; 958 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.0 MiB/s wr, 65 op/s 2024-04-25T17:37:11.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 40.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:37:12.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:12 smithi098 ceph-mon[29192]: pgmap v41: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.0 MiB/s wr, 69 op/s 2024-04-25T17:37:12.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:12 smithi203 ceph-mon[32834]: pgmap v41: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.0 MiB/s wr, 69 op/s 2024-04-25T17:37:12.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:12 smithi173 ceph-mon[32856]: pgmap v41: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.0 MiB/s wr, 69 op/s 2024-04-25T17:37:14.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:14 smithi098 ceph-mon[29192]: pgmap v42: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.2 MiB/s wr, 56 op/s 2024-04-25T17:37:14.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:14 smithi203 ceph-mon[32834]: pgmap v42: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.2 MiB/s wr, 56 op/s 2024-04-25T17:37:14.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:14 smithi173 ceph-mon[32856]: pgmap v42: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.2 MiB/s wr, 56 op/s 2024-04-25T17:37:15.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.788+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3235374868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c81021c0 msgr2=0x7fc1c81025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.788+0000 7fc1cfd19640 1 --2- 172.21.15.98:0/3235374868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c81021c0 0x7fc1c81025a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009900 tx=0x7fc1c402a410 comp rx=0 tx=0).stop 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.788+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3235374868 shutdown_connections 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.788+0000 7fc1cfd19640 1 --2- 172.21.15.98:0/3235374868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1c8107290 0x7fc1c810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.788+0000 7fc1cfd19640 1 --2- 172.21.15.98:0/3235374868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1c8102b70 0x7fc1c8106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.788+0000 7fc1cfd19640 1 --2- 172.21.15.98:0/3235374868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c81021c0 0x7fc1c81025a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.788+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3235374868 >> 172.21.15.98:0/3235374868 conn(0x7fc1c80fda20 msgr2=0x7fc1c80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.788+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3235374868 shutdown_connections 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3235374868 wait complete. 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cfd19640 1 Processor -- start 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cfd19640 1 -- start start 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cfd19640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1c81021c0 0x7fc1c8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cfd19640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c8107290 0x7fc1c8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cfd19640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1c8074cd0 0x7fc1c81a6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cfd19640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c81121d0 con 0x7fc1c8107290 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cfd19640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c8112050 con 0x7fc1c81021c0 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cfd19640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c8112350 con 0x7fc1c8074cd0 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cd2b4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c8107290 0x7fc1c8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cd2b4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c8107290 0x7fc1c8074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35686/0 (socket says 172.21.15.98:35686) 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.789+0000 7fc1cd2b4640 1 -- 172.21.15.98:0/3342949232 learned_addr learned my addr 172.21.15.98:0/3342949232 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1ce2b6640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1c8074cd0 0x7fc1c81a6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1cdab5640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1c81021c0 0x7fc1c8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1cd2b4640 1 -- 172.21.15.98:0/3342949232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1c8074cd0 msgr2=0x7fc1c81a6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1c8074cd0 0x7fc1c81a6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1cd2b4640 1 -- 172.21.15.98:0/3342949232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1c81021c0 msgr2=0x7fc1c8074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1c81021c0 0x7fc1c8074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1cd2b4640 1 -- 172.21.15.98:0/3342949232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c81a6fa0 con 0x7fc1c8107290 2024-04-25T17:37:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1cdab5640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1c81021c0 0x7fc1c8074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:37:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c8107290 0x7fc1c8074790 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800d770 tx=0x7fc1b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1beffd640 1 -- 172.21.15.98:0/3342949232 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8017070 con 0x7fc1c8107290 2024-04-25T17:37:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1beffd640 1 -- 172.21.15.98:0/3342949232 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1b8009c30 con 0x7fc1c8107290 2024-04-25T17:37:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3342949232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c81a7290 con 0x7fc1c8107290 2024-04-25T17:37:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1beffd640 1 -- 172.21.15.98:0/3342949232 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b800f620 con 0x7fc1c8107290 2024-04-25T17:37:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.790+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3342949232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c81a77e0 con 0x7fc1c8107290 2024-04-25T17:37:15.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.792+0000 7fc1beffd640 1 -- 172.21.15.98:0/3342949232 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fc1b801a070 con 0x7fc1c8107290 2024-04-25T17:37:15.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.793+0000 7fc1d2a2c740 1 Processor -- start 2024-04-25T17:37:15.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.793+0000 7fc1d2a2c740 1 -- start start 2024-04-25T17:37:15.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.793+0000 7fc1beffd640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc1b40a3ee0 0x7fc1b40a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.793+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dc620 0x5614608dca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.793+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608dcf40 0x5614608e72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.793+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608e7800 0x5614608e9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561460841970 con 0x5614608e7800 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5614607de4a0 con 0x5614608dcf40 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56146083d930 con 0x5614608dc620 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc1b40a3ee0 0x7fc1b40a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1beffd640 1 -- 172.21.15.98:0/3342949232 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fc1b80d06a0 con 0x7fc1c8107290 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608e7800 0x5614608e9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608e7800 0x5614608e9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35690/0 (socket says 172.21.15.98:35690) 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 -- 172.21.15.98:0/1890365471 learned_addr learned my addr 172.21.15.98:0/1890365471 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1cdab5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dc620 0x5614608dca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/1890365471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608dcf40 0x5614608e72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 -- 172.21.15.98:0/1890365471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dc620 msgr2=0x5614608dca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 --2- 172.21.15.98:0/1890365471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dc620 0x5614608dca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 -- 172.21.15.98:0/1890365471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608dcf40 msgr2=0x5614608e72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 --2- 172.21.15.98:0/1890365471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608dcf40 0x5614608e72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 -- 172.21.15.98:0/1890365471 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561460554210 con 0x5614608e7800 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc1b40a3ee0 0x7fc1b40a63a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1c000bfd0 tx=0x7fc1c000bd40 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1ce2b6640 1 --2- 172.21.15.98:0/1890365471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608e7800 0x5614608e9bf0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0010d90 tx=0x7fc1c001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f21082646bf0ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1b3fff640 1 -- 172.21.15.98:0/1890365471 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c000ee60 con 0x5614608e7800 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1b3fff640 1 -- 172.21.15.98:0/1890365471 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1c0010e80 con 0x5614608e7800 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.794+0000 7fc1b3fff640 1 -- 172.21.15.98:0/1890365471 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0003d90 con 0x5614608e7800 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.795+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/1890365471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608e7800 msgr2=0x5614608e9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.795+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/1890365471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608e7800 0x5614608e9bf0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0010d90 tx=0x7fc1c001a000 comp rx=0 tx=0).stop 2024-04-25T17:37:15.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.795+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/1890365471 shutdown_connections 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.795+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/1890365471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608e7800 0x5614608e9bf0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.795+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/1890365471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608dcf40 0x5614608e72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.795+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/1890365471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dc620 0x5614608dca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.795+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/1890365471 >> 172.21.15.98:0/1890365471 conn(0x5614608d8b30 msgr2=0x5614608d8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.795+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/1890365471 shutdown_connections 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.795+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/1890365471 wait complete. 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1d2a2c740 1 -- start start 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1d2a2c740 1 Processor -- start 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608e7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dcf40 0x5614608e4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5614608ea3d0 con 0x5614608dc620 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5614607dc0b0 con 0x5614608e7800 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5614608ea550 con 0x5614608dcf40 2024-04-25T17:37:15.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1d2a2c740 10 client.0 _ll_register_callbacks cb 0x5614607c9d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1cdab5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1cdab5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608e7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35702/0 (socket says 172.21.15.98:35702) 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1cdab5640 1 -- 172.21.15.98:0/483758317 learned_addr learned my addr 172.21.15.98:0/483758317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.796+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dcf40 0x5614608e4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1ce2b6640 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1cdab5640 1 -- 172.21.15.98:0/483758317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dcf40 msgr2=0x5614608e4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1cdab5640 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dcf40 0x5614608e4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1cdab5640 1 -- 172.21.15.98:0/483758317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 msgr2=0x5614608e49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1cdab5640 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608e49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1cdab5640 1 -- 172.21.15.98:0/483758317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561460620620 con 0x5614608dc620 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1cdab5640 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608e7170 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc1c402a930 tx=0x7fc1c40318a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c783cc3f1da0a45a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1b17fa640 1 -- 172.21.15.98:0/483758317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4035040 con 0x5614608dc620 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1b17fa640 1 -- 172.21.15.98:0/483758317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fc1c4031d10 con 0x5614608dc620 2024-04-25T17:37:15.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/483758317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56146084f390 con 0x5614608dc620 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1b17fa640 1 -- 172.21.15.98:0/483758317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc1c4004100 con 0x5614608dc620 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1d2a2c740 10 client.15444 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/483758317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614608f6390 con 0x5614608dc620 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1b17fa640 1 -- 172.21.15.98:0/483758317 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56146084f390 con 0x5614608dc620 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1b17fa640 1 -- 172.21.15.98:0/483758317 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fc1c4004730 con 0x5614608dc620 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1d2a2c740 10 client.15444 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.797+0000 7fc1d2a2c740 10 client.15444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.798+0000 7fc1d2a2c740 20 client.15444 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.798+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5614608f7800 0x5614608f9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.798+0000 7fc1d2a2c740 4 client.15444 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.798+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/483758317 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561460849200 con 0x5614608f7800 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.798+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5614608f7800 0x5614608f9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.798+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5614608f7800 0x5614608f9bf0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ba5a96d8397c93a server_cookie=9ee2bdfa74c4886e in_seq=0 out_seq=0 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.798+0000 7fc1b17fa640 10 client.15444 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1b17fa640 1 -- 172.21.15.98:0/483758317 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561460849200 con 0x5614608f7800 2024-04-25T17:37:15.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1b17fa640 10 client.15444 handle_command_reply: tid=0 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 client.15444 shutdown 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/483758317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 msgr2=0x5614608e7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608e7170 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc1c402a930 tx=0x7fc1c40318a0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/483758317 shutdown_connections 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5614608f7800 0x5614608f9bf0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1b17fa640 0 client.15444 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dcf40 0x5614608e4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/483758317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608e7170 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/483758317 >> 172.21.15.98:0/483758317 conn(0x5614608d8b30 msgr2=0x5614608de8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/483758317 shutdown_connections 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/483758317 wait complete. 2024-04-25T17:37:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.799+0000 7fc1d2a2c740 20 client.15444 trim_cache size 0 max 16384 2024-04-25T17:37:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.814+0000 7fc1d2a2c740 1 Processor -- start 2024-04-25T17:37:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.814+0000 7fc1d2a2c740 1 -- start start 2024-04-25T17:37:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.814+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dc620 0x5614608f3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.814+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dcf40 0x5614608ff500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.814+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.819 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.814+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c81b24e0 con 0x5614608dcf40 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.814+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c81afd80 con 0x5614608e7800 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.814+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c81adbd0 con 0x5614608dc620 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1cd2b4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dcf40 0x5614608ff500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1cd2b4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dcf40 0x5614608ff500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35716/0 (socket says 172.21.15.98:35716) 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1cd2b4640 1 -- 172.21.15.98:0/436552093 learned_addr learned my addr 172.21.15.98:0/436552093 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1cdab5640 1 --2- 172.21.15.98:0/436552093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dc620 0x5614608f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1cd2b4640 1 -- 172.21.15.98:0/436552093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dc620 msgr2=0x5614608f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/436552093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dc620 0x5614608f3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1cd2b4640 1 -- 172.21.15.98:0/436552093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 msgr2=0x5614608ffa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/436552093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1cd2b4640 1 -- 172.21.15.98:0/436552093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614608f9f60 con 0x5614608dcf40 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/436552093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dcf40 0x5614608ff500 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc1b80ce420 tx=0x7fc1b8098040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff689ba45be6032c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1b37fe640 1 -- 172.21.15.98:0/436552093 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8021040 con 0x5614608dcf40 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1b37fe640 1 -- 172.21.15.98:0/436552093 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1b80a07d0 con 0x5614608dcf40 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.815+0000 7fc1b37fe640 1 -- 172.21.15.98:0/436552093 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8096120 con 0x5614608dcf40 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/436552093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dcf40 msgr2=0x5614608ff500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/436552093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dcf40 0x5614608ff500 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc1b80ce420 tx=0x7fc1b8098040 comp rx=0 tx=0).stop 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/436552093 shutdown_connections 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/436552093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/436552093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dcf40 0x5614608ff500 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/436552093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dc620 0x5614608f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/436552093 >> 172.21.15.98:0/436552093 conn(0x5614608d8b30 msgr2=0x5614608d7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/436552093 shutdown_connections 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/436552093 wait complete. 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 -- start start 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 Processor -- start 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608ed0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dcf40 0x5614608ed620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608edb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c81ae240 con 0x5614608dc620 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c81af6b0 con 0x5614608e7800 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.816+0000 7fc1d2a2c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c81ae910 con 0x5614608dcf40 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1d2a2c740 10 client.0 _ll_register_callbacks cb 0x7fc1c4032f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1cdab5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608ed0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1cdab5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608ed0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35720/0 (socket says 172.21.15.98:35720) 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1cdab5640 1 -- 172.21.15.98:0/805242956 learned_addr learned my addr 172.21.15.98:0/805242956 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dcf40 0x5614608ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1ce2b6640 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608edb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1cdab5640 1 -- 172.21.15.98:0/805242956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dcf40 msgr2=0x5614608ed620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1cdab5640 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dcf40 0x5614608ed620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1cdab5640 1 -- 172.21.15.98:0/805242956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 msgr2=0x5614608edb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1cdab5640 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608edb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1cdab5640 1 -- 172.21.15.98:0/805242956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614608e9f60 con 0x5614608dc620 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1cdab5640 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608ed0e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009850 tx=0x7fc1c40035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803100fca1adcd86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1b17fa640 1 -- 172.21.15.98:0/805242956 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4035040 con 0x5614608dc620 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1b17fa640 1 -- 172.21.15.98:0/805242956 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fc1c4003780 con 0x5614608dc620 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/805242956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1b80a07d0 con 0x5614608dc620 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1b17fa640 1 -- 172.21.15.98:0/805242956 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc1c40044b0 con 0x5614608dc620 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1d2a2c740 10 client.15450 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/805242956 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56146095de20 con 0x5614608dc620 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.817+0000 7fc1b17fa640 1 -- 172.21.15.98:0/805242956 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1b80a07d0 con 0x5614608dc620 2024-04-25T17:37:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.818+0000 7fc1b17fa640 1 -- 172.21.15.98:0/805242956 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fc1c4055030 con 0x5614608dc620 2024-04-25T17:37:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.818+0000 7fc1d2a2c740 10 client.15450 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.818+0000 7fc1d2a2c740 10 client.15450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:37:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.818+0000 7fc1d2a2c740 20 client.15450 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.818+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56146095e600 0x561460960a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.818+0000 7fc1d2a2c740 4 client.15450 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:37:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.818+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/805242956 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614607c55f0 con 0x56146095e600 2024-04-25T17:37:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.818+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56146095e600 0x561460960a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.818+0000 7fc1cd2b4640 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56146095e600 0x561460960a20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fb8ba5732c20c94 server_cookie=af39387c6e74ca6a in_seq=0 out_seq=0 2024-04-25T17:37:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.818+0000 7fc1b17fa640 10 client.15450 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1b17fa640 1 -- 172.21.15.98:0/805242956 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614607c55f0 con 0x56146095e600 2024-04-25T17:37:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1b17fa640 10 client.15450 handle_command_reply: tid=0 2024-04-25T17:37:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 client.15450 shutdown 2024-04-25T17:37:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/805242956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 msgr2=0x5614608ed0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608ed0e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009850 tx=0x7fc1c40035f0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/805242956 shutdown_connections 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56146095e600 0x561460960a20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5614608e7800 0x5614608edb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614608dcf40 0x5614608ed620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1b17fa640 0 client.15450 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 --2- 172.21.15.98:0/805242956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5614608dc620 0x5614608ed0e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/805242956 >> 172.21.15.98:0/805242956 conn(0x5614608d8b30 msgr2=0x5614608f2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/805242956 shutdown_connections 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 1 -- 172.21.15.98:0/805242956 wait complete. 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1d2a2c740 20 client.15450 trim_cache size 0 max 16384 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:37:15.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:37:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:37:15.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3342949232 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc1b40a3ee0 msgr2=0x7fc1b40a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc1b40a3ee0 0x7fc1b40a63a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1c000bfd0 tx=0x7fc1c000bd40 comp rx=0 tx=0).stop 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3342949232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c8107290 msgr2=0x7fc1c8074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c8107290 0x7fc1c8074790 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800d770 tx=0x7fc1b800dc30 comp rx=0 tx=0).stop 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3342949232 shutdown_connections 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc1b40a3ee0 0x7fc1b40a63a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1c8074cd0 0x7fc1c81a6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c8107290 0x7fc1c8074790 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 --2- 172.21.15.98:0/3342949232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1c81021c0 0x7fc1c8074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3342949232 >> 172.21.15.98:0/3342949232 conn(0x7fc1c80fda20 msgr2=0x7fc1c80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3342949232 shutdown_connections 2024-04-25T17:37:15.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:15.835+0000 7fc1cfd19640 1 -- 172.21.15.98:0/3342949232 wait complete. 2024-04-25T17:37:16.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:16 smithi098 ceph-mon[29192]: pgmap v43: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.2 MiB/s wr, 56 op/s 2024-04-25T17:37:16.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:16 smithi203 ceph-mon[32834]: pgmap v43: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.2 MiB/s wr, 56 op/s 2024-04-25T17:37:16.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:16 smithi173 ceph-mon[32856]: pgmap v43: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.2 MiB/s wr, 56 op/s 2024-04-25T17:37:16.599 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 45.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:37:18.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:18 smithi098 ceph-mon[29192]: pgmap v44: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 58 op/s 2024-04-25T17:37:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:18 smithi203 ceph-mon[32834]: pgmap v44: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 58 op/s 2024-04-25T17:37:18.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:18 smithi173 ceph-mon[32856]: pgmap v44: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 58 op/s 2024-04-25T17:37:20.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:20 smithi098 ceph-mon[29192]: pgmap v45: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-04-25T17:37:20.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:20 smithi203 ceph-mon[32834]: pgmap v45: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-04-25T17:37:20.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:20 smithi173 ceph-mon[32856]: pgmap v45: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-04-25T17:37:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 50.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:37:22.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:37:22.447 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:22 smithi098 ceph-mon[29192]: pgmap v46: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-04-25T17:37:22.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:22 smithi203 ceph-mon[32834]: pgmap v46: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-04-25T17:37:22.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:22 smithi173 ceph-mon[32856]: pgmap v46: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 KiB/s wr, 13 op/s 2024-04-25T17:37:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:37:24.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:24 smithi098 ceph-mon[29192]: pgmap v47: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-04-25T17:37:24.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:24 smithi203 ceph-mon[32834]: pgmap v47: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-04-25T17:37:24.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:24 smithi173 ceph-mon[32856]: pgmap v47: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-04-25T17:37:24.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.535+0000 7f1433f86700 1 -- 172.21.15.98:0/3753795879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142c102d50 msgr2=0x7f142c1065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.535+0000 7f1433f86700 1 --2- 172.21.15.98:0/3753795879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142c102d50 0x7f142c1065e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f141c0099c0 tx=0x7f141c009cd0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.535+0000 7f1433f86700 1 -- 172.21.15.98:0/3753795879 shutdown_connections 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.535+0000 7f1433f86700 1 --2- 172.21.15.98:0/3753795879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f142c1023a0 0x7f142c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.535+0000 7f1433f86700 1 --2- 172.21.15.98:0/3753795879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f142c106cc0 0x7f142c10e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.535+0000 7f1433f86700 1 --2- 172.21.15.98:0/3753795879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142c102d50 0x7f142c1065e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.535+0000 7f1433f86700 1 -- 172.21.15.98:0/3753795879 >> 172.21.15.98:0/3753795879 conn(0x7f142c077fb0 msgr2=0x7f142c0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.535+0000 7f1433f86700 1 -- 172.21.15.98:0/3753795879 shutdown_connections 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.535+0000 7f1433f86700 1 -- 172.21.15.98:0/3753795879 wait complete. 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1433f86700 1 Processor -- start 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1433f86700 1 -- start start 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1433f86700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142c1023a0 0x7f142c199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1433f86700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f142c102d50 0x7f142c19a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1433f86700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f142c106cc0 0x7f142c1a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1433f86700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f142c111b70 con 0x7f142c1023a0 2024-04-25T17:37:24.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1433f86700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f142c1119f0 con 0x7f142c106cc0 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1433f86700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f142c111cf0 con 0x7f142c102d50 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1431d22700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142c1023a0 0x7f142c199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1431d22700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142c1023a0 0x7f142c199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55598/0 (socket says 172.21.15.98:55598) 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1431d22700 1 -- 172.21.15.98:0/1253618651 learned_addr learned my addr 172.21.15.98:0/1253618651 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1432523700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f142c106cc0 0x7f142c1a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1431521700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f142c102d50 0x7f142c19a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1431d22700 1 -- 172.21.15.98:0/1253618651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f142c102d50 msgr2=0x7f142c19a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1431d22700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f142c102d50 0x7f142c19a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1431d22700 1 -- 172.21.15.98:0/1253618651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f142c106cc0 msgr2=0x7f142c1a5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.536+0000 7f1431d22700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f142c106cc0 0x7f142c1a5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.537+0000 7f1431d22700 1 -- 172.21.15.98:0/1253618651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142c1a5590 con 0x7f142c1023a0 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.537+0000 7f1431d22700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142c1023a0 0x7f142c199fd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f142800c9b0 tx=0x7f142800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.537+0000 7f1422ffd700 1 -- 172.21.15.98:0/1253618651 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1428007b00 con 0x7f142c1023a0 2024-04-25T17:37:24.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.537+0000 7f1433f86700 1 -- 172.21.15.98:0/1253618651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142c1a57c0 con 0x7f142c1023a0 2024-04-25T17:37:24.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.537+0000 7f1422ffd700 1 -- 172.21.15.98:0/1253618651 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f142800ce50 con 0x7f142c1023a0 2024-04-25T17:37:24.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.537+0000 7f1433f86700 1 -- 172.21.15.98:0/1253618651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f142c1a5d20 con 0x7f142c1023a0 2024-04-25T17:37:24.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.537+0000 7f1422ffd700 1 -- 172.21.15.98:0/1253618651 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1428018660 con 0x7f142c1023a0 2024-04-25T17:37:24.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.538+0000 7f1433f86700 1 -- 172.21.15.98:0/1253618651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f142c069730 con 0x7f142c1023a0 2024-04-25T17:37:24.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.540+0000 7f1422ffd700 1 -- 172.21.15.98:0/1253618651 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f142800f450 con 0x7f142c1023a0 2024-04-25T17:37:24.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.541+0000 7f1422ffd700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f14180a3ed0 0x7f14180a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:24.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.541+0000 7f1431521700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f14180a3ed0 0x7f14180a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:24.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.541+0000 7f1422ffd700 1 -- 172.21.15.98:0/1253618651 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f14280d0c40 con 0x7f142c1023a0 2024-04-25T17:37:24.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.541+0000 7f1431521700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f14180a3ed0 0x7f14180a6390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f141c006010 tx=0x7f141c005c20 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:24.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.544+0000 7f1422ffd700 1 -- 172.21.15.98:0/1253618651 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1428014030 con 0x7f142c1023a0 2024-04-25T17:37:24.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.937+0000 7f1433f86700 1 -- 172.21.15.98:0/1253618651 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f142c02df80 con 0x7f142c1023a0 2024-04-25T17:37:24.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.937+0000 7f1422ffd700 1 -- 172.21.15.98:0/1253618651 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11881 (secure 0 0 0) 0x7f142809d150 con 0x7f142c1023a0 2024-04-25T17:37:24.939 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:37:24.939 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:36:39.350510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:37:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 -- 172.21.15.98:0/1253618651 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f14180a3ed0 msgr2=0x7f14180a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:24.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f14180a3ed0 0x7f14180a6390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f141c006010 tx=0x7f141c005c20 comp rx=0 tx=0).stop 2024-04-25T17:37:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 -- 172.21.15.98:0/1253618651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142c1023a0 msgr2=0x7f142c199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142c1023a0 0x7f142c199fd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f142800c9b0 tx=0x7f142800ccc0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 -- 172.21.15.98:0/1253618651 shutdown_connections 2024-04-25T17:37:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f14180a3ed0 0x7f14180a6390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f142c106cc0 0x7f142c1a5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f142c1023a0 0x7f142c199fd0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 --2- 172.21.15.98:0/1253618651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f142c102d50 0x7f142c19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 -- 172.21.15.98:0/1253618651 >> 172.21.15.98:0/1253618651 conn(0x7f142c077fb0 msgr2=0x7f142c0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 -- 172.21.15.98:0/1253618651 shutdown_connections 2024-04-25T17:37:24.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:24.940+0000 7f1433f86700 1 -- 172.21.15.98:0/1253618651 wait complete. 2024-04-25T17:37:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-04-25T17:37:25.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:25 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1253618651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:37:25.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:25 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1253618651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:37:25.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:25 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1253618651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:37:25.743 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id caaeb197 --release --await' 2024-04-25T17:37:25.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id caaeb197 --release --await 2024-04-25T17:37:26.028 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:37:26.394 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:26 smithi098 ceph-mon[29192]: pgmap v48: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-04-25T17:37:26.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:26 smithi203 ceph-mon[32834]: pgmap v48: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-04-25T17:37:26.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:26 smithi173 ceph-mon[32856]: pgmap v48: 97 pgs: 97 active+clean; 958 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.6 KiB/s wr, 8 op/s 2024-04-25T17:37:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 55.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.643+0000 7f19d2ccf700 1 -- 172.21.15.98:0/1322560013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19cc108a10 msgr2=0x7f19cc108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.643+0000 7f19d2ccf700 1 --2- 172.21.15.98:0/1322560013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19cc108a10 0x7f19cc108df0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f19c0009990 tx=0x7f19c0009ca0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.643+0000 7f19d2ccf700 1 -- 172.21.15.98:0/1322560013 shutdown_connections 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.643+0000 7f19d2ccf700 1 --2- 172.21.15.98:0/1322560013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19cc1034c0 0x7f19cc10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.643+0000 7f19d2ccf700 1 --2- 172.21.15.98:0/1322560013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19cc108a10 0x7f19cc108df0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.643+0000 7f19d2ccf700 1 --2- 172.21.15.98:0/1322560013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19cc102b00 0x7f19cc102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.643+0000 7f19d2ccf700 1 -- 172.21.15.98:0/1322560013 >> 172.21.15.98:0/1322560013 conn(0x7f19cc079c00 msgr2=0x7f19cc07a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.643+0000 7f19d2ccf700 1 -- 172.21.15.98:0/1322560013 shutdown_connections 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.643+0000 7f19d2ccf700 1 -- 172.21.15.98:0/1322560013 wait complete. 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d2ccf700 1 Processor -- start 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d2ccf700 1 -- start start 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d2ccf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19cc102b00 0x7f19cc1990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d2ccf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19cc1034c0 0x7f19cc199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d2ccf700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19cc108a10 0x7f19cc19d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d2ccf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc112c10 con 0x7f19cc1034c0 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d2ccf700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc112a90 con 0x7f19cc108a10 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d2ccf700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc112d90 con 0x7f19cc102b00 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19cc1034c0 0x7f19cc199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19cc1034c0 0x7f19cc199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55618/0 (socket says 172.21.15.98:55618) 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19cbfff700 1 -- 172.21.15.98:0/799461219 learned_addr learned my addr 172.21.15.98:0/799461219 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19cc102b00 0x7f19cc1990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d126c700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19cc108a10 0x7f19cc19d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19cbfff700 1 -- 172.21.15.98:0/799461219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19cc102b00 msgr2=0x7f19cc1990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19cbfff700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19cc102b00 0x7f19cc1990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19cbfff700 1 -- 172.21.15.98:0/799461219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19cc108a10 msgr2=0x7f19cc19d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19cbfff700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19cc108a10 0x7f19cc19d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19cbfff700 1 -- 172.21.15.98:0/799461219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19cc19e0d0 con 0x7f19cc1034c0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.644+0000 7f19d126c700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19cc108a10 0x7f19cc19d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.645+0000 7f19cbfff700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19cc1034c0 0x7f19cc199600 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f19b800c9b0 tx=0x7f19b800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.645+0000 7f19c9ffb700 1 -- 172.21.15.98:0/799461219 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8007b00 con 0x7f19cc1034c0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.645+0000 7f19c9ffb700 1 -- 172.21.15.98:0/799461219 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b8007ca0 con 0x7f19cc1034c0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.645+0000 7f19d2ccf700 1 -- 172.21.15.98:0/799461219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19cc19e3c0 con 0x7f19cc1034c0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.645+0000 7f19c9ffb700 1 -- 172.21.15.98:0/799461219 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8018660 con 0x7f19cc1034c0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.645+0000 7f19d2ccf700 1 -- 172.21.15.98:0/799461219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19cc1a5c40 con 0x7f19cc1034c0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.647+0000 7f19c9ffb700 1 -- 172.21.15.98:0/799461219 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f19b800f450 con 0x7f19cc1034c0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.647+0000 7f19c9ffb700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f19bc0a3db0 0x7f19bc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.648+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f19bc0a3db0 0x7f19bc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.648+0000 7f19c9ffb700 1 -- 172.21.15.98:0/799461219 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f19b80d07a0 con 0x7f19cc1034c0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.648+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f19bc0a3db0 0x7f19bc0a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f19c000b5c0 tx=0x7f19c0005960 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:27.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.648+0000 7f19dd558b80 1 Processor -- start 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.648+0000 7f19dd558b80 1 -- start start 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec937650 0x55afec937a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec942930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 0x55afec945260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19dd558b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55afec88ad90 con 0x55afec937f70 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19dd558b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55afec831a30 con 0x55afec942e70 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19dd558b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55afec886d50 con 0x55afec937650 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec942930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec942930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55626/0 (socket says 172.21.15.98:55626) 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19cbfff700 1 -- 172.21.15.98:0/3496185262 learned_addr learned my addr 172.21.15.98:0/3496185262 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/3496185262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec937650 0x55afec937a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19d126c700 1 --2- 172.21.15.98:0/3496185262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 0x55afec945260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19cbfff700 1 -- 172.21.15.98:0/3496185262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec937650 msgr2=0x55afec937a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:27.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19cbfff700 1 --2- 172.21.15.98:0/3496185262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec937650 0x55afec937a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19cbfff700 1 -- 172.21.15.98:0/3496185262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 msgr2=0x55afec945260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19cbfff700 1 --2- 172.21.15.98:0/3496185262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 0x55afec945260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19cbfff700 1 -- 172.21.15.98:0/3496185262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afec7f5840 con 0x55afec937f70 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19d126c700 1 --2- 172.21.15.98:0/3496185262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 0x55afec945260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19cbfff700 1 --2- 172.21.15.98:0/3496185262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec942930 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f19b80a4eb0 tx=0x7f19b809c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737c7f7e973c54c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19b6ffd700 1 -- 172.21.15.98:0/3496185262 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b809c770 con 0x55afec937f70 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19b6ffd700 1 -- 172.21.15.98:0/3496185262 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b80a0b20 con 0x55afec937f70 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.649+0000 7f19b6ffd700 1 -- 172.21.15.98:0/3496185262 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b809c9d0 con 0x55afec937f70 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.650+0000 7f19dd558b80 1 -- 172.21.15.98:0/3496185262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 msgr2=0x55afec942930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.650+0000 7f19dd558b80 1 --2- 172.21.15.98:0/3496185262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec942930 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f19b80a4eb0 tx=0x7f19b809c330 comp rx=0 tx=0).stop 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.650+0000 7f19dd558b80 1 -- 172.21.15.98:0/3496185262 shutdown_connections 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.650+0000 7f19dd558b80 1 --2- 172.21.15.98:0/3496185262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec937650 0x55afec937a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.650+0000 7f19dd558b80 1 --2- 172.21.15.98:0/3496185262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 0x55afec945260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.650+0000 7f19dd558b80 1 --2- 172.21.15.98:0/3496185262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec942930 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.650+0000 7f19dd558b80 1 -- 172.21.15.98:0/3496185262 >> 172.21.15.98:0/3496185262 conn(0x55afec864990 msgr2=0x55afec933b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.650+0000 7f19dd558b80 1 -- 172.21.15.98:0/3496185262 shutdown_connections 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.650+0000 7f19dd558b80 1 -- 172.21.15.98:0/3496185262 wait complete. 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19dd558b80 1 -- start start 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19dd558b80 1 Processor -- start 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937650 0x55afec942650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec937f70 0x55afec93f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 0x55afec93ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19dd558b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55afec945900 con 0x55afec937650 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19dd558b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55afec8446b0 con 0x55afec942e70 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19dd558b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55afec945a80 con 0x55afec937f70 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19dd558b80 10 client.0 _ll_register_callbacks cb 0x55afec823a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19d0a6b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937650 0x55afec942650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19d0a6b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937650 0x55afec942650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55630/0 (socket says 172.21.15.98:55630) 2024-04-25T17:37:27.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19d0a6b700 1 -- 172.21.15.98:0/697106204 learned_addr learned my addr 172.21.15.98:0/697106204 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19d126c700 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 0x55afec93ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19cbfff700 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec937f70 0x55afec93f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19d0a6b700 1 -- 172.21.15.98:0/697106204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec937f70 msgr2=0x55afec93f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec937f70 0x55afec93f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19d0a6b700 1 -- 172.21.15.98:0/697106204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 msgr2=0x55afec93ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 0x55afec93ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.651+0000 7f19d0a6b700 1 -- 172.21.15.98:0/697106204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afec7d89d0 con 0x55afec937650 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937650 0x55afec942650 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f19c0000c00 tx=0x7f19c0009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32622ec14e4f3f6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19abfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19abfff700 1 -- 172.21.15.98:0/697106204 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c0036060 con 0x55afec937650 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19abfff700 1 -- 172.21.15.98:0/697106204 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f19c0031680 con 0x55afec937650 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19dd558b80 1 -- 172.21.15.98:0/697106204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19b80a0b20 con 0x55afec937650 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19abfff700 1 -- 172.21.15.98:0/697106204 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f19c0031880 con 0x55afec937650 2024-04-25T17:37:27.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19dd558b80 10 client.15468 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19dd558b80 1 -- 172.21.15.98:0/697106204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55afec950200 con 0x55afec937650 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19abfff700 1 -- 172.21.15.98:0/697106204 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b80a0b20 con 0x55afec937650 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19abfff700 1 -- 172.21.15.98:0/697106204 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f19c000f9f0 con 0x55afec937650 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19dd558b80 10 client.15468 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19dd558b80 10 client.15468 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19dd558b80 20 client.15468 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19dd558b80 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55afec952840 0x55afec954c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19dd558b80 4 client.15468 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.652+0000 7f19dd558b80 1 -- 172.21.15.98:0/697106204 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afec898580 con 0x55afec952840 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.653+0000 7f19cbfff700 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55afec952840 0x55afec954c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.653+0000 7f19cbfff700 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55afec952840 0x55afec954c30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6acd7e6a0741449 server_cookie=66bf2d80878d9a48 in_seq=0 out_seq=0 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.653+0000 7f19abfff700 10 client.15468 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19abfff700 1 -- 172.21.15.98:0/697106204 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55afec898580 con 0x55afec952840 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19abfff700 10 client.15468 handle_command_reply: tid=0 2024-04-25T17:37:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 client.15468 shutdown 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 -- 172.21.15.98:0/697106204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937650 msgr2=0x55afec942650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937650 0x55afec942650 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f19c0000c00 tx=0x7f19c0009ca0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 -- 172.21.15.98:0/697106204 shutdown_connections 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55afec952840 0x55afec954c30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec942e70 0x55afec93ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19abfff700 0 client.15468 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937650 0x55afec942650 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 --2- 172.21.15.98:0/697106204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec937f70 0x55afec93f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 -- 172.21.15.98:0/697106204 >> 172.21.15.98:0/697106204 conn(0x55afec864990 msgr2=0x55afec939ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 -- 172.21.15.98:0/697106204 shutdown_connections 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 1 -- 172.21.15.98:0/697106204 wait complete. 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.654+0000 7f19dd558b80 20 client.15468 trim_cache size 0 max 16384 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.673+0000 7f19dd558b80 1 Processor -- start 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.673+0000 7f19dd558b80 1 -- start start 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.673+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 0x55afec94e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.673+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec95a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.673+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec942e70 0x55afec95a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.673+0000 7f19dd558b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc1afa80 con 0x55afec937f70 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.673+0000 7f19dd558b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc1ad2e0 con 0x55afec937650 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.673+0000 7f19dd558b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc1ab130 con 0x55afec942e70 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.673+0000 7f19cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec95a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec95a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55646/0 (socket says 172.21.15.98:55646) 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19cbfff700 1 -- 172.21.15.98:0/2886099113 learned_addr learned my addr 172.21.15.98:0/2886099113 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/2886099113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 0x55afec94e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19cbfff700 1 -- 172.21.15.98:0/2886099113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec942e70 msgr2=0x55afec95a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19d126c700 1 --2- 172.21.15.98:0/2886099113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec942e70 0x55afec95a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19cbfff700 1 --2- 172.21.15.98:0/2886099113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec942e70 0x55afec95a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19cbfff700 1 -- 172.21.15.98:0/2886099113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 msgr2=0x55afec94e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19cbfff700 1 --2- 172.21.15.98:0/2886099113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 0x55afec94e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 29, 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 4, 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: "caaeb197": { 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 4, 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:37:27.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 87.5, 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:37:27.735 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19cbfff700 1 -- 172.21.15.98:0/2886099113 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afec939500 con 0x55afec937f70 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/2886099113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 0x55afec94e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19cbfff700 1 --2- 172.21.15.98:0/2886099113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec95a480 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f19b80d0ac0 tx=0x7f19b800a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e41c65fc7bbdd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19b67fc700 1 -- 172.21.15.98:0/2886099113 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b80a53e0 con 0x55afec937f70 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19b67fc700 1 -- 172.21.15.98:0/2886099113 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b80870b0 con 0x55afec937f70 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.674+0000 7f19b67fc700 1 -- 172.21.15.98:0/2886099113 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b80a4f50 con 0x55afec937f70 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 -- 172.21.15.98:0/2886099113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 msgr2=0x55afec95a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 --2- 172.21.15.98:0/2886099113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec95a480 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f19b80d0ac0 tx=0x7f19b800a5f0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 -- 172.21.15.98:0/2886099113 shutdown_connections 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 --2- 172.21.15.98:0/2886099113 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec942e70 0x55afec95a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 --2- 172.21.15.98:0/2886099113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 0x55afec94e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 --2- 172.21.15.98:0/2886099113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec95a480 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 -- 172.21.15.98:0/2886099113 >> 172.21.15.98:0/2886099113 conn(0x55afec864990 msgr2=0x55afec94a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 -- 172.21.15.98:0/2886099113 shutdown_connections 2024-04-25T17:37:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 -- 172.21.15.98:0/2886099113 wait complete. 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 -- start start 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 Processor -- start 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 0x55afec9b6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec9b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.675+0000 7f19dd558b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec942e70 0x55afec9b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19dd558b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc1ab7a0 con 0x55afec937f70 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19dd558b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc1acc10 con 0x55afec937650 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19dd558b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc1abe70 con 0x55afec942e70 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19dd558b80 10 client.0 _ll_register_callbacks cb 0x55afec823a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec9b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec9b6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55648/0 (socket says 172.21.15.98:55648) 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19cbfff700 1 -- 172.21.15.98:0/1929189024 learned_addr learned my addr 172.21.15.98:0/1929189024 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19d126c700 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec942e70 0x55afec9b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 0x55afec9b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19cbfff700 1 -- 172.21.15.98:0/1929189024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec942e70 msgr2=0x55afec9b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19cbfff700 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec942e70 0x55afec9b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19cbfff700 1 -- 172.21.15.98:0/1929189024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 msgr2=0x55afec9b6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19cbfff700 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 0x55afec9b6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19cbfff700 1 -- 172.21.15.98:0/1929189024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afec9b13b0 con 0x55afec937f70 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19cbfff700 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec9b6c40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f19b80074a0 tx=0x7f19b80983d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b278c31562c9ac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19abfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19abfff700 1 -- 172.21.15.98:0/1929189024 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8065030 con 0x55afec937f70 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19abfff700 1 -- 172.21.15.98:0/1929189024 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f19b800f740 con 0x55afec937f70 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19dd558b80 1 -- 172.21.15.98:0/1929189024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19b80870b0 con 0x55afec937f70 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19abfff700 1 -- 172.21.15.98:0/1929189024 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f19b80bb050 con 0x55afec937f70 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19dd558b80 10 client.15474 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19dd558b80 1 -- 172.21.15.98:0/1929189024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55afec9bdf70 con 0x55afec937f70 2024-04-25T17:37:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.676+0000 7f19abfff700 1 -- 172.21.15.98:0/1929189024 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b80870b0 con 0x55afec937f70 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.677+0000 7f19abfff700 1 -- 172.21.15.98:0/1929189024 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f19b809ca60 con 0x55afec937f70 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.677+0000 7f19dd558b80 10 client.15474 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.677+0000 7f19dd558b80 10 client.15474 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.677+0000 7f19dd558b80 20 client.15474 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.677+0000 7f19dd558b80 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55afec9be190 0x55afec9c0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.677+0000 7f19dd558b80 4 client.15474 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "caaeb197", "release": true, "await": true}] 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.677+0000 7f19dd558b80 1 -- 172.21.15.98:0/1929189024 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "caaeb197", "release": true, "await": true}) -- 0x55afec892620 con 0x55afec9be190 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.677+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55afec9be190 0x55afec9c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.677+0000 7f19abfff700 10 client.15474 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.677+0000 7f19d0a6b700 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55afec9be190 0x55afec9c0580 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d74f1731b5daed2 server_cookie=40e96a3db4d77775 in_seq=0 out_seq=0 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19abfff700 1 -- 172.21.15.98:0/1929189024 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55afec892620 con 0x55afec9be190 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19abfff700 10 client.15474 handle_command_reply: tid=0 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19dd558b80 1 client.15474 shutdown 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19dd558b80 1 -- 172.21.15.98:0/1929189024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 msgr2=0x55afec9b6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19dd558b80 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec9b6c40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f19b80074a0 tx=0x7f19b80983d0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19dd558b80 1 -- 172.21.15.98:0/1929189024 shutdown_connections 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19dd558b80 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55afec9be190 0x55afec9c0580 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19dd558b80 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55afec942e70 0x55afec9b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19abfff700 0 client.15474 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19dd558b80 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55afec937650 0x55afec9b6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19dd558b80 1 --2- 172.21.15.98:0/1929189024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55afec937f70 0x55afec9b6c40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.715+0000 7f19dd558b80 1 -- 172.21.15.98:0/1929189024 >> 172.21.15.98:0/1929189024 conn(0x55afec864990 msgr2=0x55afec94a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.716+0000 7f19dd558b80 1 -- 172.21.15.98:0/1929189024 shutdown_connections 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.717+0000 7f19dd558b80 1 -- 172.21.15.98:0/1929189024 wait complete. 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.717+0000 7f19dd558b80 20 client.15474 trim_cache size 0 max 16384 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 -- 172.21.15.98:0/799461219 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f19bc0a3db0 msgr2=0x7f19bc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f19bc0a3db0 0x7f19bc0a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f19c000b5c0 tx=0x7f19c0005960 comp rx=0 tx=0).stop 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 -- 172.21.15.98:0/799461219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19cc1034c0 msgr2=0x7f19cc199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19cc1034c0 0x7f19cc199600 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f19b800c9b0 tx=0x7f19b800ccc0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 -- 172.21.15.98:0/799461219 shutdown_connections 2024-04-25T17:37:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f19bc0a3db0 0x7f19bc0a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19cc102b00 0x7f19cc1990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19cc108a10 0x7f19cc19d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 --2- 172.21.15.98:0/799461219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19cc1034c0 0x7f19cc199600 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 -- 172.21.15.98:0/799461219 >> 172.21.15.98:0/799461219 conn(0x7f19cc079c00 msgr2=0x7f19cc100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 -- 172.21.15.98:0/799461219 shutdown_connections 2024-04-25T17:37:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:27.718+0000 7f19d2ccf700 1 -- 172.21.15.98:0/799461219 wait complete. 2024-04-25T17:37:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:28 smithi098 ceph-mon[29192]: pgmap v49: 97 pgs: 97 active+clean; 960 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 15 MiB/s wr, 259 op/s 2024-04-25T17:37:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:28 smithi098 ceph-mon[29192]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T17:37:28.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:28 smithi203 ceph-mon[32834]: pgmap v49: 97 pgs: 97 active+clean; 960 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 15 MiB/s wr, 259 op/s 2024-04-25T17:37:28.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:28 smithi203 ceph-mon[32834]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T17:37:28.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:28 smithi173 ceph-mon[32856]: pgmap v49: 97 pgs: 97 active+clean; 960 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 15 MiB/s wr, 259 op/s 2024-04-25T17:37:28.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:28 smithi173 ceph-mon[32856]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T17:37:28.565 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'caaeb197', seconds elapsed: 43.4 2024-04-25T17:37:28.565 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 27.55 2024-04-25T17:37:28.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:37:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:37:28.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.947+0000 7f08444a0640 1 -- 172.21.15.98:0/2331172315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c101aa0 msgr2=0x7f083c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:28.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.947+0000 7f08444a0640 1 --2- 172.21.15.98:0/2331172315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c101aa0 0x7f083c105a90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f082c009900 tx=0x7f082c02a410 comp rx=0 tx=0).stop 2024-04-25T17:37:28.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.947+0000 7f08444a0640 1 -- 172.21.15.98:0/2331172315 shutdown_connections 2024-04-25T17:37:28.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.947+0000 7f08444a0640 1 --2- 172.21.15.98:0/2331172315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f083c106170 0x7f083c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.947+0000 7f08444a0640 1 --2- 172.21.15.98:0/2331172315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c101aa0 0x7f083c105a90 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.947+0000 7f08444a0640 1 --2- 172.21.15.98:0/2331172315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f083c1010f0 0x7f083c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.947+0000 7f08444a0640 1 -- 172.21.15.98:0/2331172315 >> 172.21.15.98:0/2331172315 conn(0x7f083c077fd0 msgr2=0x7f083c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.947+0000 7f08444a0640 1 -- 172.21.15.98:0/2331172315 shutdown_connections 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.947+0000 7f08444a0640 1 -- 172.21.15.98:0/2331172315 wait complete. 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f08444a0640 1 Processor -- start 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f08444a0640 1 -- start start 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f08444a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1010f0 0x7f083c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f08444a0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f083c106170 0x7f083c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f08444a0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f083c19d000 0x7f083c19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f08444a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f083c112220 con 0x7f083c1010f0 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f08444a0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f083c1120a0 con 0x7f083c19d000 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f08444a0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f083c1123a0 con 0x7f083c106170 2024-04-25T17:37:28.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f084223c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1010f0 0x7f083c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f0842a3d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f083c19d000 0x7f083c19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f0841a3b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f083c106170 0x7f083c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f084223c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1010f0 0x7f083c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55656/0 (socket says 172.21.15.98:55656) 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f084223c640 1 -- 172.21.15.98:0/3509169871 learned_addr learned my addr 172.21.15.98:0/3509169871 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f0842a3d640 1 -- 172.21.15.98:0/3509169871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f083c106170 msgr2=0x7f083c198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f0842a3d640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f083c106170 0x7f083c198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f0842a3d640 1 -- 172.21.15.98:0/3509169871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1010f0 msgr2=0x7f083c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f0842a3d640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1010f0 0x7f083c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f0842a3d640 1 -- 172.21.15.98:0/3509169871 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083c19daf0 con 0x7f083c19d000 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.948+0000 7f084223c640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1010f0 0x7f083c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:37:28.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.949+0000 7f0842a3d640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f083c19d000 0x7f083c19d4b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0834009fd0 tx=0x7f083400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.949+0000 7f08337fe640 1 -- 172.21.15.98:0/3509169871 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083400ce20 con 0x7f083c19d000 2024-04-25T17:37:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.949+0000 7f08337fe640 1 -- 172.21.15.98:0/3509169871 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0834005020 con 0x7f083c19d000 2024-04-25T17:37:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.949+0000 7f08337fe640 1 -- 172.21.15.98:0/3509169871 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834010690 con 0x7f083c19d000 2024-04-25T17:37:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.951+0000 7f08444a0640 1 -- 172.21.15.98:0/3509169871 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083c19dde0 con 0x7f083c19d000 2024-04-25T17:37:28.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.951+0000 7f08444a0640 1 -- 172.21.15.98:0/3509169871 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f083c1a54f0 con 0x7f083c19d000 2024-04-25T17:37:28.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.953+0000 7f08337fe640 1 -- 172.21.15.98:0/3509169871 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f08340054d0 con 0x7f083c19d000 2024-04-25T17:37:28.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.954+0000 7f08337fe640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f08280a3e40 0x7f08280a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.954+0000 7f08337fe640 1 -- 172.21.15.98:0/3509169871 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f08340d11c0 con 0x7f083c19d000 2024-04-25T17:37:28.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.954+0000 7f084223c640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f08280a3e40 0x7f08280a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.955+0000 7f08471bc740 1 Processor -- start 2024-04-25T17:37:28.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.955+0000 7f084223c640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f08280a3e40 0x7f08280a6300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0838011500 tx=0x7f0838005f90 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:28.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.955+0000 7f08471bc740 1 -- start start 2024-04-25T17:37:28.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.955+0000 7f08471bc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2b0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.955+0000 7f08471bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2bf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.955+0000 7f08471bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2bfc10 0x563a6d2c2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.955+0000 7f08471bc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563a6d215970 con 0x563a6d2b0520 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f08471bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563a6d1b24a0 con 0x563a6d2b0e40 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f08471bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563a6d211930 con 0x563a6d2bfc10 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f084223c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f084223c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2b0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55672/0 (socket says 172.21.15.98:55672) 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f084223c640 1 -- 172.21.15.98:0/3234852742 learned_addr learned my addr 172.21.15.98:0/3234852742 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f0841a3b640 1 --2- 172.21.15.98:0/3234852742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2bf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f0842a3d640 1 --2- 172.21.15.98:0/3234852742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2bfc10 0x563a6d2c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f084223c640 1 -- 172.21.15.98:0/3234852742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2bfc10 msgr2=0x563a6d2c2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f084223c640 1 --2- 172.21.15.98:0/3234852742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2bfc10 0x563a6d2c2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f084223c640 1 -- 172.21.15.98:0/3234852742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 msgr2=0x563a6d2bf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:28.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f084223c640 1 --2- 172.21.15.98:0/3234852742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2bf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f084223c640 1 -- 172.21.15.98:0/3234852742 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6cf28210 con 0x563a6d2b0520 2024-04-25T17:37:28.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f084223c640 1 --2- 172.21.15.98:0/3234852742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2b0900 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f083800f7c0 tx=0x7f0838014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0c6d4041830960 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f0830bfe640 1 -- 172.21.15.98:0/3234852742 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083802b070 con 0x563a6d2b0520 2024-04-25T17:37:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f0830bfe640 1 -- 172.21.15.98:0/3234852742 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f083801ce40 con 0x563a6d2b0520 2024-04-25T17:37:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.956+0000 7f0830bfe640 1 -- 172.21.15.98:0/3234852742 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838026e20 con 0x563a6d2b0520 2024-04-25T17:37:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.957+0000 7f08471bc740 1 -- 172.21.15.98:0/3234852742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 msgr2=0x563a6d2b0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.957+0000 7f08471bc740 1 --2- 172.21.15.98:0/3234852742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2b0900 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f083800f7c0 tx=0x7f0838014000 comp rx=0 tx=0).stop 2024-04-25T17:37:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.957+0000 7f08471bc740 1 -- 172.21.15.98:0/3234852742 shutdown_connections 2024-04-25T17:37:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.957+0000 7f08471bc740 1 --2- 172.21.15.98:0/3234852742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2bfc10 0x563a6d2c2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.958+0000 7f08471bc740 1 --2- 172.21.15.98:0/3234852742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2bf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.958+0000 7f08471bc740 1 --2- 172.21.15.98:0/3234852742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2b0900 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.958+0000 7f08471bc740 1 -- 172.21.15.98:0/3234852742 >> 172.21.15.98:0/3234852742 conn(0x563a6d2acb30 msgr2=0x563a6d2acf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.958+0000 7f08471bc740 1 -- 172.21.15.98:0/3234852742 shutdown_connections 2024-04-25T17:37:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.958+0000 7f08471bc740 1 -- 172.21.15.98:0/3234852742 wait complete. 2024-04-25T17:37:28.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.959+0000 7f08471bc740 1 -- start start 2024-04-25T17:37:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.959+0000 7f08471bc740 1 Processor -- start 2024-04-25T17:37:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.959+0000 7f08471bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2bf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.959+0000 7f08471bc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0e40 0x563a6d2b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f08471bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2bfc10 0x563a6d2b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f08471bc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563a6d2c2840 con 0x563a6d2b0e40 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f08471bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563a6d1b00b0 con 0x563a6d2bfc10 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f08471bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563a6d2c29c0 con 0x563a6d2b0520 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f08471bc740 10 client.0 _ll_register_callbacks cb 0x563a6d19dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f0841a3b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0e40 0x563a6d2b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f0841a3b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0e40 0x563a6d2b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55688/0 (socket says 172.21.15.98:55688) 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f0841a3b640 1 -- 172.21.15.98:0/655267758 learned_addr learned my addr 172.21.15.98:0/655267758 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f084223c640 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2bf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f0842a3d640 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2bfc10 0x563a6d2b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f084223c640 1 -- 172.21.15.98:0/655267758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2bfc10 msgr2=0x563a6d2b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f084223c640 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2bfc10 0x563a6d2b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f084223c640 1 -- 172.21.15.98:0/655267758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0e40 msgr2=0x563a6d2b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f084223c640 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0e40 0x563a6d2b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f084223c640 1 -- 172.21.15.98:0/655267758 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6cff4620 con 0x563a6d2b0520 2024-04-25T17:37:28.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f0841a3b640 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0e40 0x563a6d2b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f084223c640 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2bf1e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f083800f790 tx=0x7f083800fd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c14c0d89caed620 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f0825ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.960+0000 7f0825ffb640 1 -- 172.21.15.98:0/655267758 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083802b070 con 0x563a6d2b0520 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f0825ffb640 1 -- 172.21.15.98:0/655267758 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f083801c990 con 0x563a6d2b0520 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f08471bc740 1 -- 172.21.15.98:0/655267758 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563a6d223390 con 0x563a6d2b0520 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f0825ffb640 1 -- 172.21.15.98:0/655267758 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f083801cb90 con 0x563a6d2b0520 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f08471bc740 10 client.25054 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f08471bc740 1 -- 172.21.15.98:0/655267758 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563a6d2d2c10 con 0x563a6d2b0520 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f0825ffb640 1 -- 172.21.15.98:0/655267758 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563a6d223390 con 0x563a6d2b0520 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f0825ffb640 1 -- 172.21.15.98:0/655267758 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f08380046c0 con 0x563a6d2b0520 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f08471bc740 10 client.25054 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f08471bc740 10 client.25054 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f08471bc740 20 client.25054 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:28.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f08471bc740 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563a6d2d3f60 0x563a6d2d6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f08471bc740 4 client.25054 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f08471bc740 1 -- 172.21.15.98:0/655267758 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a6d21d200 con 0x563a6d2d3f60 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.961+0000 7f0841a3b640 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563a6d2d3f60 0x563a6d2d6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.962+0000 7f0825ffb640 10 client.25054 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.962+0000 7f0841a3b640 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563a6d2d3f60 0x563a6d2d6350 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdc2b0077a5d8881 server_cookie=4dae6edeb7293e18 in_seq=0 out_seq=0 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.962+0000 7f0825ffb640 1 -- 172.21.15.98:0/655267758 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a6d21d200 con 0x563a6d2d3f60 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.962+0000 7f0825ffb640 10 client.25054 handle_command_reply: tid=0 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.962+0000 7f08471bc740 1 client.25054 shutdown 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 1 -- 172.21.15.98:0/655267758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 msgr2=0x563a6d2bf1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2bf1e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f083800f790 tx=0x7f083800fd40 comp rx=0 tx=0).stop 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 1 -- 172.21.15.98:0/655267758 shutdown_connections 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563a6d2d3f60 0x563a6d2d6350 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2bfc10 0x563a6d2b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0e40 0x563a6d2b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f0825ffb640 0 client.25054 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 1 --2- 172.21.15.98:0/655267758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2bf1e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 1 -- 172.21.15.98:0/655267758 >> 172.21.15.98:0/655267758 conn(0x563a6d2acb30 msgr2=0x563a6d2b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 1 -- 172.21.15.98:0/655267758 shutdown_connections 2024-04-25T17:37:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 1 -- 172.21.15.98:0/655267758 wait complete. 2024-04-25T17:37:28.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.963+0000 7f08471bc740 20 client.25054 trim_cache size 0 max 16384 2024-04-25T17:37:28.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f08471bc740 1 Processor -- start 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f08471bc740 1 -- start start 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f08471bc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2db9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f08471bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2dbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f08471bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2bfc10 0x563a6d2bc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f08471bc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f083c1ae190 con 0x563a6d2b0520 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f08471bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f083c1aba30 con 0x563a6d2b0e40 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f08471bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f083c1a9880 con 0x563a6d2bfc10 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f084223c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2db9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f084223c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2db9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55698/0 (socket says 172.21.15.98:55698) 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f084223c640 1 -- 172.21.15.98:0/1400163075 learned_addr learned my addr 172.21.15.98:0/1400163075 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f0841a3b640 1 --2- 172.21.15.98:0/1400163075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2dbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.983+0000 7f0842a3d640 1 --2- 172.21.15.98:0/1400163075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2bfc10 0x563a6d2bc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.984+0000 7f084223c640 1 -- 172.21.15.98:0/1400163075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2bfc10 msgr2=0x563a6d2bc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:28.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.984+0000 7f084223c640 1 --2- 172.21.15.98:0/1400163075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2bfc10 0x563a6d2bc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.984+0000 7f084223c640 1 -- 172.21.15.98:0/1400163075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 msgr2=0x563a6d2dbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:28.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.984+0000 7f084223c640 1 --2- 172.21.15.98:0/1400163075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2dbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.984+0000 7f084223c640 1 -- 172.21.15.98:0/1400163075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6d2b1380 con 0x563a6d2b0520 2024-04-25T17:37:28.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.984+0000 7f0841a3b640 1 --2- 172.21.15.98:0/1400163075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2dbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:37:28.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.984+0000 7f084223c640 1 --2- 172.21.15.98:0/1400163075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2db9b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0838010010 tx=0x7f08380095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202aa3559de89b82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:28.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.984+0000 7f0827fff640 1 -- 172.21.15.98:0/1400163075 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083803e030 con 0x563a6d2b0520 2024-04-25T17:37:28.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.984+0000 7f0827fff640 1 -- 172.21.15.98:0/1400163075 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f083802d050 con 0x563a6d2b0520 2024-04-25T17:37:28.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.984+0000 7f0827fff640 1 -- 172.21.15.98:0/1400163075 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838020070 con 0x563a6d2b0520 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 -- 172.21.15.98:0/1400163075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 msgr2=0x563a6d2db9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 --2- 172.21.15.98:0/1400163075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2db9b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0838010010 tx=0x7f08380095b0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 -- 172.21.15.98:0/1400163075 shutdown_connections 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 --2- 172.21.15.98:0/1400163075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2bfc10 0x563a6d2bc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 --2- 172.21.15.98:0/1400163075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2dbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 --2- 172.21.15.98:0/1400163075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2b0520 0x563a6d2db9b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 -- 172.21.15.98:0/1400163075 >> 172.21.15.98:0/1400163075 conn(0x563a6d2acb30 msgr2=0x563a6d2bbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 -- 172.21.15.98:0/1400163075 shutdown_connections 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 -- 172.21.15.98:0/1400163075 wait complete. 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 -- start start 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.985+0000 7f08471bc740 1 Processor -- start 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f08471bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2cd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.990 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f08471bc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f08471bc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2bfc10 0x563a6d338220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f08471bc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f083c1a9ef0 con 0x563a6d2bfc10 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f08471bc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f083c1ab360 con 0x563a6d2b0e40 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f08471bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f083c1aa5c0 con 0x563a6d2b0520 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f08471bc740 10 client.0 _ll_register_callbacks cb 0x7f08340993d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f0842a3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2bfc10 0x563a6d338220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f0841a3b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f084223c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f084223c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2cd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:50956/0 (socket says 172.21.15.98:50956) 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f084223c640 1 -- 172.21.15.98:0/4079688137 learned_addr learned my addr 172.21.15.98:0/4079688137 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f0842a3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2bfc10 0x563a6d338220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55708/0 (socket says 172.21.15.98:55708) 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f0841a3b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:47556/0 (socket says 172.21.15.98:47556) 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f084223c640 1 -- 172.21.15.98:0/4079688137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 msgr2=0x563a6d2cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f084223c640 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f084223c640 1 -- 172.21.15.98:0/4079688137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2bfc10 msgr2=0x563a6d338220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:28.991 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f084223c640 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2bfc10 0x563a6d338220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f084223c640 1 -- 172.21.15.98:0/4079688137 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6d2c2000 con 0x563a6d2b0520 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f0842a3d640 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2bfc10 0x563a6d338220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f084223c640 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2cd790 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0838009f80 tx=0x7f0838009300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8179514501aa1d4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f0825ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.986+0000 7f0825ffb640 1 -- 172.21.15.98:0/4079688137 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838047060 con 0x563a6d2b0520 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f0825ffb640 1 -- 172.21.15.98:0/4079688137 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f0838026620 con 0x563a6d2b0520 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f08471bc740 1 -- 172.21.15.98:0/4079688137 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f083802d050 con 0x563a6d2b0520 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f0825ffb640 1 -- 172.21.15.98:0/4079688137 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f083803e070 con 0x563a6d2b0520 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f08471bc740 10 client.25060 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f08471bc740 1 -- 172.21.15.98:0/4079688137 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563a6d3396a0 con 0x563a6d2b0520 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f0825ffb640 1 -- 172.21.15.98:0/4079688137 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f083802d050 con 0x563a6d2b0520 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f0825ffb640 1 -- 172.21.15.98:0/4079688137 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f0838004fc0 con 0x563a6d2b0520 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f08471bc740 10 client.25060 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f08471bc740 10 client.25060 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f08471bc740 20 client.25060 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f08471bc740 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563a6d339d00 0x563a6d33c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:28.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f08471bc740 4 client.25060 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:37:28.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f08471bc740 1 -- 172.21.15.98:0/4079688137 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563a6d1995f0 con 0x563a6d339d00 2024-04-25T17:37:28.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.987+0000 7f0841a3b640 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563a6d339d00 0x563a6d33c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:28.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.988+0000 7f0825ffb640 10 client.25060 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:28.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.988+0000 7f0841a3b640 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563a6d339d00 0x563a6d33c120 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e16261e07e89e5b8 server_cookie=f8889db64035cb53 in_seq=0 out_seq=0 2024-04-25T17:37:29.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f0825ffb640 1 -- 172.21.15.98:0/4079688137 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563a6d1995f0 con 0x563a6d339d00 2024-04-25T17:37:29.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f0825ffb640 10 client.25060 handle_command_reply: tid=0 2024-04-25T17:37:29.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 client.25060 shutdown 2024-04-25T17:37:29.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 -- 172.21.15.98:0/4079688137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 msgr2=0x563a6d2cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:29.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2cd790 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0838009f80 tx=0x7f0838009300 comp rx=0 tx=0).stop 2024-04-25T17:37:29.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 -- 172.21.15.98:0/4079688137 shutdown_connections 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563a6d339d00 0x563a6d33c120 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563a6d2bfc10 0x563a6d338220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f0825ffb640 0 client.25060 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563a6d2b0e40 0x563a6d2cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 --2- 172.21.15.98:0/4079688137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563a6d2b0520 0x563a6d2cd790 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 -- 172.21.15.98:0/4079688137 >> 172.21.15.98:0/4079688137 conn(0x563a6d2acb30 msgr2=0x563a6d2b8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 -- 172.21.15.98:0/4079688137 shutdown_connections 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:28.999+0000 7f08471bc740 1 -- 172.21.15.98:0/4079688137 wait complete. 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08471bc740 20 client.25060 trim_cache size 0 max 16384 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:37:29.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:37:29.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:37:29.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:37:29.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 -- 172.21.15.98:0/3509169871 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f08280a3e40 msgr2=0x7f08280a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f08280a3e40 0x7f08280a6300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0838011500 tx=0x7f0838005f90 comp rx=0 tx=0).stop 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 -- 172.21.15.98:0/3509169871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f083c19d000 msgr2=0x7f083c19d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f083c19d000 0x7f083c19d4b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0834009fd0 tx=0x7f083400ef10 comp rx=0 tx=0).stop 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 -- 172.21.15.98:0/3509169871 shutdown_connections 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f08280a3e40 0x7f08280a6300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f083c19d000 0x7f083c19d4b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f083c106170 0x7f083c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 --2- 172.21.15.98:0/3509169871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f083c1010f0 0x7f083c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 -- 172.21.15.98:0/3509169871 >> 172.21.15.98:0/3509169871 conn(0x7f083c077fd0 msgr2=0x7f083c0fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 -- 172.21.15.98:0/3509169871 shutdown_connections 2024-04-25T17:37:29.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:29.000+0000 7f08444a0640 1 -- 172.21.15.98:0/3509169871 wait complete. 2024-04-25T17:37:30.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:30 smithi098 ceph-mon[29192]: pgmap v50: 97 pgs: 97 active+clean; 960 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 15 MiB/s wr, 257 op/s 2024-04-25T17:37:30.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:30 smithi203 ceph-mon[32834]: pgmap v50: 97 pgs: 97 active+clean; 960 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 15 MiB/s wr, 257 op/s 2024-04-25T17:37:30.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:30 smithi173 ceph-mon[32856]: pgmap v50: 97 pgs: 97 active+clean; 960 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 15 MiB/s wr, 257 op/s 2024-04-25T17:37:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.749+0000 7f61f77a8700 1 -- 172.21.15.98:0/3180184927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f0102330 msgr2=0x7f61f010e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.749+0000 7f61f77a8700 1 --2- 172.21.15.98:0/3180184927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f0102330 0x7f61f010e7f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f61e800b0a0 tx=0x7f61e800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:37:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.749+0000 7f61f77a8700 1 -- 172.21.15.98:0/3180184927 shutdown_connections 2024-04-25T17:37:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.749+0000 7f61f77a8700 1 --2- 172.21.15.98:0/3180184927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f0102330 0x7f61f010e7f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.749+0000 7f61f77a8700 1 --2- 172.21.15.98:0/3180184927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f0107920 0x7f61f0107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.749+0000 7f61f77a8700 1 --2- 172.21.15.98:0/3180184927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61f0101970 0x7f61f0101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.749+0000 7f61f77a8700 1 -- 172.21.15.98:0/3180184927 >> 172.21.15.98:0/3180184927 conn(0x7f61f0078910 msgr2=0x7f61f0078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.749+0000 7f61f77a8700 1 -- 172.21.15.98:0/3180184927 shutdown_connections 2024-04-25T17:37:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.749+0000 7f61f77a8700 1 -- 172.21.15.98:0/3180184927 wait complete. 2024-04-25T17:37:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f77a8700 1 Processor -- start 2024-04-25T17:37:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f77a8700 1 -- start start 2024-04-25T17:37:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f77a8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f0101970 0x7f61f0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f77a8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61f0102330 0x7f61f0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f77a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f0107920 0x7f61f019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f77a8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0111bb0 con 0x7f61f0107920 2024-04-25T17:37:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f77a8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0111a30 con 0x7f61f0102330 2024-04-25T17:37:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f77a8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0111d30 con 0x7f61f0101970 2024-04-25T17:37:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f5544700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f0101970 0x7f61f0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f5d45700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f0107920 0x7f61f019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f5544700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f0101970 0x7f61f0197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:50980/0 (socket says 172.21.15.98:50980) 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f5544700 1 -- 172.21.15.98:0/2874104847 learned_addr learned my addr 172.21.15.98:0/2874104847 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.750+0000 7f61f4d43700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61f0102330 0x7f61f0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.751+0000 7f61f5d45700 1 -- 172.21.15.98:0/2874104847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f0101970 msgr2=0x7f61f0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.751+0000 7f61f5d45700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f0101970 0x7f61f0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.751+0000 7f61f5d45700 1 -- 172.21.15.98:0/2874104847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61f0102330 msgr2=0x7f61f0198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.751+0000 7f61f5d45700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61f0102330 0x7f61f0198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.751+0000 7f61f5d45700 1 -- 172.21.15.98:0/2874104847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f019cff0 con 0x7f61f0107920 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.751+0000 7f61f5544700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f0101970 0x7f61f0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.751+0000 7f61f5d45700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f0107920 0x7f61f019c8b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f61e80062a0 tx=0x7f61e8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.751+0000 7f61e67fc700 1 -- 172.21.15.98:0/2874104847 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8022070 con 0x7f61f0107920 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.751+0000 7f61f77a8700 1 -- 172.21.15.98:0/2874104847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f019d280 con 0x7f61f0107920 2024-04-25T17:37:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.752+0000 7f61f77a8700 1 -- 172.21.15.98:0/2874104847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f006b190 con 0x7f61f0107920 2024-04-25T17:37:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.753+0000 7f61e67fc700 1 -- 172.21.15.98:0/2874104847 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61e8007e30 con 0x7f61f0107920 2024-04-25T17:37:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.753+0000 7f61e67fc700 1 -- 172.21.15.98:0/2874104847 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e801cbb0 con 0x7f61f0107920 2024-04-25T17:37:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.754+0000 7f61f77a8700 1 -- 172.21.15.98:0/2874104847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61d4005360 con 0x7f61f0107920 2024-04-25T17:37:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.755+0000 7f61e67fc700 1 -- 172.21.15.98:0/2874104847 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f61e8007980 con 0x7f61f0107920 2024-04-25T17:37:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.756+0000 7f61e67fc700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f61dc0a3e30 0x7f61dc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.756+0000 7f61e67fc700 1 -- 172.21.15.98:0/2874104847 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f61e80d50e0 con 0x7f61f0107920 2024-04-25T17:37:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.756+0000 7f61f5544700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f61dc0a3e30 0x7f61dc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.757+0000 7f61f5544700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f61dc0a3e30 0x7f61dc0a62f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00e460 tx=0x7f61ec010040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:30.762+0000 7f61e67fc700 1 -- 172.21.15.98:0/2874104847 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61e80a1950 con 0x7f61f0107920 2024-04-25T17:37:31.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.159+0000 7f61f77a8700 1 -- 172.21.15.98:0/2874104847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61d4005920 con 0x7f61f0107920 2024-04-25T17:37:31.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.161+0000 7f61e67fc700 1 -- 172.21.15.98:0/2874104847 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11881 (secure 0 0 0) 0x7f61e80a1250 con 0x7f61f0107920 2024-04-25T17:37:31.162 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:37:31.162 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:36:39.350510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:37:31.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 -- 172.21.15.98:0/2874104847 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f61dc0a3e30 msgr2=0x7f61dc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:31.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f61dc0a3e30 0x7f61dc0a62f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00e460 tx=0x7f61ec010040 comp rx=0 tx=0).stop 2024-04-25T17:37:31.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 -- 172.21.15.98:0/2874104847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f0107920 msgr2=0x7f61f019c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f0107920 0x7f61f019c8b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f61e80062a0 tx=0x7f61e8004c80 comp rx=0 tx=0).stop 2024-04-25T17:37:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 -- 172.21.15.98:0/2874104847 shutdown_connections 2024-04-25T17:37:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f61dc0a3e30 0x7f61dc0a62f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f0101970 0x7f61f0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61f0107920 0x7f61f019c8b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 --2- 172.21.15.98:0/2874104847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61f0102330 0x7f61f0198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 -- 172.21.15.98:0/2874104847 >> 172.21.15.98:0/2874104847 conn(0x7f61f0078910 msgr2=0x7f61f00ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 -- 172.21.15.98:0/2874104847 shutdown_connections 2024-04-25T17:37:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:31.166+0000 7f61f77a8700 1 -- 172.21.15.98:0/2874104847 wait complete. 2024-04-25T17:37:31.167 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-04-25T17:37:31.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 60.0 s, 1661.6 tps, lat 463.495 ms stddev 3775.731 2024-04-25T17:37:32.114 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 87.55 --await-for 42.0' 2024-04-25T17:37:32.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 87.55 --await-for 42.0 2024-04-25T17:37:32.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:32 smithi098 ceph-mon[29192]: pgmap v51: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 36 MiB/s wr, 843 op/s 2024-04-25T17:37:32.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:32 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2874104847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:37:32.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:32 smithi203 ceph-mon[32834]: pgmap v51: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 36 MiB/s wr, 843 op/s 2024-04-25T17:37:32.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:32 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2874104847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:37:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:37:32.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:32 smithi173 ceph-mon[32856]: pgmap v51: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 36 MiB/s wr, 843 op/s 2024-04-25T17:37:32.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:32 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2874104847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:37:34.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:34 smithi098 ceph-mon[29192]: pgmap v52: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 36 MiB/s wr, 837 op/s 2024-04-25T17:37:34.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:34 smithi203 ceph-mon[32834]: pgmap v52: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 36 MiB/s wr, 837 op/s 2024-04-25T17:37:34.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:34 smithi173 ceph-mon[32856]: pgmap v52: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 36 MiB/s wr, 837 op/s 2024-04-25T17:37:34.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.579+0000 7fcd75e22700 1 -- 172.21.15.98:0/684458670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd7006bad0 msgr2=0x7fcd7010f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:34.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.579+0000 7fcd75e22700 1 --2- 172.21.15.98:0/684458670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd7006bad0 0x7fcd7010f910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcd6800b0a0 tx=0x7fcd6800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.579+0000 7fcd75e22700 1 -- 172.21.15.98:0/684458670 shutdown_connections 2024-04-25T17:37:34.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.579+0000 7fcd75e22700 1 --2- 172.21.15.98:0/684458670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd70108a90 0x7fcd70108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.579+0000 7fcd75e22700 1 --2- 172.21.15.98:0/684458670 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd7006bad0 0x7fcd7010f910 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.579+0000 7fcd75e22700 1 --2- 172.21.15.98:0/684458670 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7006b110 0x7fcd7006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.579+0000 7fcd75e22700 1 -- 172.21.15.98:0/684458670 >> 172.21.15.98:0/684458670 conn(0x7fcd700786a0 msgr2=0x7fcd70078ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:34.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.579+0000 7fcd75e22700 1 -- 172.21.15.98:0/684458670 shutdown_connections 2024-04-25T17:37:34.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.580+0000 7fcd75e22700 1 -- 172.21.15.98:0/684458670 wait complete. 2024-04-25T17:37:34.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd75e22700 1 Processor -- start 2024-04-25T17:37:34.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd75e22700 1 -- start start 2024-04-25T17:37:34.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd75e22700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd7006b110 0x7fcd701a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd75e22700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7006bad0 0x7fcd701a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd75e22700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd70108a90 0x7fcd7019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd75e22700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd70119380 con 0x7fcd7006bad0 2024-04-25T17:37:34.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd75e22700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcd70119200 con 0x7fcd70108a90 2024-04-25T17:37:34.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd75e22700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcd70119500 con 0x7fcd7006b110 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd6f7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd7006b110 0x7fcd701a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd6ffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd70108a90 0x7fcd7019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd6ffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd70108a90 0x7fcd7019f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43202/0 (socket says 172.21.15.98:43202) 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd6ffff700 1 -- 172.21.15.98:0/2783266400 learned_addr learned my addr 172.21.15.98:0/2783266400 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.581+0000 7fcd6effd700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7006bad0 0x7fcd701a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.582+0000 7fcd6ffff700 1 -- 172.21.15.98:0/2783266400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd7006b110 msgr2=0x7fcd701a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.582+0000 7fcd6ffff700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd7006b110 0x7fcd701a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.582+0000 7fcd6ffff700 1 -- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7006bad0 msgr2=0x7fcd701a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.582+0000 7fcd6ffff700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7006bad0 0x7fcd701a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.582+0000 7fcd6ffff700 1 -- 172.21.15.98:0/2783266400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7019fd60 con 0x7fcd70108a90 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.582+0000 7fcd6f7fe700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd7006b110 0x7fcd701a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.582+0000 7fcd6effd700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7006bad0 0x7fcd701a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.582+0000 7fcd6ffff700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd70108a90 0x7fcd7019f600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd680062a0 tx=0x7fcd6800cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:34.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.582+0000 7fcd6cff9700 1 -- 172.21.15.98:0/2783266400 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd68021020 con 0x7fcd70108a90 2024-04-25T17:37:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.582+0000 7fcd6cff9700 1 -- 172.21.15.98:0/2783266400 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd68007d40 con 0x7fcd70108a90 2024-04-25T17:37:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.583+0000 7fcd6cff9700 1 -- 172.21.15.98:0/2783266400 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6801d7f0 con 0x7fcd70108a90 2024-04-25T17:37:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.583+0000 7fcd75e22700 1 -- 172.21.15.98:0/2783266400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7019fff0 con 0x7fcd70108a90 2024-04-25T17:37:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.583+0000 7fcd75e22700 1 -- 172.21.15.98:0/2783266400 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd701a0420 con 0x7fcd70108a90 2024-04-25T17:37:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.586+0000 7fcd6cff9700 1 -- 172.21.15.98:0/2783266400 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcd68024020 con 0x7fcd70108a90 2024-04-25T17:37:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.587+0000 7fcd6cff9700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd580a3f50 0x7fcd580a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.587+0000 7fcd6f7fe700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd580a3f50 0x7fcd580a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.588+0000 7fcd6cff9700 1 -- 172.21.15.98:0/2783266400 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fcd6800f020 con 0x7fcd70108a90 2024-04-25T17:37:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.588+0000 7fcd806abb80 1 Processor -- start 2024-04-25T17:37:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.588+0000 7fcd6f7fe700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd580a3f50 0x7fcd580a6410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcd60006fd0 tx=0x7fcd60008040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.588+0000 7fcd806abb80 1 -- start start 2024-04-25T17:37:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2279e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf2284d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf2285270 0x55ebf2287660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd806abb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ebf21ccec0 con 0x55ebf227a3a0 2024-04-25T17:37:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd806abb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ebf2173af0 con 0x55ebf2285270 2024-04-25T17:37:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd806abb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ebf21c8e80 con 0x55ebf2279a80 2024-04-25T17:37:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf2284d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf2284d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38636/0 (socket says 172.21.15.98:38636) 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6effd700 1 -- 172.21.15.98:0/3679312070 learned_addr learned my addr 172.21.15.98:0/3679312070 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6ffff700 1 --2- 172.21.15.98:0/3679312070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf2285270 0x55ebf2287660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6f7fe700 1 --2- 172.21.15.98:0/3679312070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2279e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6effd700 1 -- 172.21.15.98:0/3679312070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 msgr2=0x55ebf2279e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6effd700 1 --2- 172.21.15.98:0/3679312070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2279e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6effd700 1 -- 172.21.15.98:0/3679312070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf2285270 msgr2=0x55ebf2287660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6effd700 1 --2- 172.21.15.98:0/3679312070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf2285270 0x55ebf2287660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6effd700 1 -- 172.21.15.98:0/3679312070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebf211aa60 con 0x55ebf227a3a0 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd6effd700 1 --2- 172.21.15.98:0/3679312070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf2284d30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcd64007870 tx=0x7fcd64007b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c22b3f4e41fc8ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.589+0000 7fcd55ffb700 1 -- 172.21.15.98:0/3679312070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd640125f0 con 0x55ebf227a3a0 2024-04-25T17:37:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.590+0000 7fcd55ffb700 1 -- 172.21.15.98:0/3679312070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd64018070 con 0x55ebf227a3a0 2024-04-25T17:37:34.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.590+0000 7fcd55ffb700 1 -- 172.21.15.98:0/3679312070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd64013ab0 con 0x55ebf227a3a0 2024-04-25T17:37:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.591+0000 7fcd806abb80 1 -- 172.21.15.98:0/3679312070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 msgr2=0x55ebf2284d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.591+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3679312070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf2284d30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcd64007870 tx=0x7fcd64007b80 comp rx=0 tx=0).stop 2024-04-25T17:37:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.591+0000 7fcd806abb80 1 -- 172.21.15.98:0/3679312070 shutdown_connections 2024-04-25T17:37:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.591+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3679312070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2279e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.591+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3679312070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf2285270 0x55ebf2287660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.591+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3679312070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf2284d30 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.591+0000 7fcd806abb80 1 -- 172.21.15.98:0/3679312070 >> 172.21.15.98:0/3679312070 conn(0x55ebf21a7820 msgr2=0x55ebf2275f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.591+0000 7fcd806abb80 1 -- 172.21.15.98:0/3679312070 shutdown_connections 2024-04-25T17:37:34.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.592+0000 7fcd806abb80 1 -- 172.21.15.98:0/3679312070 wait complete. 2024-04-25T17:37:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.592+0000 7fcd806abb80 1 -- start start 2024-04-25T17:37:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd806abb80 1 Processor -- start 2024-04-25T17:37:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2279a80 0x55ebf2284900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 0x55ebf2281c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2285270 0x55ebf22821b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd806abb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ebf2287cc0 con 0x55ebf2279a80 2024-04-25T17:37:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd806abb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ebf216bd60 con 0x55ebf227a3a0 2024-04-25T17:37:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd806abb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ebf2287e40 con 0x55ebf2285270 2024-04-25T17:37:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd6f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2279a80 0x55ebf2284900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd6f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2279a80 0x55ebf2284900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38638/0 (socket says 172.21.15.98:38638) 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd6f7fe700 1 -- 172.21.15.98:0/3333692257 learned_addr learned my addr 172.21.15.98:0/3333692257 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd806abb80 10 client.0 _ll_register_callbacks cb 0x55ebf2165ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd6effd700 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 0x55ebf2281c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd6ffff700 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2285270 0x55ebf22821b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd6f7fe700 1 -- 172.21.15.98:0/3333692257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2285270 msgr2=0x55ebf22821b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd6f7fe700 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2285270 0x55ebf22821b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd6f7fe700 1 -- 172.21.15.98:0/3333692257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 msgr2=0x55ebf2281c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd6f7fe700 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 0x55ebf2281c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.593+0000 7fcd6f7fe700 1 -- 172.21.15.98:0/3333692257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebf2184400 con 0x55ebf2279a80 2024-04-25T17:37:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.594+0000 7fcd6f7fe700 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2279a80 0x55ebf2284900 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000cc30 tx=0x7fcd600047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d2ed32865a164c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.594+0000 7fcd4effd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.594+0000 7fcd4effd700 1 -- 172.21.15.98:0/3333692257 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60023070 con 0x55ebf2279a80 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.594+0000 7fcd806abb80 1 -- 172.21.15.98:0/3333692257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd64018070 con 0x55ebf2279a80 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.594+0000 7fcd4effd700 1 -- 172.21.15.98:0/3333692257 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fcd60013ca0 con 0x55ebf2279a80 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.594+0000 7fcd4effd700 1 -- 172.21.15.98:0/3333692257 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcd6002f050 con 0x55ebf2279a80 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.594+0000 7fcd806abb80 10 client.15507 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.594+0000 7fcd806abb80 1 -- 172.21.15.98:0/3333692257 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ebf2292330 con 0x55ebf2279a80 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.594+0000 7fcd4effd700 1 -- 172.21.15.98:0/3333692257 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd64018070 con 0x55ebf2279a80 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.595+0000 7fcd4effd700 1 -- 172.21.15.98:0/3333692257 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fcd6001e470 con 0x55ebf2279a80 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.595+0000 7fcd806abb80 10 client.15507 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.595+0000 7fcd806abb80 10 client.15507 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:37:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.595+0000 7fcd806abb80 20 client.15507 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.595+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ebf2294910 0x55ebf2296d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.595+0000 7fcd806abb80 4 client.15507 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:37:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.595+0000 7fcd806abb80 1 -- 172.21.15.98:0/3333692257 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebf21da6b0 con 0x55ebf2294910 2024-04-25T17:37:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.595+0000 7fcd6effd700 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ebf2294910 0x55ebf2296d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.596+0000 7fcd6effd700 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ebf2294910 0x55ebf2296d80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5853756ebd392049 server_cookie=f2f71d91af93cbea in_seq=0 out_seq=0 2024-04-25T17:37:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.596+0000 7fcd4effd700 10 client.15507 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.598+0000 7fcd4effd700 1 -- 172.21.15.98:0/3333692257 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebf21da6b0 con 0x55ebf2294910 2024-04-25T17:37:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.598+0000 7fcd4effd700 10 client.15507 handle_command_reply: tid=0 2024-04-25T17:37:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.598+0000 7fcd806abb80 1 client.15507 shutdown 2024-04-25T17:37:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.598+0000 7fcd806abb80 1 -- 172.21.15.98:0/3333692257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2279a80 msgr2=0x55ebf2284900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.598+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2279a80 0x55ebf2284900 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000cc30 tx=0x7fcd600047e0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.598+0000 7fcd806abb80 1 -- 172.21.15.98:0/3333692257 shutdown_connections 2024-04-25T17:37:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.598+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ebf2294910 0x55ebf2296d80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.598+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2285270 0x55ebf22821b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.598+0000 7fcd4effd700 0 client.15507 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.599+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2279a80 0x55ebf2284900 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.599+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3333692257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 0x55ebf2281c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.599+0000 7fcd806abb80 1 -- 172.21.15.98:0/3333692257 >> 172.21.15.98:0/3333692257 conn(0x55ebf21a7820 msgr2=0x55ebf2275b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.599+0000 7fcd806abb80 1 -- 172.21.15.98:0/3333692257 shutdown_connections 2024-04-25T17:37:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.599+0000 7fcd806abb80 1 -- 172.21.15.98:0/3333692257 wait complete. 2024-04-25T17:37:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.599+0000 7fcd806abb80 20 client.15507 trim_cache size 0 max 16384 2024-04-25T17:37:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.628+0000 7fcd806abb80 1 Processor -- start 2024-04-25T17:37:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.628+0000 7fcd806abb80 1 -- start start 2024-04-25T17:37:34.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2275e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 0x55ebf229c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2285270 0x55ebf229cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6f7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2275e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 0x55ebf229c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6f7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2275e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:34454/0 (socket says 172.21.15.98:34454) 2024-04-25T17:37:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 0x55ebf229c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43232/0 (socket says 172.21.15.98:43232) 2024-04-25T17:37:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd806abb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd701b6090 con 0x55ebf2285270 2024-04-25T17:37:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd806abb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcd701b38f0 con 0x55ebf227a3a0 2024-04-25T17:37:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd806abb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcd701b1760 con 0x55ebf2279a80 2024-04-25T17:37:34.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2285270 0x55ebf229cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2285270 0x55ebf229cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38642/0 (socket says 172.21.15.98:38642) 2024-04-25T17:37:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6ffff700 1 -- 172.21.15.98:0/1942892745 learned_addr learned my addr 172.21.15.98:0/1942892745 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:34.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6effd700 1 -- 172.21.15.98:0/1942892745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 msgr2=0x55ebf2275e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6effd700 1 --2- 172.21.15.98:0/1942892745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2275e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6effd700 1 -- 172.21.15.98:0/1942892745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2285270 msgr2=0x55ebf229cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6effd700 1 --2- 172.21.15.98:0/1942892745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2285270 0x55ebf229cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6effd700 1 -- 172.21.15.98:0/1942892745 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebf227b410 con 0x55ebf227a3a0 2024-04-25T17:37:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.629+0000 7fcd6ffff700 1 --2- 172.21.15.98:0/1942892745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2285270 0x55ebf229cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:37:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.630+0000 7fcd6effd700 1 --2- 172.21.15.98:0/1942892745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 0x55ebf229c690 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcd64004790 tx=0x7fcd64013690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c332f6d4010d2dff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.630+0000 7fcd557fa700 1 -- 172.21.15.98:0/1942892745 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd640138d0 con 0x55ebf227a3a0 2024-04-25T17:37:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.630+0000 7fcd557fa700 1 -- 172.21.15.98:0/1942892745 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd64018030 con 0x55ebf227a3a0 2024-04-25T17:37:34.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.630+0000 7fcd557fa700 1 -- 172.21.15.98:0/1942892745 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd640125f0 con 0x55ebf227a3a0 2024-04-25T17:37:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.632+0000 7fcd806abb80 1 -- 172.21.15.98:0/1942892745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 msgr2=0x55ebf229c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.632+0000 7fcd806abb80 1 --2- 172.21.15.98:0/1942892745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 0x55ebf229c690 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcd64004790 tx=0x7fcd64013690 comp rx=0 tx=0).stop 2024-04-25T17:37:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.632+0000 7fcd806abb80 1 -- 172.21.15.98:0/1942892745 shutdown_connections 2024-04-25T17:37:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.632+0000 7fcd806abb80 1 --2- 172.21.15.98:0/1942892745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2275e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.632+0000 7fcd806abb80 1 --2- 172.21.15.98:0/1942892745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf2285270 0x55ebf229cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.632+0000 7fcd806abb80 1 --2- 172.21.15.98:0/1942892745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf227a3a0 0x55ebf229c690 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.632+0000 7fcd806abb80 1 -- 172.21.15.98:0/1942892745 >> 172.21.15.98:0/1942892745 conn(0x55ebf21a7820 msgr2=0x55ebf2283d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.632+0000 7fcd806abb80 1 -- 172.21.15.98:0/1942892745 shutdown_connections 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.632+0000 7fcd806abb80 1 -- 172.21.15.98:0/1942892745 wait complete. 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.633+0000 7fcd806abb80 1 -- start start 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.633+0000 7fcd806abb80 1 Processor -- start 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.633+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2289ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.633+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf228a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.633+0000 7fcd806abb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf2285270 0x55ebf228a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.633+0000 7fcd806abb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd701b1db0 con 0x55ebf227a3a0 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.633+0000 7fcd806abb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcd701b3220 con 0x55ebf2285270 2024-04-25T17:37:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.633+0000 7fcd806abb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcd701b2480 con 0x55ebf2279a80 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.633+0000 7fcd806abb80 10 client.0 _ll_register_callbacks cb 0x55ebf2165ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.633+0000 7fcd6f7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2289ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6f7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2289ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:34464/0 (socket says 172.21.15.98:34464) 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf228a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6f7fe700 1 -- 172.21.15.98:0/3744125812 learned_addr learned my addr 172.21.15.98:0/3744125812 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6ffff700 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf2285270 0x55ebf228a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6f7fe700 1 -- 172.21.15.98:0/3744125812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf2285270 msgr2=0x55ebf228a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6f7fe700 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf2285270 0x55ebf228a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6f7fe700 1 -- 172.21.15.98:0/3744125812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 msgr2=0x55ebf228a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6f7fe700 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf228a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6f7fe700 1 -- 172.21.15.98:0/3744125812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebf22faf40 con 0x55ebf2279a80 2024-04-25T17:37:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6effd700 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf228a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:37:34.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd6f7fe700 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2289ac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000cc30 tx=0x7fcd6000a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5248c78b0759f0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:34.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd4effd700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:37:34.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd4effd700 1 -- 172.21.15.98:0/3744125812 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60038060 con 0x55ebf2279a80 2024-04-25T17:37:34.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd4effd700 1 -- 172.21.15.98:0/3744125812 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7fcd60013aa0 con 0x55ebf2279a80 2024-04-25T17:37:34.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.634+0000 7fcd806abb80 1 -- 172.21.15.98:0/3744125812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd64018030 con 0x55ebf2279a80 2024-04-25T17:37:34.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.635+0000 7fcd4effd700 1 -- 172.21.15.98:0/3744125812 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcd60023040 con 0x55ebf2279a80 2024-04-25T17:37:34.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.635+0000 7fcd806abb80 10 client.25078 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.635+0000 7fcd806abb80 1 -- 172.21.15.98:0/3744125812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ebf22fc260 con 0x55ebf2279a80 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.636+0000 7fcd4effd700 1 -- 172.21.15.98:0/3744125812 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd6001ec90 con 0x55ebf2279a80 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.636+0000 7fcd4effd700 1 -- 172.21.15.98:0/3744125812 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7fcd60004360 con 0x55ebf2279a80 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.636+0000 7fcd806abb80 10 client.25078 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.636+0000 7fcd806abb80 10 client.25078 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.636+0000 7fcd806abb80 20 client.25078 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.636+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ebf2294910 0x55ebf22feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.636+0000 7fcd806abb80 4 client.25078 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.55, "await_for": 42.0}] 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.636+0000 7fcd806abb80 1 -- 172.21.15.98:0/3744125812 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 87.55, "await_for": 42.0}) -- 0x7fcd64018030 con 0x55ebf2294910 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.638+0000 7fcd6effd700 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ebf2294910 0x55ebf22feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.638+0000 7fcd4effd700 10 client.25078 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:34.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.638+0000 7fcd6effd700 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ebf2294910 0x55ebf22feab0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95d93d8a7b4c0ec9 server_cookie=39e084941ebf0f18 in_seq=0 out_seq=0 2024-04-25T17:37:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.841+0000 7fcd4effd700 1 -- 172.21.15.98:0/3744125812 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x7fcd64018030 con 0x55ebf2294910 2024-04-25T17:37:34.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.841+0000 7fcd4effd700 10 client.25078 handle_command_reply: tid=0 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.841+0000 7fcd806abb80 1 client.25078 shutdown 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 1 -- 172.21.15.98:0/3744125812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 msgr2=0x55ebf2289ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2289ac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000cc30 tx=0x7fcd6000a0d0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 1 -- 172.21.15.98:0/3744125812 shutdown_connections 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ebf2294910 0x55ebf22feab0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ebf2279a80 0x55ebf2289ac0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd4effd700 0 client.25078 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ebf2285270 0x55ebf228a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 1 --2- 172.21.15.98:0/3744125812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ebf227a3a0 0x55ebf228a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 1 -- 172.21.15.98:0/3744125812 >> 172.21.15.98:0/3744125812 conn(0x55ebf21a7820 msgr2=0x55ebf22811d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 1 -- 172.21.15.98:0/3744125812 shutdown_connections 2024-04-25T17:37:34.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 1 -- 172.21.15.98:0/3744125812 wait complete. 2024-04-25T17:37:34.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd806abb80 20 client.25078 trim_cache size 0 max 16384 2024-04-25T17:37:34.844 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:37:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 29, 2024-04-25T17:37:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:37:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 6, 2024-04-25T17:37:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:37:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "12afe7db": { 2024-04-25T17:37:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 6, 2024-04-25T17:37:34.844 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 87.5, 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:37:34.845 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:37:34.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:37:34.846 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:37:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd75e22700 1 -- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd580a3f50 msgr2=0x7fcd580a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd75e22700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd580a3f50 0x7fcd580a6410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcd60006fd0 tx=0x7fcd60008040 comp rx=0 tx=0).stop 2024-04-25T17:37:34.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd75e22700 1 -- 172.21.15.98:0/2783266400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd70108a90 msgr2=0x7fcd7019f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.842+0000 7fcd75e22700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd70108a90 0x7fcd7019f600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcd680062a0 tx=0x7fcd6800cf20 comp rx=0 tx=0).stop 2024-04-25T17:37:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.843+0000 7fcd75e22700 1 -- 172.21.15.98:0/2783266400 shutdown_connections 2024-04-25T17:37:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.843+0000 7fcd75e22700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd580a3f50 0x7fcd580a6410 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.843+0000 7fcd75e22700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd7006b110 0x7fcd701a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.843+0000 7fcd75e22700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd70108a90 0x7fcd7019f600 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.843+0000 7fcd75e22700 1 --2- 172.21.15.98:0/2783266400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd7006bad0 0x7fcd701a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.843+0000 7fcd75e22700 1 -- 172.21.15.98:0/2783266400 >> 172.21.15.98:0/2783266400 conn(0x7fcd700786a0 msgr2=0x7fcd701007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.843+0000 7fcd75e22700 1 -- 172.21.15.98:0/2783266400 shutdown_connections 2024-04-25T17:37:34.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:37:34.843+0000 7fcd75e22700 1 -- 172.21.15.98:0/2783266400 wait complete. 2024-04-25T17:37:35.658 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '12afe7db', quiesce took 7.1 seconds. Will release after: 27.55 2024-04-25T17:37:36.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:36 smithi098 ceph-mon[29192]: pgmap v53: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 36 MiB/s wr, 837 op/s 2024-04-25T17:37:36.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:36 smithi203 ceph-mon[32834]: pgmap v53: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 36 MiB/s wr, 837 op/s 2024-04-25T17:37:36.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:36 smithi173 ceph-mon[32856]: pgmap v53: 97 pgs: 97 active+clean; 984 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 36 MiB/s wr, 837 op/s 2024-04-25T17:37:36.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 65.0 s, 2238.3 tps, lat 27.609 ms stddev 28.690 2024-04-25T17:37:38.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:38 smithi098 ceph-mon[29192]: pgmap v54: 97 pgs: 97 active+clean; 1023 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 40 MiB/s wr, 1.12k op/s 2024-04-25T17:37:38.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:38 smithi203 ceph-mon[32834]: pgmap v54: 97 pgs: 97 active+clean; 1023 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 40 MiB/s wr, 1.12k op/s 2024-04-25T17:37:38.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:38 smithi173 ceph-mon[32856]: pgmap v54: 97 pgs: 97 active+clean; 1023 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 40 MiB/s wr, 1.12k op/s 2024-04-25T17:37:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:40 smithi098 ceph-mon[29192]: pgmap v55: 97 pgs: 97 active+clean; 1023 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 25 MiB/s wr, 872 op/s 2024-04-25T17:37:40.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:40 smithi203 ceph-mon[32834]: pgmap v55: 97 pgs: 97 active+clean; 1023 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 25 MiB/s wr, 872 op/s 2024-04-25T17:37:40.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:40 smithi173 ceph-mon[32856]: pgmap v55: 97 pgs: 97 active+clean; 1023 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 25 MiB/s wr, 872 op/s 2024-04-25T17:37:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 70.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:37:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.096+0000 7feb3b22a640 1 -- 172.21.15.98:0/2894932974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb341005c0 msgr2=0x7feb34100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.096+0000 7feb3b22a640 1 --2- 172.21.15.98:0/2894932974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb341005c0 0x7feb34100a40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7feb2000aa40 tx=0x7feb2002b750 comp rx=0 tx=0).stop 2024-04-25T17:37:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.096+0000 7feb3b22a640 1 -- 172.21.15.98:0/2894932974 shutdown_connections 2024-04-25T17:37:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.096+0000 7feb3b22a640 1 --2- 172.21.15.98:0/2894932974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb34100f80 0x7feb3410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.096+0000 7feb3b22a640 1 --2- 172.21.15.98:0/2894932974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb341005c0 0x7feb34100a40 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.096+0000 7feb3b22a640 1 --2- 172.21.15.98:0/2894932974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb34106660 0x7feb34106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.096+0000 7feb3b22a640 1 -- 172.21.15.98:0/2894932974 >> 172.21.15.98:0/2894932974 conn(0x7feb34077370 msgr2=0x7feb340fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.096+0000 7feb3b22a640 1 -- 172.21.15.98:0/2894932974 shutdown_connections 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb3b22a640 1 -- 172.21.15.98:0/2894932974 wait complete. 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb3b22a640 1 Processor -- start 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb3b22a640 1 -- start start 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb3b22a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb341005c0 0x7feb341985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb3b22a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb34100f80 0x7feb34198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb3b22a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb34106660 0x7feb3419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb3b22a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb3410ffd0 con 0x7feb34106660 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb3b22a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb3410fe50 con 0x7feb341005c0 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb3b22a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feb34110150 con 0x7feb34100f80 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb397c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb34106660 0x7feb3419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb397c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb34106660 0x7feb3419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38672/0 (socket says 172.21.15.98:38672) 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.097+0000 7feb397c7640 1 -- 172.21.15.98:0/88229598 learned_addr learned my addr 172.21.15.98:0/88229598 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb38fc6640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb341005c0 0x7feb341985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb33fff640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb34100f80 0x7feb34198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb397c7640 1 -- 172.21.15.98:0/88229598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb34100f80 msgr2=0x7feb34198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb397c7640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb34100f80 0x7feb34198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb397c7640 1 -- 172.21.15.98:0/88229598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb341005c0 msgr2=0x7feb341985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb397c7640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb341005c0 0x7feb341985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb397c7640 1 -- 172.21.15.98:0/88229598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3419d5d0 con 0x7feb34106660 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb38fc6640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb341005c0 0x7feb341985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb397c7640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb34106660 0x7feb3419cea0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00eb10 tx=0x7feb2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb31ffb640 1 -- 172.21.15.98:0/88229598 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c004b20 con 0x7feb34106660 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb31ffb640 1 -- 172.21.15.98:0/88229598 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb2c004cc0 con 0x7feb34106660 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb3b22a640 1 -- 172.21.15.98:0/88229598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb3419d8c0 con 0x7feb34106660 2024-04-25T17:37:42.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb31ffb640 1 -- 172.21.15.98:0/88229598 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c010670 con 0x7feb34106660 2024-04-25T17:37:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.098+0000 7feb3b22a640 1 -- 172.21.15.98:0/88229598 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb341a5150 con 0x7feb34106660 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.101+0000 7feb31ffb640 1 -- 172.21.15.98:0/88229598 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feb2c005020 con 0x7feb34106660 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.101+0000 7feb31ffb640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7feb240a3dc0 0x7feb240a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.101+0000 7feb3e019740 1 Processor -- start 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.101+0000 7feb3e019740 1 -- start start 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.101+0000 7feb38fc6640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7feb240a3dc0 0x7feb240a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.101+0000 7feb31ffb640 1 -- 172.21.15.98:0/88229598 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7feb2c0d0eb0 con 0x7feb34106660 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.101+0000 7feb3e019740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 0x55be6f2a7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.101+0000 7feb3e019740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7f40 0x55be6f2b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.101+0000 7feb3e019740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb3e019740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55be6f20c970 con 0x55be6f2b2800 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb3e019740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55be6f1a94a0 con 0x55be6f2a7620 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb3e019740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55be6f208930 con 0x55be6f2a7f40 2024-04-25T17:37:42.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb397c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb397c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2b4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38680/0 (socket says 172.21.15.98:38680) 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb397c7640 1 -- 172.21.15.98:0/3979903659 learned_addr learned my addr 172.21.15.98:0/3979903659 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb33fff640 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7f40 0x55be6f2b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb38fc6640 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 0x55be6f2a7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb397c7640 1 -- 172.21.15.98:0/3979903659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7f40 msgr2=0x55be6f2b22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb397c7640 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7f40 0x55be6f2b22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb397c7640 1 -- 172.21.15.98:0/3979903659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 msgr2=0x55be6f2a7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb397c7640 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 0x55be6f2a7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb397c7640 1 -- 172.21.15.98:0/3979903659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be6ef1f210 con 0x55be6f2b2800 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb33fff640 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7f40 0x55be6f2b22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb38fc6640 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 0x55be6f2a7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb38fc6640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7feb240a3dc0 0x7feb240a6280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7feb34101700 tx=0x7feb2800e350 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb397c7640 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2b4bf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0992e0 tx=0x7feb2c09ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0407e78b35f9e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb1effd640 1 -- 172.21.15.98:0/3979903659 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c09ce80 con 0x55be6f2b2800 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb1effd640 1 -- 172.21.15.98:0/3979903659 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb2c0a1230 con 0x55be6f2b2800 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.102+0000 7feb1effd640 1 -- 172.21.15.98:0/3979903659 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0a5660 con 0x55be6f2b2800 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 -- 172.21.15.98:0/3979903659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 msgr2=0x55be6f2b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2b4bf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0992e0 tx=0x7feb2c09ca80 comp rx=0 tx=0).stop 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 -- 172.21.15.98:0/3979903659 shutdown_connections 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2b4bf0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7f40 0x55be6f2b22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 --2- 172.21.15.98:0/3979903659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 0x55be6f2a7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 -- 172.21.15.98:0/3979903659 >> 172.21.15.98:0/3979903659 conn(0x55be6f2a3b30 msgr2=0x55be6f2a3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 -- 172.21.15.98:0/3979903659 shutdown_connections 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 -- 172.21.15.98:0/3979903659 wait complete. 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 -- start start 2024-04-25T17:37:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.103+0000 7feb3e019740 1 Processor -- start 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb3e019740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 0x55be6f2b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb3e019740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7f40 0x55be6f2af490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb3e019740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2afa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb3e019740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55be6f2b5430 con 0x55be6f2b2800 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb3e019740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55be6f1a70b0 con 0x55be6f2a7f40 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb3e019740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55be6f2b55b0 con 0x55be6f2a7620 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb3e019740 10 client.0 _ll_register_callbacks cb 0x55be6f194d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb397c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb397c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38694/0 (socket says 172.21.15.98:38694) 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb397c7640 1 -- 172.21.15.98:0/1809562743 learned_addr learned my addr 172.21.15.98:0/1809562743 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb33fff640 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7f40 0x55be6f2af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb38fc6640 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 0x55be6f2b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb397c7640 1 -- 172.21.15.98:0/1809562743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 msgr2=0x55be6f2b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb397c7640 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 0x55be6f2b21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb397c7640 1 -- 172.21.15.98:0/1809562743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7f40 msgr2=0x55be6f2af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb397c7640 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7f40 0x55be6f2af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb397c7640 1 -- 172.21.15.98:0/1809562743 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be6efeb620 con 0x55be6f2b2800 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.104+0000 7feb38fc6640 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 0x55be6f2b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb397c7640 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2afa30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0ce6e0 tx=0x7feb2c005540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca3f3ad6d1800bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb13fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb13fff640 1 -- 172.21.15.98:0/1809562743 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0a9020 con 0x55be6f2b2800 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb13fff640 1 -- 172.21.15.98:0/1809562743 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7feb2c0a5660 con 0x55be6f2b2800 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb3e019740 1 -- 172.21.15.98:0/1809562743 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb2c0a1230 con 0x55be6f2b2800 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb13fff640 1 -- 172.21.15.98:0/1809562743 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feb2c09ce80 con 0x55be6f2b2800 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb3e019740 10 client.15525 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb3e019740 1 -- 172.21.15.98:0/1809562743 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55be6f2c13a0 con 0x55be6f2b2800 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb13fff640 1 -- 172.21.15.98:0/1809562743 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb2c0a1230 con 0x55be6f2b2800 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb13fff640 1 -- 172.21.15.98:0/1809562743 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7feb2c09bd40 con 0x55be6f2b2800 2024-04-25T17:37:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb3e019740 10 client.15525 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb3e019740 10 client.15525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb3e019740 20 client.15525 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb3e019740 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55be6f2c26f0 0x55be6f2c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb3e019740 4 client.15525 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.105+0000 7feb3e019740 1 -- 172.21.15.98:0/1809562743 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be6f21a390 con 0x55be6f2c26f0 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.106+0000 7feb38fc6640 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55be6f2c26f0 0x55be6f2c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.106+0000 7feb13fff640 10 client.15525 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.106+0000 7feb38fc6640 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55be6f2c26f0 0x55be6f2c4ae0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd60a668731c4f9 server_cookie=6afec54c4c9ca1f6 in_seq=0 out_seq=0 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb13fff640 1 -- 172.21.15.98:0/1809562743 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55be6f21a390 con 0x55be6f2c26f0 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb13fff640 10 client.15525 handle_command_reply: tid=0 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 client.15525 shutdown 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 -- 172.21.15.98:0/1809562743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 msgr2=0x55be6f2afa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2afa30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0ce6e0 tx=0x7feb2c005540 comp rx=0 tx=0).stop 2024-04-25T17:37:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 -- 172.21.15.98:0/1809562743 shutdown_connections 2024-04-25T17:37:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55be6f2c26f0 0x55be6f2c4ae0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2b2800 0x55be6f2afa30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7f40 0x55be6f2af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 --2- 172.21.15.98:0/1809562743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 0x55be6f2b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb13fff640 0 client.15525 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 -- 172.21.15.98:0/1809562743 >> 172.21.15.98:0/1809562743 conn(0x55be6f2a3b30 msgr2=0x55be6f2a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 -- 172.21.15.98:0/1809562743 shutdown_connections 2024-04-25T17:37:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 1 -- 172.21.15.98:0/1809562743 wait complete. 2024-04-25T17:37:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.107+0000 7feb3e019740 20 client.15525 trim_cache size 0 max 16384 2024-04-25T17:37:42.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.122+0000 7feb3e019740 1 Processor -- start 2024-04-25T17:37:42.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.122+0000 7feb3e019740 1 -- start start 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.122+0000 7feb3e019740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 0x55be6f2be2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.122+0000 7feb3e019740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.122+0000 7feb3e019740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2b2800 0x55be6f2ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.122+0000 7feb3e019740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb341b0280 con 0x55be6f2a7f40 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.122+0000 7feb3e019740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb341adb20 con 0x55be6f2a7620 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.122+0000 7feb3e019740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feb341ab970 con 0x55be6f2b2800 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38700/0 (socket says 172.21.15.98:38700) 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb33fff640 1 -- 172.21.15.98:0/1885852707 learned_addr learned my addr 172.21.15.98:0/1885852707 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb38fc6640 1 --2- 172.21.15.98:0/1885852707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 0x55be6f2be2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb397c7640 1 --2- 172.21.15.98:0/1885852707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2b2800 0x55be6f2ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb33fff640 1 -- 172.21.15.98:0/1885852707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2b2800 msgr2=0x55be6f2ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb33fff640 1 --2- 172.21.15.98:0/1885852707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2b2800 0x55be6f2ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb33fff640 1 -- 172.21.15.98:0/1885852707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 msgr2=0x55be6f2be2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb33fff640 1 --2- 172.21.15.98:0/1885852707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 0x55be6f2be2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb33fff640 1 -- 172.21.15.98:0/1885852707 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be6f2c4e50 con 0x55be6f2a7f40 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb33fff640 1 --2- 172.21.15.98:0/1885852707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2a2b30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7feb20034000 tx=0x7feb20030eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c717d36fbe188970 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb1e7fc640 1 -- 172.21.15.98:0/1885852707 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20036040 con 0x55be6f2a7f40 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb1e7fc640 1 -- 172.21.15.98:0/1885852707 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb20040070 con 0x55be6f2a7f40 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.123+0000 7feb1e7fc640 1 -- 172.21.15.98:0/1885852707 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb200055b0 con 0x55be6f2a7f40 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 -- 172.21.15.98:0/1885852707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 msgr2=0x55be6f2a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 --2- 172.21.15.98:0/1885852707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2a2b30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7feb20034000 tx=0x7feb20030eb0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 -- 172.21.15.98:0/1885852707 shutdown_connections 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 --2- 172.21.15.98:0/1885852707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2b2800 0x55be6f2ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 --2- 172.21.15.98:0/1885852707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2a2b30 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 --2- 172.21.15.98:0/1885852707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2a7620 0x55be6f2be2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 -- 172.21.15.98:0/1885852707 >> 172.21.15.98:0/1885852707 conn(0x55be6f2a3b30 msgr2=0x55be6f2a2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 -- 172.21.15.98:0/1885852707 shutdown_connections 2024-04-25T17:37:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 -- 172.21.15.98:0/1885852707 wait complete. 2024-04-25T17:37:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 -- start start 2024-04-25T17:37:42.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.124+0000 7feb3e019740 1 Processor -- start 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb3e019740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 0x55be6f2b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb3e019740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2b8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb3e019740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2b2800 0x55be6f2b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb3e019740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feb341abfe0 con 0x55be6f2a7f40 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb3e019740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feb341ad450 con 0x55be6f2b2800 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb3e019740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feb341ac6b0 con 0x55be6f2a7620 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb3e019740 10 client.0 _ll_register_callbacks cb 0x7feb2c0994f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2b8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38702/0 (socket says 172.21.15.98:38702) 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb33fff640 1 -- 172.21.15.98:0/3761271793 learned_addr learned my addr 172.21.15.98:0/3761271793 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb38fc6640 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 0x55be6f2b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb397c7640 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2b2800 0x55be6f2b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb33fff640 1 -- 172.21.15.98:0/3761271793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 msgr2=0x55be6f2b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb33fff640 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 0x55be6f2b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb33fff640 1 -- 172.21.15.98:0/3761271793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2b2800 msgr2=0x55be6f2b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb33fff640 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2b2800 0x55be6f2b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb33fff640 1 -- 172.21.15.98:0/3761271793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be6f32bdf0 con 0x55be6f2a7f40 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb33fff640 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2b8830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7feb20006010 tx=0x7feb20005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7171a851f3fe5a50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb13fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb13fff640 1 -- 172.21.15.98:0/3761271793 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2004d060 con 0x55be6f2a7f40 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb13fff640 1 -- 172.21.15.98:0/3761271793 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7feb20004030 con 0x55be6f2a7f40 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.125+0000 7feb3e019740 1 -- 172.21.15.98:0/3761271793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb20040070 con 0x55be6f2a7f40 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb13fff640 1 -- 172.21.15.98:0/3761271793 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feb20036070 con 0x55be6f2a7f40 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb3e019740 10 client.15531 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb3e019740 1 -- 172.21.15.98:0/3761271793 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55be6f32d0e0 con 0x55be6f2a7f40 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb13fff640 1 -- 172.21.15.98:0/3761271793 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb20040070 con 0x55be6f2a7f40 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb13fff640 1 -- 172.21.15.98:0/3761271793 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7feb2000b040 con 0x55be6f2a7f40 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb3e019740 10 client.15531 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:42.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb3e019740 10 client.15531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:37:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb3e019740 20 client.15531 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb3e019740 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55be6f32d750 0x55be6f32fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb3e019740 4 client.15531 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:37:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb3e019740 1 -- 172.21.15.98:0/3761271793 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55be6f214200 con 0x55be6f32d750 2024-04-25T17:37:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.126+0000 7feb38fc6640 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55be6f32d750 0x55be6f32fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.127+0000 7feb13fff640 10 client.15531 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:42.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.127+0000 7feb38fc6640 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55be6f32d750 0x55be6f32fb70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee18ba5e7a88012e server_cookie=6a7c1a3cc94e4075 in_seq=0 out_seq=0 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb13fff640 1 -- 172.21.15.98:0/3761271793 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55be6f214200 con 0x55be6f32d750 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb13fff640 10 client.15531 handle_command_reply: tid=0 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 client.15531 shutdown 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 -- 172.21.15.98:0/3761271793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 msgr2=0x55be6f2b8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2b8830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7feb20006010 tx=0x7feb20005050 comp rx=0 tx=0).stop 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 -- 172.21.15.98:0/3761271793 shutdown_connections 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55be6f32d750 0x55be6f32fb70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55be6f2b2800 0x55be6f2b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55be6f2a7f40 0x55be6f2b8830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 --2- 172.21.15.98:0/3761271793 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55be6f2a7620 0x55be6f2b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb13fff640 0 client.15531 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 -- 172.21.15.98:0/3761271793 >> 172.21.15.98:0/3761271793 conn(0x55be6f2a3b30 msgr2=0x55be6f2a2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 -- 172.21.15.98:0/3761271793 shutdown_connections 2024-04-25T17:37:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 1 -- 172.21.15.98:0/3761271793 wait complete. 2024-04-25T17:37:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.140+0000 7feb3e019740 20 client.15531 trim_cache size 0 max 16384 2024-04-25T17:37:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:37:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:37:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:37:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:37:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 -- 172.21.15.98:0/88229598 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7feb240a3dc0 msgr2=0x7feb240a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7feb240a3dc0 0x7feb240a6280 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7feb34101700 tx=0x7feb2800e350 comp rx=0 tx=0).stop 2024-04-25T17:37:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 -- 172.21.15.98:0/88229598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb34106660 msgr2=0x7feb3419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb34106660 0x7feb3419cea0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00eb10 tx=0x7feb2c00c2d0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 -- 172.21.15.98:0/88229598 shutdown_connections 2024-04-25T17:37:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7feb240a3dc0 0x7feb240a6280 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb34106660 0x7feb3419cea0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb34100f80 0x7feb34198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 --2- 172.21.15.98:0/88229598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb341005c0 0x7feb341985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 -- 172.21.15.98:0/88229598 >> 172.21.15.98:0/88229598 conn(0x7feb34077370 msgr2=0x7feb341046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 -- 172.21.15.98:0/88229598 shutdown_connections 2024-04-25T17:37:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:42.141+0000 7feb3b22a640 1 -- 172.21.15.98:0/88229598 wait complete. 2024-04-25T17:37:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:42 smithi098 ceph-mon[29192]: pgmap v56: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 33 MiB/s wr, 1.66k op/s 2024-04-25T17:37:42.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:42 smithi203 ceph-mon[32834]: pgmap v56: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 33 MiB/s wr, 1.66k op/s 2024-04-25T17:37:42.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:42 smithi173 ceph-mon[32856]: pgmap v56: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 33 MiB/s wr, 1.66k op/s 2024-04-25T17:37:44.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:44 smithi098 ceph-mon[29192]: pgmap v57: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-25T17:37:44.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:44 smithi203 ceph-mon[32834]: pgmap v57: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-25T17:37:44.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:44 smithi173 ceph-mon[32856]: pgmap v57: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-25T17:37:46.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:46 smithi098 ceph-mon[29192]: pgmap v58: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-25T17:37:46.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:46 smithi203 ceph-mon[32834]: pgmap v58: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-25T17:37:46.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:46 smithi173 ceph-mon[32856]: pgmap v58: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-25T17:37:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 75.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:37:48.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:48 smithi098 ceph-mon[29192]: pgmap v59: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 14 MiB/s wr, 1.23k op/s 2024-04-25T17:37:48.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:48 smithi203 ceph-mon[32834]: pgmap v59: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 14 MiB/s wr, 1.23k op/s 2024-04-25T17:37:48.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:48 smithi173 ceph-mon[32856]: pgmap v59: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 14 MiB/s wr, 1.23k op/s 2024-04-25T17:37:50.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:50 smithi098 ceph-mon[29192]: pgmap v60: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 946 op/s 2024-04-25T17:37:50.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:50 smithi203 ceph-mon[32834]: pgmap v60: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 946 op/s 2024-04-25T17:37:50.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:50 smithi173 ceph-mon[32856]: pgmap v60: 97 pgs: 97 active+clean; 1.1 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 946 op/s 2024-04-25T17:37:51.599 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 80.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:37:52.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:52 smithi098 ceph-mon[29192]: pgmap v61: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 950 op/s 2024-04-25T17:37:52.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:52 smithi203 ceph-mon[32834]: pgmap v61: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 950 op/s 2024-04-25T17:37:52.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:52 smithi173 ceph-mon[32856]: pgmap v61: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.9 MiB/s wr, 950 op/s 2024-04-25T17:37:54.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:54 smithi098 ceph-mon[29192]: pgmap v62: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.8 MiB/s wr, 161 op/s 2024-04-25T17:37:54.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:54 smithi203 ceph-mon[32834]: pgmap v62: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.8 MiB/s wr, 161 op/s 2024-04-25T17:37:54.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:54 smithi173 ceph-mon[32856]: pgmap v62: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.8 MiB/s wr, 161 op/s 2024-04-25T17:37:55.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.239+0000 7f9a19a60640 1 -- 172.21.15.98:0/830341656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a141021d0 msgr2=0x7f9a141025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.239+0000 7f9a19a60640 1 --2- 172.21.15.98:0/830341656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a141021d0 0x7f9a141025b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009900 tx=0x7f9a0402a410 comp rx=0 tx=0).stop 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.239+0000 7f9a19a60640 1 -- 172.21.15.98:0/830341656 shutdown_connections 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.239+0000 7f9a19a60640 1 --2- 172.21.15.98:0/830341656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a141072a0 0x7f9a1410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.239+0000 7f9a19a60640 1 --2- 172.21.15.98:0/830341656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a14102b80 0x7f9a14106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.239+0000 7f9a19a60640 1 --2- 172.21.15.98:0/830341656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a141021d0 0x7f9a141025b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.239+0000 7f9a19a60640 1 -- 172.21.15.98:0/830341656 >> 172.21.15.98:0/830341656 conn(0x7f9a140793e0 msgr2=0x7f9a140797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.239+0000 7f9a19a60640 1 -- 172.21.15.98:0/830341656 shutdown_connections 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.239+0000 7f9a19a60640 1 -- 172.21.15.98:0/830341656 wait complete. 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.239+0000 7f9a19a60640 1 Processor -- start 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a19a60640 1 -- start start 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a19a60640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a141021d0 0x7f9a14198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a19a60640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a141072a0 0x7f9a14198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a19a60640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a1419cf40 0x7f9a1419d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a19a60640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a14112260 con 0x7f9a141072a0 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a19a60640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141120e0 con 0x7f9a1419cf40 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a19a60640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141123e0 con 0x7f9a141021d0 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a0bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a141072a0 0x7f9a14198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a0bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a141072a0 0x7f9a14198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34780/0 (socket says 172.21.15.98:34780) 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a0bfff640 1 -- 172.21.15.98:0/1628291087 learned_addr learned my addr 172.21.15.98:0/1628291087 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a141021d0 0x7f9a14198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a137fe640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a1419cf40 0x7f9a1419d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a0bfff640 1 -- 172.21.15.98:0/1628291087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a141021d0 msgr2=0x7f9a14198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a0bfff640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a141021d0 0x7f9a14198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a0bfff640 1 -- 172.21.15.98:0/1628291087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a1419cf40 msgr2=0x7f9a1419d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a0bfff640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a1419cf40 0x7f9a1419d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a0bfff640 1 -- 172.21.15.98:0/1628291087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1419da30 con 0x7f9a141072a0 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a0bfff640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a141072a0 0x7f9a14198bc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00e9c0 tx=0x7f99fc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.240+0000 7f9a18a5e640 1 -- 172.21.15.98:0/1628291087 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00cd80 con 0x7f9a141072a0 2024-04-25T17:37:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.241+0000 7f9a18a5e640 1 -- 172.21.15.98:0/1628291087 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99fc0054d0 con 0x7f9a141072a0 2024-04-25T17:37:55.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.241+0000 7f9a19a60640 1 -- 172.21.15.98:0/1628291087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a1419dd20 con 0x7f9a141072a0 2024-04-25T17:37:55.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.241+0000 7f9a18a5e640 1 -- 172.21.15.98:0/1628291087 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc010430 con 0x7f9a141072a0 2024-04-25T17:37:55.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.241+0000 7f9a19a60640 1 -- 172.21.15.98:0/1628291087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a141a5560 con 0x7f9a141072a0 2024-04-25T17:37:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.243+0000 7f9a18a5e640 1 -- 172.21.15.98:0/1628291087 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f99fc004fd0 con 0x7f9a141072a0 2024-04-25T17:37:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.243+0000 7f9a1c774740 1 Processor -- start 2024-04-25T17:37:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.243+0000 7f9a1c774740 1 -- start start 2024-04-25T17:37:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.243+0000 7f9a18a5e640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a000a3e40 0x7f9a000a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.243+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcd520 0x562e2abcd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.243+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abdc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.243+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abdcc10 0x562e2abdf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.243+0000 7f9a1c774740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562e2ab32970 con 0x562e2abcd520 2024-04-25T17:37:55.249 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a1c774740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562e2aacf4a0 con 0x562e2abdcc10 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a1c774740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562e2ab2e930 con 0x562e2abcde40 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a12ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcd520 0x562e2abcd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a137fe640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a000a3e40 0x7f9a000a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a12ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcd520 0x562e2abcd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34786/0 (socket says 172.21.15.98:34786) 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a12ffd640 1 -- 172.21.15.98:0/3574953876 learned_addr learned my addr 172.21.15.98:0/3574953876 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a0bfff640 1 --2- 172.21.15.98:0/3574953876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abdc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a18a5e640 1 -- 172.21.15.98:0/1628291087 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f99fc0d0b20 con 0x7f9a141072a0 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a137fe640 1 --2- 172.21.15.98:0/3574953876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abdcc10 0x562e2abdf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a12ffd640 1 -- 172.21.15.98:0/3574953876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 msgr2=0x562e2abdc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/3574953876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abdc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a12ffd640 1 -- 172.21.15.98:0/3574953876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abdcc10 msgr2=0x562e2abdf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/3574953876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abdcc10 0x562e2abdf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a12ffd640 1 -- 172.21.15.98:0/3574953876 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e2a845210 con 0x562e2abcd520 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/3574953876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcd520 0x562e2abcd900 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9a04000c00 tx=0x7f9a04004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f401ea17861064 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a137fe640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a000a3e40 0x7f9a000a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c005fd0 tx=0x7f9a0c014b70 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a09ffb640 1 -- 172.21.15.98:0/3574953876 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a04035040 con 0x562e2abcd520 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a09ffb640 1 -- 172.21.15.98:0/3574953876 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a040054f0 con 0x562e2abcd520 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.244+0000 7f9a09ffb640 1 -- 172.21.15.98:0/3574953876 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a040035f0 con 0x562e2abcd520 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.245+0000 7f9a1c774740 1 -- 172.21.15.98:0/3574953876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcd520 msgr2=0x562e2abcd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.245+0000 7f9a1c774740 1 --2- 172.21.15.98:0/3574953876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcd520 0x562e2abcd900 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9a04000c00 tx=0x7f9a04004170 comp rx=0 tx=0).stop 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.245+0000 7f9a1c774740 1 -- 172.21.15.98:0/3574953876 shutdown_connections 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.245+0000 7f9a1c774740 1 --2- 172.21.15.98:0/3574953876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abdcc10 0x562e2abdf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.245+0000 7f9a1c774740 1 --2- 172.21.15.98:0/3574953876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abdc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.245+0000 7f9a1c774740 1 --2- 172.21.15.98:0/3574953876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcd520 0x562e2abcd900 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.245+0000 7f9a1c774740 1 -- 172.21.15.98:0/3574953876 >> 172.21.15.98:0/3574953876 conn(0x562e2abc9b30 msgr2=0x562e2abc9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.245+0000 7f9a1c774740 1 -- 172.21.15.98:0/3574953876 shutdown_connections 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.245+0000 7f9a1c774740 1 -- 172.21.15.98:0/3574953876 wait complete. 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.245+0000 7f9a1c774740 1 -- start start 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a1c774740 1 Processor -- start 2024-04-25T17:37:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abdc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abd97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abd9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a1c774740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562e2abdf7e0 con 0x562e2abdcc10 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a1c774740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562e2aacd0b0 con 0x562e2abcd520 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a1c774740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562e2abdf960 con 0x562e2abcde40 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a1c774740 10 client.0 _ll_register_callbacks cb 0x562e2aabad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abd9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abd9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34800/0 (socket says 172.21.15.98:34800) 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a137fe640 1 -- 172.21.15.98:0/805822582 learned_addr learned my addr 172.21.15.98:0/805822582 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abdc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a0bfff640 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abd97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a137fe640 1 -- 172.21.15.98:0/805822582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 msgr2=0x562e2abd97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a137fe640 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abd97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a137fe640 1 -- 172.21.15.98:0/805822582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 msgr2=0x562e2abdc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a137fe640 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abdc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.246+0000 7f9a137fe640 1 -- 172.21.15.98:0/805822582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e2a911620 con 0x562e2abdcc10 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f9a137fe640 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abd9cf0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c012f60 tx=0x7f9a0c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1eb57521bf07c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f99f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f99f2ffd640 1 -- 172.21.15.98:0/805822582 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c025070 con 0x562e2abdcc10 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f99f2ffd640 1 -- 172.21.15.98:0/805822582 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f9a0c021420 con 0x562e2abdcc10 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f9a1c774740 1 -- 172.21.15.98:0/805822582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a040054f0 con 0x562e2abdcc10 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f99f2ffd640 1 -- 172.21.15.98:0/805822582 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9a0c031050 con 0x562e2abdcc10 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f9a1c774740 10 client.15543 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f9a1c774740 1 -- 172.21.15.98:0/805822582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e2abeb6d0 con 0x562e2abdcc10 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f99f2ffd640 1 -- 172.21.15.98:0/805822582 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a040054f0 con 0x562e2abdcc10 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f99f2ffd640 1 -- 172.21.15.98:0/805822582 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f9a0c00f040 con 0x562e2abdcc10 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f9a1c774740 10 client.15543 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f9a1c774740 10 client.15543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:37:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f9a1c774740 20 client.15543 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f9a1c774740 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562e2abecc00 0x562e2abef010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f9a1c774740 4 client.15543 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:37:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.247+0000 7f9a1c774740 1 -- 172.21.15.98:0/805822582 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e2ab40390 con 0x562e2abecc00 2024-04-25T17:37:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.248+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562e2abecc00 0x562e2abef010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.248+0000 7f99f2ffd640 10 client.15543 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.248+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562e2abecc00 0x562e2abef010 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a7b56b12504f529 server_cookie=1853744ebca46671 in_seq=0 out_seq=0 2024-04-25T17:37:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f99f2ffd640 1 -- 172.21.15.98:0/805822582 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e2ab40390 con 0x562e2abecc00 2024-04-25T17:37:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f99f2ffd640 10 client.15543 handle_command_reply: tid=0 2024-04-25T17:37:55.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 client.15543 shutdown 2024-04-25T17:37:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 -- 172.21.15.98:0/805822582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 msgr2=0x562e2abd9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abd9cf0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c012f60 tx=0x7f9a0c004ab0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 -- 172.21.15.98:0/805822582 shutdown_connections 2024-04-25T17:37:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562e2abecc00 0x562e2abef010 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abd9cf0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f99f2ffd640 0 client.15543 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abd97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 --2- 172.21.15.98:0/805822582 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abdc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 -- 172.21.15.98:0/805822582 >> 172.21.15.98:0/805822582 conn(0x562e2abc9b30 msgr2=0x562e2abcf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:55.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 -- 172.21.15.98:0/805822582 shutdown_connections 2024-04-25T17:37:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 1 -- 172.21.15.98:0/805822582 wait complete. 2024-04-25T17:37:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.249+0000 7f9a1c774740 20 client.15543 trim_cache size 0 max 16384 2024-04-25T17:37:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a1c774740 1 Processor -- start 2024-04-25T17:37:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a1c774740 1 -- start start 2024-04-25T17:37:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abe86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abd4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abdbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a1c774740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141ae170 con 0x562e2abdcc10 2024-04-25T17:37:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a1c774740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141aba10 con 0x562e2abcd520 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a1c774740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141a9860 con 0x562e2abcde40 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abdbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abdbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34810/0 (socket says 172.21.15.98:34810) 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a137fe640 1 -- 172.21.15.98:0/414320343 learned_addr learned my addr 172.21.15.98:0/414320343 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/414320343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abe86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a0bfff640 1 --2- 172.21.15.98:0/414320343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abd4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a137fe640 1 -- 172.21.15.98:0/414320343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 msgr2=0x562e2abd4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a137fe640 1 --2- 172.21.15.98:0/414320343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abd4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a137fe640 1 -- 172.21.15.98:0/414320343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 msgr2=0x562e2abe86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a137fe640 1 --2- 172.21.15.98:0/414320343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abe86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.264+0000 7f9a137fe640 1 -- 172.21.15.98:0/414320343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e2abce380 con 0x562e2abdcc10 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a137fe640 1 --2- 172.21.15.98:0/414320343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abdbff0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c00a010 tx=0x7f9a0c02a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8a2a2b5385c82f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a097fa640 1 -- 172.21.15.98:0/414320343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c03a070 con 0x562e2abdcc10 2024-04-25T17:37:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a097fa640 1 -- 172.21.15.98:0/414320343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0c02e070 con 0x562e2abdcc10 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a097fa640 1 -- 172.21.15.98:0/414320343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c025040 con 0x562e2abdcc10 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a1c774740 1 -- 172.21.15.98:0/414320343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 msgr2=0x562e2abdbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a1c774740 1 --2- 172.21.15.98:0/414320343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abdbff0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c00a010 tx=0x7f9a0c02a440 comp rx=0 tx=0).stop 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a1c774740 1 -- 172.21.15.98:0/414320343 shutdown_connections 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a1c774740 1 --2- 172.21.15.98:0/414320343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abdcc10 0x562e2abdbff0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a1c774740 1 --2- 172.21.15.98:0/414320343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abcde40 0x562e2abd4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a1c774740 1 --2- 172.21.15.98:0/414320343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abe86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a1c774740 1 -- 172.21.15.98:0/414320343 >> 172.21.15.98:0/414320343 conn(0x562e2abc9b30 msgr2=0x562e2abd4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a1c774740 1 -- 172.21.15.98:0/414320343 shutdown_connections 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.265+0000 7f9a1c774740 1 -- 172.21.15.98:0/414320343 wait complete. 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a1c774740 1 -- start start 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a1c774740 1 Processor -- start 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abe4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcde40 0x562e2abd6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a1c774740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abdcc10 0x562e2abd65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a1c774740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141a9ed0 con 0x562e2abcde40 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a1c774740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141ab340 con 0x562e2abcd520 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a1c774740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141aa5a0 con 0x562e2abdcc10 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a1c774740 10 client.0 _ll_register_callbacks cb 0x7f9a0c012d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a0bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcde40 0x562e2abd6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a0bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcde40 0x562e2abd6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34812/0 (socket says 172.21.15.98:34812) 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.266+0000 7f9a0bfff640 1 -- 172.21.15.98:0/948334138 learned_addr learned my addr 172.21.15.98:0/948334138 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abe4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a137fe640 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abdcc10 0x562e2abd65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a0bfff640 1 -- 172.21.15.98:0/948334138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abdcc10 msgr2=0x562e2abd65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a0bfff640 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abdcc10 0x562e2abd65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a0bfff640 1 -- 172.21.15.98:0/948334138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 msgr2=0x562e2abe4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a0bfff640 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abe4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a0bfff640 1 -- 172.21.15.98:0/948334138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e2abd6b80 con 0x562e2abcde40 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abe4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:37:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a137fe640 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abdcc10 0x562e2abd65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a0bfff640 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcde40 0x562e2abd6080 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f99fc0ce8a0 tx=0x7f99fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5455b54e3cbacb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f99f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f99f2ffd640 1 -- 172.21.15.98:0/948334138 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc09d1f0 con 0x562e2abcde40 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f99f2ffd640 1 -- 172.21.15.98:0/948334138 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f99fc0bd0b0 con 0x562e2abcde40 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a1c774740 1 -- 172.21.15.98:0/948334138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a0c02e070 con 0x562e2abcde40 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f99f2ffd640 1 -- 172.21.15.98:0/948334138 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f99fc09caf0 con 0x562e2abcde40 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a1c774740 10 client.15549 fetch_fsmap learned FSMap version 29 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f9a1c774740 1 -- 172.21.15.98:0/948334138 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e2ac52fb0 con 0x562e2abcde40 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f99f2ffd640 1 -- 172.21.15.98:0/948334138 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0c02e070 con 0x562e2abcde40 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.267+0000 7f99f2ffd640 1 -- 172.21.15.98:0/948334138 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f99fc099160 con 0x562e2abcde40 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.268+0000 7f9a1c774740 10 client.15549 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.268+0000 7f9a1c774740 10 client.15549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.268+0000 7f9a1c774740 20 client.15549 populate_metadata read hostname 'smithi098' 2024-04-25T17:37:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.268+0000 7f9a1c774740 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562e2ac53710 0x562e2ac55b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:37:55.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.268+0000 7f9a1c774740 4 client.15549 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:37:55.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.268+0000 7f9a1c774740 1 -- 172.21.15.98:0/948334138 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e2abda080 con 0x562e2ac53710 2024-04-25T17:37:55.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.268+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562e2ac53710 0x562e2ac55b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:37:55.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.268+0000 7f99f2ffd640 10 client.15549 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:55.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.268+0000 7f9a12ffd640 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562e2ac53710 0x562e2ac55b30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=130923ab2de5259b server_cookie=381900c782a10d02 in_seq=0 out_seq=0 2024-04-25T17:37:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f99f2ffd640 1 -- 172.21.15.98:0/948334138 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e2abda080 con 0x562e2ac53710 2024-04-25T17:37:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f99f2ffd640 10 client.15549 handle_command_reply: tid=0 2024-04-25T17:37:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 client.15549 shutdown 2024-04-25T17:37:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 -- 172.21.15.98:0/948334138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcde40 msgr2=0x562e2abd6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcde40 0x562e2abd6080 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f99fc0ce8a0 tx=0x7f99fc005880 comp rx=0 tx=0).stop 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 -- 172.21.15.98:0/948334138 shutdown_connections 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562e2ac53710 0x562e2ac55b30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562e2abdcc10 0x562e2abd65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562e2abcde40 0x562e2abd6080 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f99f2ffd640 0 client.15549 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 --2- 172.21.15.98:0/948334138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562e2abcd520 0x562e2abe4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 -- 172.21.15.98:0/948334138 >> 172.21.15.98:0/948334138 conn(0x562e2abc9b30 msgr2=0x562e2abc8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 -- 172.21.15.98:0/948334138 shutdown_connections 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 1 -- 172.21.15.98:0/948334138 wait complete. 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a1c774740 20 client.15549 trim_cache size 0 max 16384 2024-04-25T17:37:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:37:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:37:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:37:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:37:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a19a60640 1 -- 172.21.15.98:0/1628291087 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a000a3e40 msgr2=0x7f9a000a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a19a60640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a000a3e40 0x7f9a000a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c005fd0 tx=0x7f9a0c014b70 comp rx=0 tx=0).stop 2024-04-25T17:37:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a19a60640 1 -- 172.21.15.98:0/1628291087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a141072a0 msgr2=0x7f9a14198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:37:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.281+0000 7f9a19a60640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a141072a0 0x7f9a14198bc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00e9c0 tx=0x7f99fc00ee80 comp rx=0 tx=0).stop 2024-04-25T17:37:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.282+0000 7f9a19a60640 1 -- 172.21.15.98:0/1628291087 shutdown_connections 2024-04-25T17:37:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.282+0000 7f9a19a60640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a000a3e40 0x7f9a000a6300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.282+0000 7f9a19a60640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a1419cf40 0x7f9a1419d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.282+0000 7f9a19a60640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a141072a0 0x7f9a14198bc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.282+0000 7f9a19a60640 1 --2- 172.21.15.98:0/1628291087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a141021d0 0x7f9a14198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:37:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.282+0000 7f9a19a60640 1 -- 172.21.15.98:0/1628291087 >> 172.21.15.98:0/1628291087 conn(0x7f9a140793e0 msgr2=0x7f9a140ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:37:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.282+0000 7f9a19a60640 1 -- 172.21.15.98:0/1628291087 shutdown_connections 2024-04-25T17:37:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:37:55.282+0000 7f9a19a60640 1 -- 172.21.15.98:0/1628291087 wait complete. 2024-04-25T17:37:56.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:56 smithi098 ceph-mon[29192]: pgmap v63: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.8 MiB/s wr, 161 op/s 2024-04-25T17:37:56.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:56 smithi203 ceph-mon[32834]: pgmap v63: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.8 MiB/s wr, 161 op/s 2024-04-25T17:37:56.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:56 smithi173 ceph-mon[32856]: pgmap v63: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 1.8 MiB/s wr, 161 op/s 2024-04-25T17:37:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 85.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:37:58.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:37:58 smithi098 ceph-mon[29192]: pgmap v64: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-04-25T17:37:58.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:37:58 smithi203 ceph-mon[32834]: pgmap v64: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-04-25T17:37:58.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:37:58 smithi173 ceph-mon[32856]: pgmap v64: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.8 MiB/s wr, 167 op/s 2024-04-25T17:38:00.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:00 smithi098 ceph-mon[29192]: pgmap v65: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-25T17:38:00.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:00 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:38:00.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:00 smithi203 ceph-mon[32834]: pgmap v65: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-25T17:38:00.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:00 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:38:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:00 smithi173 ceph-mon[32856]: pgmap v65: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 15 KiB/s wr, 9 op/s 2024-04-25T17:38:00.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:00 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:38:01.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:01 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:38:01.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:01 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:38:01.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:01 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:38:01.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:01 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:38:01.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:01 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:38:01.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:01 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:38:01.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:01 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:38:01.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:01 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:38:01.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:01 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:38:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 90.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:38:02.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:02 smithi098 ceph-mon[29192]: pgmap v66: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-04-25T17:38:02.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:02 smithi203 ceph-mon[32834]: pgmap v66: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-04-25T17:38:02.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:02 smithi173 ceph-mon[32856]: pgmap v66: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 KiB/s wr, 12 op/s 2024-04-25T17:38:03.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:38:03.462 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:38:04.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:04 smithi098 ceph-mon[29192]: pgmap v67: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-04-25T17:38:04.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:04 smithi203 ceph-mon[32834]: pgmap v67: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-04-25T17:38:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:04 smithi173 ceph-mon[32856]: pgmap v67: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-04-25T17:38:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.469+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/626310067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4108a50 msgr2=0x7f7bf4108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:05.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.469+0000 7f7bfa2cd700 1 --2- 172.21.15.98:0/626310067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4108a50 0x7f7bf4108e30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009990 tx=0x7f7be4009ca0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.469+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/626310067 shutdown_connections 2024-04-25T17:38:05.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.469+0000 7f7bfa2cd700 1 --2- 172.21.15.98:0/626310067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bf4103450 0x7f7bf410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.469+0000 7f7bfa2cd700 1 --2- 172.21.15.98:0/626310067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4108a50 0x7f7bf4108e30 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.469+0000 7f7bfa2cd700 1 --2- 172.21.15.98:0/626310067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf4102a90 0x7f7bf4102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.469+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/626310067 >> 172.21.15.98:0/626310067 conn(0x7f7bf4079ba0 msgr2=0x7f7bf4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.469+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/626310067 shutdown_connections 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.469+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/626310067 wait complete. 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.470+0000 7f7bfa2cd700 1 Processor -- start 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.470+0000 7f7bfa2cd700 1 -- start start 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.470+0000 7f7bfa2cd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bf4102a90 0x7f7bf4199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.470+0000 7f7bfa2cd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4103450 0x7f7bf4199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.470+0000 7f7bfa2cd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf4108a50 0x7f7bf419d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.470+0000 7f7bfa2cd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf4112be0 con 0x7f7bf4103450 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.470+0000 7f7bfa2cd700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf4112a60 con 0x7f7bf4102a90 2024-04-25T17:38:05.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.470+0000 7f7bfa2cd700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf4112d60 con 0x7f7bf4108a50 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4103450 0x7f7bf4199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4103450 0x7f7bf4199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58248/0 (socket says 172.21.15.98:58248) 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf37fe700 1 -- 172.21.15.98:0/2606878786 learned_addr learned my addr 172.21.15.98:0/2606878786 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf886a700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf4108a50 0x7f7bf419d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf3fff700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bf4102a90 0x7f7bf4199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf37fe700 1 -- 172.21.15.98:0/2606878786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf4108a50 msgr2=0x7f7bf419d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf37fe700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf4108a50 0x7f7bf419d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf37fe700 1 -- 172.21.15.98:0/2606878786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bf4102a90 msgr2=0x7f7bf4199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf37fe700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bf4102a90 0x7f7bf4199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf37fe700 1 -- 172.21.15.98:0/2606878786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf419e120 con 0x7f7bf4103450 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf3fff700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bf4102a90 0x7f7bf4199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf37fe700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4103450 0x7f7bf4199650 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7be800c9b0 tx=0x7f7be800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:05.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf17fa700 1 -- 172.21.15.98:0/2606878786 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be8007b00 con 0x7f7bf4103450 2024-04-25T17:38:05.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf17fa700 1 -- 172.21.15.98:0/2606878786 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7be8007ca0 con 0x7f7bf4103450 2024-04-25T17:38:05.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/2606878786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf419e410 con 0x7f7bf4103450 2024-04-25T17:38:05.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bf17fa700 1 -- 172.21.15.98:0/2606878786 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be8018730 con 0x7f7bf4103450 2024-04-25T17:38:05.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.471+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/2606878786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf41a5c90 con 0x7f7bf4103450 2024-04-25T17:38:05.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.472+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/2606878786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf4069730 con 0x7f7bf4103450 2024-04-25T17:38:05.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.474+0000 7f7bf17fa700 1 -- 172.21.15.98:0/2606878786 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7be800f450 con 0x7f7bf4103450 2024-04-25T17:38:05.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.475+0000 7f7bf17fa700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7bdc0a3e30 0x7f7bdc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:05.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.475+0000 7f7bf3fff700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7bdc0a3e30 0x7f7bdc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:05.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.475+0000 7f7bf17fa700 1 -- 172.21.15.98:0/2606878786 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f7be80d0b10 con 0x7f7bf4103450 2024-04-25T17:38:05.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.475+0000 7f7bf3fff700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7bdc0a3e30 0x7f7bdc0a62f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7be4000c00 tx=0x7f7be4005fb0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:05.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.478+0000 7f7bf17fa700 1 -- 172.21.15.98:0/2606878786 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7be8014030 con 0x7f7bf4103450 2024-04-25T17:38:05.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.729+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/2606878786 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bf419a4a0 con 0x7f7bf4103450 2024-04-25T17:38:05.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.730+0000 7f7bf17fa700 1 -- 172.21.15.98:0/2606878786 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11881 (secure 0 0 0) 0x7f7bf419a4a0 con 0x7f7bf4103450 2024-04-25T17:38:05.731 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:38:05.731 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:36:39.350510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1057173981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1057173981},{"type":"v1","addr":"172.21.15.203:6837","nonce":1057173981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:38:05.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/2606878786 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7bdc0a3e30 msgr2=0x7f7bdc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:05.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7bdc0a3e30 0x7f7bdc0a62f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7be4000c00 tx=0x7f7be4005fb0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/2606878786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4103450 msgr2=0x7f7bf4199650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:05.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4103450 0x7f7bf4199650 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7be800c9b0 tx=0x7f7be800cd70 comp rx=0 tx=0).stop 2024-04-25T17:38:05.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/2606878786 shutdown_connections 2024-04-25T17:38:05.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7bdc0a3e30 0x7f7bdc0a62f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf4108a50 0x7f7bf419d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bf4102a90 0x7f7bf4199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 --2- 172.21.15.98:0/2606878786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bf4103450 0x7f7bf4199650 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:05.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/2606878786 >> 172.21.15.98:0/2606878786 conn(0x7f7bf4079ba0 msgr2=0x7f7bf4101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:05.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/2606878786 shutdown_connections 2024-04-25T17:38:05.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:05.732+0000 7f7bfa2cd700 1 -- 172.21.15.98:0/2606878786 wait complete. 2024-04-25T17:38:05.734 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-04-25T17:38:06.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:06 smithi098 ceph-mon[29192]: pgmap v68: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-04-25T17:38:06.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:06 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2606878786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:38:06.445 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id 12afe7db --release --await' 2024-04-25T17:38:06.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id 12afe7db --release --await 2024-04-25T17:38:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:06 smithi203 ceph-mon[32834]: pgmap v68: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-04-25T17:38:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:06 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2606878786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:38:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:06 smithi173 ceph-mon[32856]: pgmap v68: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.7 KiB/s wr, 8 op/s 2024-04-25T17:38:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:06 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2606878786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:38:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 95.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:38:06.696 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:38:08.312 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:08 smithi098 ceph-mon[29192]: pgmap v69: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.377+0000 7f5c00f76640 1 -- 172.21.15.98:0/1543129501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bfc102b70 msgr2=0x7f5bfc106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.377+0000 7f5c00f76640 1 --2- 172.21.15.98:0/1543129501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bfc102b70 0x7f5bfc106bb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009900 tx=0x7f5be402a410 comp rx=0 tx=0).stop 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.377+0000 7f5c00f76640 1 -- 172.21.15.98:0/1543129501 shutdown_connections 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.377+0000 7f5c00f76640 1 --2- 172.21.15.98:0/1543129501 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bfc107290 0x7f5bfc10ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.377+0000 7f5c00f76640 1 --2- 172.21.15.98:0/1543129501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bfc102b70 0x7f5bfc106bb0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.377+0000 7f5c00f76640 1 --2- 172.21.15.98:0/1543129501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bfc1021c0 0x7f5bfc1025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.377+0000 7f5c00f76640 1 -- 172.21.15.98:0/1543129501 >> 172.21.15.98:0/1543129501 conn(0x7f5bfc0fda20 msgr2=0x7f5bfc0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.377+0000 7f5c00f76640 1 -- 172.21.15.98:0/1543129501 shutdown_connections 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.377+0000 7f5c00f76640 1 -- 172.21.15.98:0/1543129501 wait complete. 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5c00f76640 1 Processor -- start 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5c00f76640 1 -- start start 2024-04-25T17:38:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5c00f76640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bfc1021c0 0x7f5bfc198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5c00f76640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bfc107290 0x7f5bfc198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5c00f76640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bfc19cf80 0x7f5bfc19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5c00f76640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc1121d0 con 0x7f5bfc107290 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5c00f76640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc112050 con 0x7f5bfc19cf80 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5c00f76640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc112350 con 0x7f5bfc1021c0 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bf3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bfc107290 0x7f5bfc198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bf3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bfc107290 0x7f5bfc198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58274/0 (socket says 172.21.15.98:58274) 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bf3fff640 1 -- 172.21.15.98:0/1127321288 learned_addr learned my addr 172.21.15.98:0/1127321288 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bfad9d640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bfc19cf80 0x7f5bfc19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bfa59c640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bfc1021c0 0x7f5bfc198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bf3fff640 1 -- 172.21.15.98:0/1127321288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bfc1021c0 msgr2=0x7f5bfc198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bfc1021c0 0x7f5bfc198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bf3fff640 1 -- 172.21.15.98:0/1127321288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bfc19cf80 msgr2=0x7f5bfc19d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bfc19cf80 0x7f5bfc19d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bf3fff640 1 -- 172.21.15.98:0/1127321288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bfc19da70 con 0x7f5bfc107290 2024-04-25T17:38:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bfa59c640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bfc1021c0 0x7f5bfc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:38:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.378+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bfc107290 0x7f5bfc198c50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5be4032fd0 tx=0x7f5be4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.379+0000 7f5bf37fe640 1 -- 172.21.15.98:0/1127321288 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be403f030 con 0x7f5bfc107290 2024-04-25T17:38:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.379+0000 7f5bf37fe640 1 -- 172.21.15.98:0/1127321288 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5be4004490 con 0x7f5bfc107290 2024-04-25T17:38:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.379+0000 7f5c00f76640 1 -- 172.21.15.98:0/1127321288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bfc19dd00 con 0x7f5bfc107290 2024-04-25T17:38:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.379+0000 7f5bf37fe640 1 -- 172.21.15.98:0/1127321288 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4035070 con 0x7f5bfc107290 2024-04-25T17:38:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.379+0000 7f5c00f76640 1 -- 172.21.15.98:0/1127321288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bfc1a54c0 con 0x7f5bfc107290 2024-04-25T17:38:08.388 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.381+0000 7f5bf37fe640 1 -- 172.21.15.98:0/1127321288 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5be4042030 con 0x7f5bfc107290 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.381+0000 7f5c03d6e740 1 Processor -- start 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5c03d6e740 1 -- start start 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28d99a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555d28cfe970 con 0x555d28d99620 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555d28c9b4a0 con 0x555d28da4800 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555d28cfa930 con 0x555d28d99f40 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bf37fe640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5be80a3dc0 0x7f5be80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bfad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28d99a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bfad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28d99a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58282/0 (socket says 172.21.15.98:58282) 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bfad9d640 1 -- 172.21.15.98:0/3668757017 learned_addr learned my addr 172.21.15.98:0/3668757017 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bfad9d640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5be80a3dc0 0x7f5be80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.389 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bf37fe640 1 -- 172.21.15.98:0/1127321288 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f5be40f0d80 con 0x7f5bfc107290 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/3668757017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bfa59c640 1 --2- 172.21.15.98:0/3668757017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bfad9d640 1 -- 172.21.15.98:0/3668757017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 msgr2=0x555d28da42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bfad9d640 1 --2- 172.21.15.98:0/3668757017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bfad9d640 1 -- 172.21.15.98:0/3668757017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 msgr2=0x555d28da6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bfad9d640 1 --2- 172.21.15.98:0/3668757017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.382+0000 7f5bfad9d640 1 -- 172.21.15.98:0/3668757017 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d28a11210 con 0x555d28d99620 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5bfad9d640 1 --2- 172.21.15.98:0/3668757017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28d99a00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4012c50 tx=0x7f5bf4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921ddcbd0498e70e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5bfad9d640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5be80a3dc0 0x7f5be80a6280 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4010c40 tx=0x7f5bf4017000 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5bf0ff9640 1 -- 172.21.15.98:0/3668757017 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4012e00 con 0x555d28d99620 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5bf0ff9640 1 -- 172.21.15.98:0/3668757017 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf4009c00 con 0x555d28d99620 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5bf0ff9640 1 -- 172.21.15.98:0/3668757017 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4005260 con 0x555d28d99620 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5c03d6e740 1 -- 172.21.15.98:0/3668757017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 msgr2=0x555d28d99a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.390 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/3668757017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28d99a00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4012c50 tx=0x7f5bf4010740 comp rx=0 tx=0).stop 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5c03d6e740 1 -- 172.21.15.98:0/3668757017 shutdown_connections 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/3668757017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/3668757017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/3668757017 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28d99a00 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.383+0000 7f5c03d6e740 1 -- 172.21.15.98:0/3668757017 >> 172.21.15.98:0/3668757017 conn(0x555d28d95b30 msgr2=0x555d28d95f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 1 -- 172.21.15.98:0/3668757017 shutdown_connections 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 1 -- 172.21.15.98:0/3668757017 wait complete. 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 1 -- start start 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 1 Processor -- start 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28da4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.391 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555d28da73d0 con 0x555d28d99620 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x555d28c990b0 con 0x555d28da4800 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555d28da7550 con 0x555d28d99f40 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5c03d6e740 10 client.0 _ll_register_callbacks cb 0x555d28c86d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5bfa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28da4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5bfa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28da4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58294/0 (socket says 172.21.15.98:58294) 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5bfa59c640 1 -- 172.21.15.98:0/4001089970 learned_addr learned my addr 172.21.15.98:0/4001089970 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.384+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5bfad9d640 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5bfa59c640 1 -- 172.21.15.98:0/4001089970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 msgr2=0x555d28da1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5bfa59c640 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5bfa59c640 1 -- 172.21.15.98:0/4001089970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 msgr2=0x555d28da1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5bfa59c640 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5bfa59c640 1 -- 172.21.15.98:0/4001089970 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d28add620 con 0x555d28d99620 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5bfad9d640 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5bfa59c640 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28da4240 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5bec010670 tx=0x7f5bec010b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a81572ca0ff345fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5be1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5be1ffb640 1 -- 172.21.15.98:0/4001089970 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec00ead0 con 0x555d28d99620 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5be1ffb640 1 -- 172.21.15.98:0/4001089970 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f5bec00ec70 con 0x555d28d99620 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5c03d6e740 1 -- 172.21.15.98:0/4001089970 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555d28d0c390 con 0x555d28d99620 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5be1ffb640 1 -- 172.21.15.98:0/4001089970 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5bec010e70 con 0x555d28d99620 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5c03d6e740 10 client.15567 fetch_fsmap learned FSMap version 29 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5c03d6e740 1 -- 172.21.15.98:0/4001089970 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555d28db3470 con 0x555d28d99620 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5be1ffb640 1 -- 172.21.15.98:0/4001089970 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555d28d0c390 con 0x555d28d99620 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5be1ffb640 1 -- 172.21.15.98:0/4001089970 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f5bec00f040 con 0x555d28d99620 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5c03d6e740 10 client.15567 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5c03d6e740 10 client.15567 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:38:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.385+0000 7f5c03d6e740 20 client.15567 populate_metadata read hostname 'smithi098' 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.386+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555d28db4980 0x555d28db6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.386+0000 7f5c03d6e740 4 client.15567 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.386+0000 7f5c03d6e740 1 -- 172.21.15.98:0/4001089970 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d28d06200 con 0x555d28db4980 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.386+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555d28db4980 0x555d28db6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.386+0000 7f5be1ffb640 10 client.15567 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.386+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555d28db4980 0x555d28db6d90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55da7eb7a0053bb server_cookie=c36f2004db84e7c1 in_seq=0 out_seq=0 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5be1ffb640 1 -- 172.21.15.98:0/4001089970 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d28d06200 con 0x555d28db4980 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5be1ffb640 10 client.15567 handle_command_reply: tid=0 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 client.15567 shutdown 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 -- 172.21.15.98:0/4001089970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 msgr2=0x555d28da4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28da4240 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5bec010670 tx=0x7f5bec010b30 comp rx=0 tx=0).stop 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 -- 172.21.15.98:0/4001089970 shutdown_connections 2024-04-25T17:38:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555d28db4980 0x555d28db6d90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5be1ffb640 0 client.15567 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/4001089970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28da4240 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 -- 172.21.15.98:0/4001089970 >> 172.21.15.98:0/4001089970 conn(0x555d28d95b30 msgr2=0x555d28d9b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 -- 172.21.15.98:0/4001089970 shutdown_connections 2024-04-25T17:38:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 1 -- 172.21.15.98:0/4001089970 wait complete. 2024-04-25T17:38:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.387+0000 7f5c03d6e740 20 client.15567 trim_cache size 0 max 16384 2024-04-25T17:38:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5c03d6e740 1 Processor -- start 2024-04-25T17:38:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5c03d6e740 1 -- start start 2024-04-25T17:38:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28d99620 0x555d28db0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99f40 0x555d28d94b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28da4800 0x555d28da3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc1b0300 con 0x555d28d99f40 2024-04-25T17:38:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc1adba0 con 0x555d28d99620 2024-04-25T17:38:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc1aba10 con 0x555d28da4800 2024-04-25T17:38:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5bf3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99f40 0x555d28d94b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5bf3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99f40 0x555d28d94b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58296/0 (socket says 172.21.15.98:58296) 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.402+0000 7f5bf3fff640 1 -- 172.21.15.98:0/1908907990 learned_addr learned my addr 172.21.15.98:0/1908907990 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5bf3fff640 1 -- 172.21.15.98:0/1908907990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28da4800 msgr2=0x555d28da3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/1908907990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28da4800 0x555d28da3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5bf3fff640 1 -- 172.21.15.98:0/1908907990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28d99620 msgr2=0x555d28db0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/1908907990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28d99620 0x555d28db0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5bf3fff640 1 -- 172.21.15.98:0/1908907990 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d28d9a8d0 con 0x555d28d99f40 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/1908907990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99f40 0x555d28d94b30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5be40ee5b0 tx=0x7f5be4046040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32deeea9c892b29f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5be3fff640 1 -- 172.21.15.98:0/1908907990 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be4049040 con 0x555d28d99f40 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5be3fff640 1 -- 172.21.15.98:0/1908907990 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5be40c59c0 con 0x555d28d99f40 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5be3fff640 1 -- 172.21.15.98:0/1908907990 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5be40b93c0 con 0x555d28d99f40 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5c03d6e740 1 -- 172.21.15.98:0/1908907990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99f40 msgr2=0x555d28d94b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/1908907990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99f40 0x555d28d94b30 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5be40ee5b0 tx=0x7f5be4046040 comp rx=0 tx=0).stop 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5c03d6e740 1 -- 172.21.15.98:0/1908907990 shutdown_connections 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/1908907990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28da4800 0x555d28da3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/1908907990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99f40 0x555d28d94b30 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/1908907990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28d99620 0x555d28db0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5c03d6e740 1 -- 172.21.15.98:0/1908907990 >> 172.21.15.98:0/1908907990 conn(0x555d28d95b30 msgr2=0x555d28d94420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5c03d6e740 1 -- 172.21.15.98:0/1908907990 shutdown_connections 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.403+0000 7f5c03d6e740 1 -- 172.21.15.98:0/1908907990 wait complete. 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5c03d6e740 1 -- start start 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5c03d6e740 1 Processor -- start 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28dacaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5c03d6e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc1ac060 con 0x555d28d99620 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc1ad4d0 con 0x555d28da4800 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5c03d6e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5bfc1ac730 con 0x555d28d99f40 2024-04-25T17:38:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5c03d6e740 10 client.0 _ll_register_callbacks cb 0x7f5bec009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5bfa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28dacaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5bfa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28dacaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58306/0 (socket says 172.21.15.98:58306) 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5bfa59c640 1 -- 172.21.15.98:0/2383091230 learned_addr learned my addr 172.21.15.98:0/2383091230 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5bfa59c640 1 -- 172.21.15.98:0/2383091230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 msgr2=0x555d28da7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5bfa59c640 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5bfa59c640 1 -- 172.21.15.98:0/2383091230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 msgr2=0x555d28da83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5bfa59c640 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5bfa59c640 1 -- 172.21.15.98:0/2383091230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d28da8bd0 con 0x555d28d99620 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.404+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5bfa59c640 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28dacaf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5bec01c680 tx=0x7f5bec009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb25302a007ef9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5be1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:38:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5be1ffb640 1 -- 172.21.15.98:0/2383091230 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec029070 con 0x555d28d99620 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5be1ffb640 1 -- 172.21.15.98:0/2383091230 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f5bec01ccd0 con 0x555d28d99620 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5c03d6e740 1 -- 172.21.15.98:0/2383091230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5be40c59c0 con 0x555d28d99620 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5be1ffb640 1 -- 172.21.15.98:0/2383091230 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5bec01c910 con 0x555d28d99620 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5c03d6e740 10 client.15573 fetch_fsmap learned FSMap version 29 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5c03d6e740 1 -- 172.21.15.98:0/2383091230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555d28e1ae80 con 0x555d28d99620 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5be1ffb640 1 -- 172.21.15.98:0/2383091230 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5be40c59c0 con 0x555d28d99620 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5be1ffb640 1 -- 172.21.15.98:0/2383091230 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f5bec012830 con 0x555d28d99620 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5c03d6e740 10 client.15573 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5c03d6e740 10 client.15573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5c03d6e740 20 client.15573 populate_metadata read hostname 'smithi098' 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555d28e1b6b0 0x555d28e1daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5c03d6e740 4 client.15573 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5c03d6e740 1 -- 172.21.15.98:0/2383091230 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d28c825f0 con 0x555d28e1b6b0 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.405+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555d28e1b6b0 0x555d28e1daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.406+0000 7f5bf3fff640 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555d28e1b6b0 0x555d28e1daa0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a00655a17fcb31d server_cookie=9d2ed2d96570ac25 in_seq=0 out_seq=0 2024-04-25T17:38:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.406+0000 7f5be1ffb640 10 client.15573 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5be1ffb640 1 -- 172.21.15.98:0/2383091230 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d28c825f0 con 0x555d28e1b6b0 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5be1ffb640 10 client.15573 handle_command_reply: tid=0 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 client.15573 shutdown 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 -- 172.21.15.98:0/2383091230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 msgr2=0x555d28dacaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28dacaf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5bec01c680 tx=0x7f5bec009c90 comp rx=0 tx=0).stop 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 -- 172.21.15.98:0/2383091230 shutdown_connections 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555d28e1b6b0 0x555d28e1daa0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555d28da4800 0x555d28da83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5be1ffb640 0 client.15573 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d28d99f40 0x555d28da7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 --2- 172.21.15.98:0/2383091230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555d28d99620 0x555d28dacaf0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 -- 172.21.15.98:0/2383091230 >> 172.21.15.98:0/2383091230 conn(0x555d28d95b30 msgr2=0x555d28d94cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 -- 172.21.15.98:0/2383091230 shutdown_connections 2024-04-25T17:38:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.418+0000 7f5c03d6e740 1 -- 172.21.15.98:0/2383091230 wait complete. 2024-04-25T17:38:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c03d6e740 20 client.15573 trim_cache size 0 max 16384 2024-04-25T17:38:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:38:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:38:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:38:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:38:08.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c00f76640 1 -- 172.21.15.98:0/1127321288 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5be80a3dc0 msgr2=0x7f5be80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c00f76640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5be80a3dc0 0x7f5be80a6280 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4010c40 tx=0x7f5bf4017000 comp rx=0 tx=0).stop 2024-04-25T17:38:08.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c00f76640 1 -- 172.21.15.98:0/1127321288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bfc107290 msgr2=0x7f5bfc198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c00f76640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bfc107290 0x7f5bfc198c50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5be4032fd0 tx=0x7f5be4004270 comp rx=0 tx=0).stop 2024-04-25T17:38:08.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c00f76640 1 -- 172.21.15.98:0/1127321288 shutdown_connections 2024-04-25T17:38:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c00f76640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5be80a3dc0 0x7f5be80a6280 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c00f76640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bfc19cf80 0x7f5bfc19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c00f76640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5bfc107290 0x7f5bfc198c50 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c00f76640 1 --2- 172.21.15.98:0/1127321288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bfc1021c0 0x7f5bfc198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.419+0000 7f5c00f76640 1 -- 172.21.15.98:0/1127321288 >> 172.21.15.98:0/1127321288 conn(0x7f5bfc0fda20 msgr2=0x7f5bfc0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.420+0000 7f5c00f76640 1 -- 172.21.15.98:0/1127321288 shutdown_connections 2024-04-25T17:38:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:08.420+0000 7f5c00f76640 1 -- 172.21.15.98:0/1127321288 wait complete. 2024-04-25T17:38:08.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:08 smithi203 ceph-mon[32834]: pgmap v69: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-25T17:38:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:08 smithi173 ceph-mon[32856]: pgmap v69: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-04-25T17:38:08.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.696+0000 7f5e28e93700 1 -- 172.21.15.98:0/3448692332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e24107e30 msgr2=0x7f5e2410f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.696+0000 7f5e28e93700 1 --2- 172.21.15.98:0/3448692332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e24107e30 0x7f5e2410f9b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00b0d0 tx=0x7f5e1c00b3e0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.697+0000 7f5e28e93700 1 -- 172.21.15.98:0/3448692332 shutdown_connections 2024-04-25T17:38:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.697+0000 7f5e28e93700 1 --2- 172.21.15.98:0/3448692332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e24101210 0x7f5e241015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.697+0000 7f5e28e93700 1 --2- 172.21.15.98:0/3448692332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e24107e30 0x7f5e2410f9b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.697+0000 7f5e28e93700 1 --2- 172.21.15.98:0/3448692332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e24101bc0 0x7f5e24107700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.697+0000 7f5e28e93700 1 -- 172.21.15.98:0/3448692332 >> 172.21.15.98:0/3448692332 conn(0x7f5e24077f40 msgr2=0x7f5e240ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.697+0000 7f5e28e93700 1 -- 172.21.15.98:0/3448692332 shutdown_connections 2024-04-25T17:38:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.697+0000 7f5e28e93700 1 -- 172.21.15.98:0/3448692332 wait complete. 2024-04-25T17:38:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.697+0000 7f5e28e93700 1 Processor -- start 2024-04-25T17:38:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.697+0000 7f5e28e93700 1 -- start start 2024-04-25T17:38:08.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e28e93700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e24101210 0x7f5e24199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e28e93700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e24101bc0 0x7f5e241997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e28e93700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e24107e30 0x7f5e2419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e28e93700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e24112d50 con 0x7f5e24107e30 2024-04-25T17:38:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e28e93700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e24112bd0 con 0x7f5e24101210 2024-04-25T17:38:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e28e93700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5e24112ed0 con 0x7f5e24101bc0 2024-04-25T17:38:08.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e22d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e24107e30 0x7f5e2419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e22d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e24107e30 0x7f5e2419db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58320/0 (socket says 172.21.15.98:58320) 2024-04-25T17:38:08.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e22d9d700 1 -- 172.21.15.98:0/2493636231 learned_addr learned my addr 172.21.15.98:0/2493636231 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:08.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e2259c700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e24101210 0x7f5e24199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e21d9b700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e24101bc0 0x7f5e241997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e22d9d700 1 -- 172.21.15.98:0/2493636231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e24101bc0 msgr2=0x7f5e241997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e24101bc0 0x7f5e241997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e22d9d700 1 -- 172.21.15.98:0/2493636231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e24101210 msgr2=0x7f5e24199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e24101210 0x7f5e24199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e22d9d700 1 -- 172.21.15.98:0/2493636231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2419e270 con 0x7f5e24107e30 2024-04-25T17:38:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.698+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e24107e30 0x7f5e2419db30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00b820 tx=0x7f5e1c0079a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.699+0000 7f5e137fe700 1 -- 172.21.15.98:0/2493636231 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c021070 con 0x7f5e24107e30 2024-04-25T17:38:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.699+0000 7f5e28e93700 1 -- 172.21.15.98:0/2493636231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2419e500 con 0x7f5e24107e30 2024-04-25T17:38:08.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.699+0000 7f5e137fe700 1 -- 172.21.15.98:0/2493636231 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e1c01db00 con 0x7f5e24107e30 2024-04-25T17:38:08.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.699+0000 7f5e28e93700 1 -- 172.21.15.98:0/2493636231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e241a5de0 con 0x7f5e24107e30 2024-04-25T17:38:08.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.699+0000 7f5e137fe700 1 -- 172.21.15.98:0/2493636231 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c01bb30 con 0x7f5e24107e30 2024-04-25T17:38:08.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.701+0000 7f5e137fe700 1 -- 172.21.15.98:0/2493636231 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5e1c024070 con 0x7f5e24107e30 2024-04-25T17:38:08.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.701+0000 7f5e137fe700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5e0c0a3db0 0x7f5e0c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.701+0000 7f5e2259c700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5e0c0a3db0 0x7f5e0c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.701+0000 7f5e137fe700 1 -- 172.21.15.98:0/2493636231 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f5e1c00f020 con 0x7f5e24107e30 2024-04-25T17:38:08.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.702+0000 7f5e2259c700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5e0c0a3db0 0x7f5e0c0a6270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009a10 tx=0x7f5e14009190 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.702+0000 7f5e3371cb80 1 Processor -- start 2024-04-25T17:38:08.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.702+0000 7f5e3371cb80 1 -- start start 2024-04-25T17:38:08.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429830a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 0x56442983b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 0x56442983e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564429783d90 con 0x564429830650 2024-04-25T17:38:08.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56442972aa30 con 0x564429830f70 2024-04-25T17:38:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56442977fd50 con 0x56442983be70 2024-04-25T17:38:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e2259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429830a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e2259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429830a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58332/0 (socket says 172.21.15.98:58332) 2024-04-25T17:38:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e2259c700 1 -- 172.21.15.98:0/74001156 learned_addr learned my addr 172.21.15.98:0/74001156 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 0x56442983e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e21d9b700 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 0x56442983b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e2259c700 1 -- 172.21.15.98:0/74001156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 msgr2=0x56442983e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e2259c700 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 0x56442983e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e2259c700 1 -- 172.21.15.98:0/74001156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 msgr2=0x56442983b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e2259c700 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 0x56442983b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e2259c700 1 -- 172.21.15.98:0/74001156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644296ee840 con 0x564429830650 2024-04-25T17:38:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e21d9b700 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 0x56442983b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:38:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 0x56442983e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:38:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e2259c700 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429830a30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400ea00 tx=0x7f5e14011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c1ac8ea348f264d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e10ff9700 1 -- 172.21.15.98:0/74001156 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1401b040 con 0x564429830650 2024-04-25T17:38:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e10ff9700 1 -- 172.21.15.98:0/74001156 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e14024070 con 0x564429830650 2024-04-25T17:38:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.703+0000 7f5e10ff9700 1 -- 172.21.15.98:0/74001156 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14013740 con 0x564429830650 2024-04-25T17:38:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.704+0000 7f5e3371cb80 1 -- 172.21.15.98:0/74001156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 msgr2=0x564429830a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.704+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429830a30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400ea00 tx=0x7f5e14011000 comp rx=0 tx=0).stop 2024-04-25T17:38:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.704+0000 7f5e3371cb80 1 -- 172.21.15.98:0/74001156 shutdown_connections 2024-04-25T17:38:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.704+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 0x56442983e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.704+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429830a30 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.704+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/74001156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 0x56442983b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.704+0000 7f5e3371cb80 1 -- 172.21.15.98:0/74001156 >> 172.21.15.98:0/74001156 conn(0x56442975d990 msgr2=0x56442982cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.704+0000 7f5e3371cb80 1 -- 172.21.15.98:0/74001156 shutdown_connections 2024-04-25T17:38:08.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.704+0000 7f5e3371cb80 1 -- 172.21.15.98:0/74001156 wait complete. 2024-04-25T17:38:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e3371cb80 1 -- start start 2024-04-25T17:38:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e3371cb80 1 Processor -- start 2024-04-25T17:38:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x56442983b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 0x564429838920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56442983be70 0x564429838e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56442983e900 con 0x564429830650 2024-04-25T17:38:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56442973d6b0 con 0x56442983be70 2024-04-25T17:38:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56442983ea80 con 0x564429830f70 2024-04-25T17:38:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e3371cb80 10 client.0 _ll_register_callbacks cb 0x56442971ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:38:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e2259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x56442983b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e2259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x56442983b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58334/0 (socket says 172.21.15.98:58334) 2024-04-25T17:38:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e2259c700 1 -- 172.21.15.98:0/1645255292 learned_addr learned my addr 172.21.15.98:0/1645255292 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56442983be70 0x564429838e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e21d9b700 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 0x564429838920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e2259c700 1 -- 172.21.15.98:0/1645255292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 msgr2=0x564429838920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e2259c700 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 0x564429838920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e2259c700 1 -- 172.21.15.98:0/1645255292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56442983be70 msgr2=0x564429838e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e2259c700 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56442983be70 0x564429838e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e2259c700 1 -- 172.21.15.98:0/1645255292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644296d19d0 con 0x564429830650 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.705+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56442983be70 0x564429838e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e2259c700 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x56442983b620 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009330 tx=0x7f5e14015a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a69d301e991b362 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e09ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e09ffb700 1 -- 172.21.15.98:0/1645255292 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1402d070 con 0x564429830650 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e09ffb700 1 -- 172.21.15.98:0/1645255292 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f5e14029810 con 0x564429830650 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e3371cb80 1 -- 172.21.15.98:0/1645255292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e14024070 con 0x564429830650 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e09ffb700 1 -- 172.21.15.98:0/1645255292 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e14029a10 con 0x564429830650 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e3371cb80 10 client.15585 fetch_fsmap learned FSMap version 29 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e3371cb80 1 -- 172.21.15.98:0/1645255292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564429849150 con 0x564429830650 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e09ffb700 1 -- 172.21.15.98:0/1645255292 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e14024070 con 0x564429830650 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e09ffb700 1 -- 172.21.15.98:0/1645255292 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f5e1401e030 con 0x564429830650 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e3371cb80 10 client.15585 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:38:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e3371cb80 10 client.15585 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e3371cb80 20 client.15585 populate_metadata read hostname 'smithi098' 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56442984b730 0x56442984db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e3371cb80 4 client.15585 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.706+0000 7f5e3371cb80 1 -- 172.21.15.98:0/1645255292 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564429791580 con 0x56442984b730 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.707+0000 7f5e21d9b700 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56442984b730 0x56442984db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.707+0000 7f5e21d9b700 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56442984b730 0x56442984db20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=675503eddf975d63 server_cookie=ee8dfc7e1d433d41 in_seq=0 out_seq=0 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.707+0000 7f5e09ffb700 10 client.15585 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e09ffb700 1 -- 172.21.15.98:0/1645255292 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564429791580 con 0x56442984b730 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e09ffb700 10 client.15585 handle_command_reply: tid=0 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 client.15585 shutdown 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 -- 172.21.15.98:0/1645255292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 msgr2=0x56442983b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x56442983b620 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009330 tx=0x7f5e14015a20 comp rx=0 tx=0).stop 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 -- 172.21.15.98:0/1645255292 shutdown_connections 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56442984b730 0x56442984db20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56442983be70 0x564429838e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x56442983b620 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/1645255292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 0x564429838920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 -- 172.21.15.98:0/1645255292 >> 172.21.15.98:0/1645255292 conn(0x56442975d990 msgr2=0x564429832ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e09ffb700 0 client.15585 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 -- 172.21.15.98:0/1645255292 shutdown_connections 2024-04-25T17:38:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 1 -- 172.21.15.98:0/1645255292 wait complete. 2024-04-25T17:38:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.708+0000 7f5e3371cb80 20 client.15585 trim_cache size 0 max 16384 2024-04-25T17:38:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e3371cb80 1 Processor -- start 2024-04-25T17:38:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e3371cb80 1 -- start start 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429847800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 0x564429853180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 0x5644298536c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e241afc30 con 0x564429830650 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e241ad490 con 0x564429830f70 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5e241ab2e0 con 0x56442983be70 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e2259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429847800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e2259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429847800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58338/0 (socket says 172.21.15.98:58338) 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.727+0000 7f5e2259c700 1 -- 172.21.15.98:0/2205940592 learned_addr learned my addr 172.21.15.98:0/2205940592 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 0x5644298536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e21d9b700 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 0x564429853180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e2259c700 1 -- 172.21.15.98:0/2205940592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 msgr2=0x5644298536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e2259c700 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 0x5644298536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e2259c700 1 -- 172.21.15.98:0/2205940592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 msgr2=0x564429853180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e2259c700 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 0x564429853180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e2259c700 1 -- 172.21.15.98:0/2205940592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56442982c330 con 0x564429830650 2024-04-25T17:38:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e21d9b700 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 0x564429853180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:38:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 0x5644298536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:38:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e2259c700 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429847800 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5e14015bc0 tx=0x7f5e1400e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede4765c2c9f10bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e0bfff700 1 -- 172.21.15.98:0/2205940592 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1403f030 con 0x564429830650 2024-04-25T17:38:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e0bfff700 1 -- 172.21.15.98:0/2205940592 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e14024040 con 0x564429830650 2024-04-25T17:38:08.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.728+0000 7f5e0bfff700 1 -- 172.21.15.98:0/2205940592 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1402d040 con 0x564429830650 2024-04-25T17:38:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 -- 172.21.15.98:0/2205940592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 msgr2=0x564429847800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429847800 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5e14015bc0 tx=0x7f5e1400e4f0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 -- 172.21.15.98:0/2205940592 shutdown_connections 2024-04-25T17:38:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56442983be70 0x5644298536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564429830650 0x564429847800 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/2205940592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830f70 0x564429853180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 -- 172.21.15.98:0/2205940592 >> 172.21.15.98:0/2205940592 conn(0x56442975d990 msgr2=0x5644298430e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 -- 172.21.15.98:0/2205940592 shutdown_connections 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 -- 172.21.15.98:0/2205940592 wait complete. 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 -- start start 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 Processor -- start 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830650 0x5644298aef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 0x5644298af490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56442983be70 0x5644298a9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e241ab950 con 0x56442983be70 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e241acdc0 con 0x564429830650 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5e241ac020 con 0x564429830f70 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e3371cb80 10 client.0 _ll_register_callbacks cb 0x56442971ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.729+0000 7f5e22d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56442983be70 0x5644298a9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e22d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56442983be70 0x5644298a9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58344/0 (socket says 172.21.15.98:58344) 2024-04-25T17:38:08.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e22d9d700 1 -- 172.21.15.98:0/488832579 learned_addr learned my addr 172.21.15.98:0/488832579 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e2259c700 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830650 0x5644298aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e21d9b700 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 0x5644298af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e22d9d700 1 -- 172.21.15.98:0/488832579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 msgr2=0x5644298af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 0x5644298af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e22d9d700 1 -- 172.21.15.98:0/488832579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830650 msgr2=0x5644298aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830650 0x5644298aef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e22d9d700 1 -- 172.21.15.98:0/488832579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644298a9a70 con 0x56442983be70 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e21d9b700 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 0x5644298af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e22d9d700 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56442983be70 0x5644298a9370 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0095c0 tx=0x7f5e1c01b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd2656b30b765db9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e09ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e09ffb700 1 -- 172.21.15.98:0/488832579 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c09dde0 con 0x56442983be70 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e09ffb700 1 -- 172.21.15.98:0/488832579 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f5e1c02e350 con 0x56442983be70 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e3371cb80 1 -- 172.21.15.98:0/488832579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e14024040 con 0x56442983be70 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e09ffb700 1 -- 172.21.15.98:0/488832579 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e1c02e550 con 0x56442983be70 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e3371cb80 10 client.15591 fetch_fsmap learned FSMap version 29 2024-04-25T17:38:08.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e3371cb80 1 -- 172.21.15.98:0/488832579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5644298b6790 con 0x56442983be70 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.730+0000 7f5e09ffb700 1 -- 172.21.15.98:0/488832579 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e14024040 con 0x56442983be70 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.731+0000 7f5e09ffb700 1 -- 172.21.15.98:0/488832579 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f5e1c011070 con 0x56442983be70 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.731+0000 7f5e3371cb80 10 client.15591 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.731+0000 7f5e3371cb80 10 client.15591 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.731+0000 7f5e3371cb80 20 client.15591 populate_metadata read hostname 'smithi098' 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.731+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5644298b6e50 0x5644298b9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.731+0000 7f5e3371cb80 4 client.15591 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "12afe7db", "release": true, "await": true}] 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.731+0000 7f5e3371cb80 1 -- 172.21.15.98:0/488832579 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "12afe7db", "release": true, "await": true}) -- 0x56442978b620 con 0x5644298b6e50 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.731+0000 7f5e2259c700 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5644298b6e50 0x5644298b9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.731+0000 7f5e09ffb700 10 client.15591 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:08.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.731+0000 7f5e2259c700 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5644298b6e50 0x5644298b9270 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbab3a5d7e6018ee server_cookie=e244b3f68ff6e2fc in_seq=0 out_seq=0 2024-04-25T17:38:08.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.772+0000 7f5e09ffb700 1 -- 172.21.15.98:0/488832579 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x56442978b620 con 0x5644298b6e50 2024-04-25T17:38:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.772+0000 7f5e09ffb700 10 client.15591 handle_command_reply: tid=0 2024-04-25T17:38:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.772+0000 7f5e3371cb80 1 client.15591 shutdown 2024-04-25T17:38:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.772+0000 7f5e3371cb80 1 -- 172.21.15.98:0/488832579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56442983be70 msgr2=0x5644298a9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.772+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56442983be70 0x5644298a9370 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c0095c0 tx=0x7f5e1c01b600 comp rx=0 tx=0).stop 2024-04-25T17:38:08.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.775+0000 7f5e3371cb80 1 -- 172.21.15.98:0/488832579 shutdown_connections 2024-04-25T17:38:08.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.775+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5644298b6e50 0x5644298b9270 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.775+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56442983be70 0x5644298a9370 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.775+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564429830650 0x5644298aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.775+0000 7f5e3371cb80 1 --2- 172.21.15.98:0/488832579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564429830f70 0x5644298af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.775+0000 7f5e3371cb80 1 -- 172.21.15.98:0/488832579 >> 172.21.15.98:0/488832579 conn(0x56442975d990 msgr2=0x564429843970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.775+0000 7f5e09ffb700 0 client.15591 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:08.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.775+0000 7f5e3371cb80 1 -- 172.21.15.98:0/488832579 shutdown_connections 2024-04-25T17:38:08.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.776+0000 7f5e3371cb80 1 -- 172.21.15.98:0/488832579 wait complete. 2024-04-25T17:38:08.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.777+0000 7f5e3371cb80 20 client.15591 trim_cache size 0 max 16384 2024-04-25T17:38:08.777 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 29, 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 8, 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "12afe7db": { 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 8, 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:38:08.778 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 87.5, 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:38:08.779 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:38:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.779+0000 7f5e28e93700 1 -- 172.21.15.98:0/2493636231 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5e0c0a3db0 msgr2=0x7f5e0c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.779+0000 7f5e28e93700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5e0c0a3db0 0x7f5e0c0a6270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009a10 tx=0x7f5e14009190 comp rx=0 tx=0).stop 2024-04-25T17:38:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.779+0000 7f5e28e93700 1 -- 172.21.15.98:0/2493636231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e24107e30 msgr2=0x7f5e2419db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.779+0000 7f5e28e93700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e24107e30 0x7f5e2419db30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00b820 tx=0x7f5e1c0079a0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.779+0000 7f5e28e93700 1 -- 172.21.15.98:0/2493636231 shutdown_connections 2024-04-25T17:38:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.779+0000 7f5e28e93700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5e0c0a3db0 0x7f5e0c0a6270 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.780+0000 7f5e28e93700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e24107e30 0x7f5e2419db30 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.780+0000 7f5e28e93700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e24101210 0x7f5e24199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.780+0000 7f5e28e93700 1 --2- 172.21.15.98:0/2493636231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e24101bc0 0x7f5e241997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.780+0000 7f5e28e93700 1 -- 172.21.15.98:0/2493636231 >> 172.21.15.98:0/2493636231 conn(0x7f5e24077f40 msgr2=0x7f5e240ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.780+0000 7f5e28e93700 1 -- 172.21.15.98:0/2493636231 shutdown_connections 2024-04-25T17:38:08.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:38:08.782+0000 7f5e28e93700 1 -- 172.21.15.98:0/2493636231 wait complete. 2024-04-25T17:38:09.760 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '12afe7db', seconds elapsed: 41.2 2024-04-25T17:38:10.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:10 smithi098 ceph-mon[29192]: pgmap v70: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 38 KiB/s wr, 23 op/s 2024-04-25T17:38:10.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:10 smithi203 ceph-mon[32834]: pgmap v70: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 38 KiB/s wr, 23 op/s 2024-04-25T17:38:10.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:10 smithi173 ceph-mon[32856]: pgmap v70: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 38 KiB/s wr, 23 op/s 2024-04-25T17:38:11.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 100.0 s, 1831.1 tps, lat 403.489 ms stddev 3551.308 2024-04-25T17:38:12.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:12 smithi098 ceph-mon[29192]: pgmap v71: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 968 KiB/s wr, 123 op/s 2024-04-25T17:38:12.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:12 smithi203 ceph-mon[32834]: pgmap v71: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 968 KiB/s wr, 123 op/s 2024-04-25T17:38:12.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:12 smithi173 ceph-mon[32856]: pgmap v71: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 968 KiB/s wr, 123 op/s 2024-04-25T17:38:14.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:14 smithi098 ceph-mon[29192]: pgmap v72: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 967 KiB/s wr, 120 op/s 2024-04-25T17:38:14.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:14 smithi203 ceph-mon[32834]: pgmap v72: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 967 KiB/s wr, 120 op/s 2024-04-25T17:38:14.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:14 smithi173 ceph-mon[32856]: pgmap v72: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 967 KiB/s wr, 120 op/s 2024-04-25T17:38:16.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:16 smithi098 ceph-mon[29192]: pgmap v73: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 967 KiB/s wr, 120 op/s 2024-04-25T17:38:16.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:16 smithi203 ceph-mon[32834]: pgmap v73: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 967 KiB/s wr, 120 op/s 2024-04-25T17:38:16.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:16 smithi173 ceph-mon[32856]: pgmap v73: 97 pgs: 97 active+clean; 1.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 967 KiB/s wr, 120 op/s 2024-04-25T17:38:16.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 105.0 s, 3805.8 tps, lat 26.204 ms stddev 25.873 2024-04-25T17:38:18.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:18 smithi098 ceph-mon[29192]: pgmap v74: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.2 MiB/s wr, 446 op/s 2024-04-25T17:38:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:18 smithi203 ceph-mon[32834]: pgmap v74: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.2 MiB/s wr, 446 op/s 2024-04-25T17:38:18.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:18 smithi173 ceph-mon[32856]: pgmap v74: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.2 MiB/s wr, 446 op/s 2024-04-25T17:38:20.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:20 smithi098 ceph-mon[29192]: pgmap v75: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.2 MiB/s wr, 425 op/s 2024-04-25T17:38:20.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:20 smithi203 ceph-mon[32834]: pgmap v75: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.2 MiB/s wr, 425 op/s 2024-04-25T17:38:20.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:20 smithi173 ceph-mon[32856]: pgmap v75: 97 pgs: 97 active+clean; 1.1 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.2 MiB/s wr, 425 op/s 2024-04-25T17:38:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.538+0000 7ff39c736640 1 -- 172.21.15.98:0/3041303245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff394101330 msgr2=0x7ff394101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.538+0000 7ff39c736640 1 --2- 172.21.15.98:0/3041303245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff394101330 0x7ff394101710 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff390009900 tx=0x7ff39002a400 comp rx=0 tx=0).stop 2024-04-25T17:38:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.538+0000 7ff39c736640 1 -- 172.21.15.98:0/3041303245 shutdown_connections 2024-04-25T17:38:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.538+0000 7ff39c736640 1 --2- 172.21.15.98:0/3041303245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3941061d0 0x7ff39410dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.538+0000 7ff39c736640 1 --2- 172.21.15.98:0/3041303245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff394101ce0 0x7ff394105af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.538+0000 7ff39c736640 1 --2- 172.21.15.98:0/3041303245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff394101330 0x7ff394101710 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.538+0000 7ff39c736640 1 -- 172.21.15.98:0/3041303245 >> 172.21.15.98:0/3041303245 conn(0x7ff394077d30 msgr2=0x7ff394078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.538+0000 7ff39c736640 1 -- 172.21.15.98:0/3041303245 shutdown_connections 2024-04-25T17:38:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.538+0000 7ff39c736640 1 -- 172.21.15.98:0/3041303245 wait complete. 2024-04-25T17:38:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39c736640 1 Processor -- start 2024-04-25T17:38:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39c736640 1 -- start start 2024-04-25T17:38:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39c736640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff394101330 0x7ff394198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39c736640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3941061d0 0x7ff394198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39c736640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff39419cff0 0x7ff39419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39c736640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff394112230 con 0x7ff39419cff0 2024-04-25T17:38:21.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39c736640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941120b0 con 0x7ff3941061d0 2024-04-25T17:38:21.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39c736640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941123b0 con 0x7ff394101330 2024-04-25T17:38:21.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39acd3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff39419cff0 0x7ff39419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39acd3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff39419cff0 0x7ff39419d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32990/0 (socket says 172.21.15.98:32990) 2024-04-25T17:38:21.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39acd3640 1 -- 172.21.15.98:0/2140923610 learned_addr learned my addr 172.21.15.98:0/2140923610 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff399cd1640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3941061d0 0x7ff394198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.539+0000 7ff39a4d2640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff394101330 0x7ff394198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff39acd3640 1 -- 172.21.15.98:0/2140923610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff394101330 msgr2=0x7ff394198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff39acd3640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff394101330 0x7ff394198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff39acd3640 1 -- 172.21.15.98:0/2140923610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3941061d0 msgr2=0x7ff394198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff39acd3640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3941061d0 0x7ff394198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff39acd3640 1 -- 172.21.15.98:0/2140923610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39419dae0 con 0x7ff39419cff0 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff39acd3640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff39419cff0 0x7ff39419d4a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00eaa0 tx=0x7ff38c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff38b7fe640 1 -- 172.21.15.98:0/2140923610 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c00cdb0 con 0x7ff39419cff0 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff39c736640 1 -- 172.21.15.98:0/2140923610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff39419ddd0 con 0x7ff39419cff0 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff38b7fe640 1 -- 172.21.15.98:0/2140923610 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff38c005020 con 0x7ff39419cff0 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff38b7fe640 1 -- 172.21.15.98:0/2140923610 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c0116d0 con 0x7ff39419cff0 2024-04-25T17:38:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.540+0000 7ff39c736640 1 -- 172.21.15.98:0/2140923610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3941a55c0 con 0x7ff39419cff0 2024-04-25T17:38:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff38b7fe640 1 -- 172.21.15.98:0/2140923610 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff38c0051d0 con 0x7ff39419cff0 2024-04-25T17:38:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff39f451740 1 Processor -- start 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff39f451740 1 -- start start 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff39f451740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b10a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff39f451740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10f40 0x5566a2b1b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff39f451740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 0x5566a2b1dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff39f451740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5566a2a75970 con 0x5566a2b1b800 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff39f451740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5566a2a124a0 con 0x5566a2b10f40 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff39f451740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5566a2a71930 con 0x5566a2b10620 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff399cd1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 0x5566a2b1dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff38b7fe640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff3800a3e40 0x7ff3800a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff399cd1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 0x5566a2b1dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32994/0 (socket says 172.21.15.98:32994) 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff399cd1640 1 -- 172.21.15.98:0/1871764145 learned_addr learned my addr 172.21.15.98:0/1871764145 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.545+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1871764145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b10a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff39a4d2640 1 --2- 172.21.15.98:0/1871764145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10f40 0x5566a2b1b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff39acd3640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff3800a3e40 0x7ff3800a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff38b7fe640 1 -- 172.21.15.98:0/2140923610 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7ff38c0d08d0 con 0x7ff39419cff0 2024-04-25T17:38:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff399cd1640 1 -- 172.21.15.98:0/1871764145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 msgr2=0x5566a2b10a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1871764145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b10a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff399cd1640 1 -- 172.21.15.98:0/1871764145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10f40 msgr2=0x5566a2b1b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1871764145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10f40 0x5566a2b1b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff399cd1640 1 -- 172.21.15.98:0/1871764145 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566a2788210 con 0x5566a2b1b800 2024-04-25T17:38:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1871764145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 0x5566a2b1dbf0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff384012720 tx=0x7ff3840096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a315fd67e509eae4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:21.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff39acd3640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff3800a3e40 0x7ff3800a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff38c0ce100 tx=0x7ff38c01eb40 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:21.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff388ff9640 1 -- 172.21.15.98:0/1871764145 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384012ca0 con 0x5566a2b1b800 2024-04-25T17:38:21.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff388ff9640 1 -- 172.21.15.98:0/1871764145 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff384012e40 con 0x5566a2b1b800 2024-04-25T17:38:21.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.546+0000 7ff388ff9640 1 -- 172.21.15.98:0/1871764145 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384004e00 con 0x5566a2b1b800 2024-04-25T17:38:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.547+0000 7ff39f451740 1 -- 172.21.15.98:0/1871764145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 msgr2=0x5566a2b1dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.547+0000 7ff39f451740 1 --2- 172.21.15.98:0/1871764145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 0x5566a2b1dbf0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff384012720 tx=0x7ff3840096c0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.547+0000 7ff39f451740 1 -- 172.21.15.98:0/1871764145 shutdown_connections 2024-04-25T17:38:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.547+0000 7ff39f451740 1 --2- 172.21.15.98:0/1871764145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 0x5566a2b1dbf0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.547+0000 7ff39f451740 1 --2- 172.21.15.98:0/1871764145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10f40 0x5566a2b1b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.547+0000 7ff39f451740 1 --2- 172.21.15.98:0/1871764145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b10a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.547+0000 7ff39f451740 1 -- 172.21.15.98:0/1871764145 >> 172.21.15.98:0/1871764145 conn(0x5566a2b0cb30 msgr2=0x5566a2b0cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.548+0000 7ff39f451740 1 -- 172.21.15.98:0/1871764145 shutdown_connections 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.548+0000 7ff39f451740 1 -- 172.21.15.98:0/1871764145 wait complete. 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.548+0000 7ff39f451740 1 -- start start 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.548+0000 7ff39f451740 1 Processor -- start 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39f451740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b1b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39f451740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b183f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39f451740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b1b800 0x5566a2b18930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39f451740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5566a2b22840 con 0x5566a2b10f40 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39f451740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5566a2a100b0 con 0x5566a2b1b800 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39f451740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5566a2b229c0 con 0x5566a2b10620 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39f451740 10 client.0 _ll_register_callbacks cb 0x5566a29fdd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39a4d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b183f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff399cd1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b1b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39a4d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b183f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33008/0 (socket says 172.21.15.98:33008) 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39a4d2640 1 -- 172.21.15.98:0/1216761317 learned_addr learned my addr 172.21.15.98:0/1216761317 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b1b800 0x5566a2b18930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff399cd1640 1 -- 172.21.15.98:0/1216761317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b1b800 msgr2=0x5566a2b18930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b1b800 0x5566a2b18930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff399cd1640 1 -- 172.21.15.98:0/1216761317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 msgr2=0x5566a2b183f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b183f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff399cd1640 1 -- 172.21.15.98:0/1216761317 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566a2854620 con 0x5566a2b10620 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.549+0000 7ff39a4d2640 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b183f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b1b100 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff384001eb0 tx=0x7ff384017040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=511b2ddb5a53edbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff37dffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff37dffb640 1 -- 172.21.15.98:0/1216761317 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384004e00 con 0x5566a2b10620 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff37dffb640 1 -- 172.21.15.98:0/1216761317 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7ff384004ba0 con 0x5566a2b10620 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff39f451740 1 -- 172.21.15.98:0/1216761317 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5566a2a83390 con 0x5566a2b10620 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff37dffb640 1 -- 172.21.15.98:0/1216761317 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff384004020 con 0x5566a2b10620 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff39f451740 10 client.25135 fetch_fsmap learned FSMap version 29 2024-04-25T17:38:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff39f451740 1 -- 172.21.15.98:0/1216761317 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5566a2b2e750 con 0x5566a2b10620 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff37dffb640 1 -- 172.21.15.98:0/1216761317 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5566a2a83390 con 0x5566a2b10620 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff37dffb640 1 -- 172.21.15.98:0/1216761317 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7ff3840035f0 con 0x5566a2b10620 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff39f451740 10 client.25135 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff39f451740 10 client.25135 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff39f451740 20 client.25135 populate_metadata read hostname 'smithi098' 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff39f451740 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5566a2b2faa0 0x5566a2b31e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff39f451740 4 client.25135 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff39f451740 1 -- 172.21.15.98:0/1216761317 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566a2a7d200 con 0x5566a2b2faa0 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.550+0000 7ff39a4d2640 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5566a2b2faa0 0x5566a2b31e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.551+0000 7ff39a4d2640 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5566a2b2faa0 0x5566a2b31e90 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ed2e6ed75240313 server_cookie=8946175b0c38a9a8 in_seq=0 out_seq=0 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.551+0000 7ff37dffb640 10 client.25135 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.551+0000 7ff37dffb640 1 -- 172.21.15.98:0/1216761317 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566a2a7d200 con 0x5566a2b2faa0 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.551+0000 7ff37dffb640 10 client.25135 handle_command_reply: tid=0 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.551+0000 7ff39f451740 1 client.25135 shutdown 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.551+0000 7ff39f451740 1 -- 172.21.15.98:0/1216761317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 msgr2=0x5566a2b1b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.551+0000 7ff39f451740 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b1b100 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff384001eb0 tx=0x7ff384017040 comp rx=0 tx=0).stop 2024-04-25T17:38:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.551+0000 7ff39f451740 1 -- 172.21.15.98:0/1216761317 shutdown_connections 2024-04-25T17:38:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.552+0000 7ff39f451740 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5566a2b2faa0 0x5566a2b31e90 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.552+0000 7ff39f451740 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b1b800 0x5566a2b18930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.552+0000 7ff39f451740 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b183f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.552+0000 7ff39f451740 1 --2- 172.21.15.98:0/1216761317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b1b100 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.552+0000 7ff39f451740 1 -- 172.21.15.98:0/1216761317 >> 172.21.15.98:0/1216761317 conn(0x5566a2b0cb30 msgr2=0x5566a2b128b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.552+0000 7ff37dffb640 0 client.25135 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.552+0000 7ff39f451740 1 -- 172.21.15.98:0/1216761317 shutdown_connections 2024-04-25T17:38:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.552+0000 7ff39f451740 1 -- 172.21.15.98:0/1216761317 wait complete. 2024-04-25T17:38:21.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.552+0000 7ff39f451740 20 client.25135 trim_cache size 0 max 16384 2024-04-25T17:38:21.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff39f451740 1 Processor -- start 2024-04-25T17:38:21.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff39f451740 1 -- start start 2024-04-25T17:38:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff39f451740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10620 0x5566a2b1ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff39f451740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10f40 0x5566a2b37690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff39f451740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 0x5566a2b37bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff39f451740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941ae210 con 0x5566a2b1b800 2024-04-25T17:38:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff39f451740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941abab0 con 0x5566a2b10620 2024-04-25T17:38:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff39f451740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941a9920 con 0x5566a2b10f40 2024-04-25T17:38:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff399cd1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10620 0x5566a2b1ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff399cd1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10620 0x5566a2b1ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:44716/0 (socket says 172.21.15.98:44716) 2024-04-25T17:38:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.574+0000 7ff399cd1640 1 -- 172.21.15.98:0/3553182246 learned_addr learned my addr 172.21.15.98:0/3553182246 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.575+0000 7ff399cd1640 1 --2- 172.21.15.98:0/3553182246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 0x5566a2b37bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.575+0000 7ff399cd1640 1 -- 172.21.15.98:0/3553182246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10f40 msgr2=0x5566a2b37690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T17:38:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.575+0000 7ff399cd1640 1 --2- 172.21.15.98:0/3553182246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10f40 0x5566a2b37690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.575+0000 7ff399cd1640 1 -- 172.21.15.98:0/3553182246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 msgr2=0x5566a2b37bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.575+0000 7ff399cd1640 1 --2- 172.21.15.98:0/3553182246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 0x5566a2b37bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.575+0000 7ff399cd1640 1 -- 172.21.15.98:0/3553182246 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566a2b08500 con 0x5566a2b10620 2024-04-25T17:38:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.575+0000 7ff399cd1640 1 --2- 172.21.15.98:0/3553182246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10620 0x5566a2b1ac30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x5566a2b38110 tx=0x7ff3840096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94c72f066ea64f88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.575+0000 7ff37ffff640 1 -- 172.21.15.98:0/3553182246 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3840099b0 con 0x5566a2b10620 2024-04-25T17:38:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.575+0000 7ff37ffff640 1 -- 172.21.15.98:0/3553182246 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff384005a70 con 0x5566a2b10620 2024-04-25T17:38:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.575+0000 7ff37ffff640 1 -- 172.21.15.98:0/3553182246 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384004420 con 0x5566a2b10620 2024-04-25T17:38:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.576+0000 7ff39f451740 1 -- 172.21.15.98:0/3553182246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10620 msgr2=0x5566a2b1ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.576+0000 7ff39f451740 1 --2- 172.21.15.98:0/3553182246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10620 0x5566a2b1ac30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x5566a2b38110 tx=0x7ff3840096c0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.576+0000 7ff39f451740 1 -- 172.21.15.98:0/3553182246 shutdown_connections 2024-04-25T17:38:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.576+0000 7ff39f451740 1 --2- 172.21.15.98:0/3553182246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b1b800 0x5566a2b37bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.576+0000 7ff39f451740 1 --2- 172.21.15.98:0/3553182246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10f40 0x5566a2b37690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.576+0000 7ff39f451740 1 --2- 172.21.15.98:0/3553182246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b10620 0x5566a2b1ac30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.576+0000 7ff39f451740 1 -- 172.21.15.98:0/3553182246 >> 172.21.15.98:0/3553182246 conn(0x5566a2b0cb30 msgr2=0x5566a2b31ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.576+0000 7ff39f451740 1 -- 172.21.15.98:0/3553182246 shutdown_connections 2024-04-25T17:38:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.576+0000 7ff39f451740 1 -- 172.21.15.98:0/3553182246 wait complete. 2024-04-25T17:38:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39f451740 1 -- start start 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39f451740 1 Processor -- start 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39f451740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b1fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39f451740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b202f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39f451740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b1b800 0x5566a2b25c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39f451740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941a9f70 con 0x5566a2b10f40 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39f451740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941ab3e0 con 0x5566a2b1b800 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39f451740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941aa640 con 0x5566a2b10620 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39f451740 10 client.0 _ll_register_callbacks cb 0x7ff384005400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39a4d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b202f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39a4d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b202f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33028/0 (socket says 172.21.15.98:33028) 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39a4d2640 1 -- 172.21.15.98:0/1060891290 learned_addr learned my addr 172.21.15.98:0/1060891290 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b1fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b1b800 0x5566a2b25c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39a4d2640 1 -- 172.21.15.98:0/1060891290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 msgr2=0x5566a2b1fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39a4d2640 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b1fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39a4d2640 1 -- 172.21.15.98:0/1060891290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b1b800 msgr2=0x5566a2b25c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39a4d2640 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b1b800 0x5566a2b25c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39a4d2640 1 -- 172.21.15.98:0/1060891290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566a2b261f0 con 0x5566a2b10f40 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff39a4d2640 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b202f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff39002a920 tx=0x7ff39002a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f4b933cb8333593 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.577+0000 7ff37dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff37dffb640 1 -- 172.21.15.98:0/1060891290 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39004a060 con 0x5566a2b10f40 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff37dffb640 1 -- 172.21.15.98:0/1060891290 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7ff390045bc0 con 0x5566a2b10f40 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff39f451740 1 -- 172.21.15.98:0/1060891290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff384005a70 con 0x5566a2b10f40 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff37dffb640 1 -- 172.21.15.98:0/1060891290 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff390035040 con 0x5566a2b10f40 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff39f451740 10 client.15609 fetch_fsmap learned FSMap version 29 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff39f451740 1 -- 172.21.15.98:0/1060891290 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5566a2b27550 con 0x5566a2b10f40 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff37dffb640 1 -- 172.21.15.98:0/1060891290 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff384005a70 con 0x5566a2b10f40 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff37dffb640 1 -- 172.21.15.98:0/1060891290 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7ff390045770 con 0x5566a2b10f40 2024-04-25T17:38:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff39f451740 10 client.15609 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:38:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff39f451740 10 client.15609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:38:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff39f451740 20 client.15609 populate_metadata read hostname 'smithi098' 2024-04-25T17:38:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff39f451740 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5566a2b27d10 0x5566a2b28120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff39f451740 4 client.15609 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:38:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff39f451740 1 -- 172.21.15.98:0/1060891290 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5566a29f95f0 con 0x5566a2b27d10 2024-04-25T17:38:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.578+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5566a2b27d10 0x5566a2b28120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.579+0000 7ff399cd1640 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5566a2b27d10 0x5566a2b28120 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f33a117785bcfeab server_cookie=80645a32772bf7b1 in_seq=0 out_seq=0 2024-04-25T17:38:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.580+0000 7ff37dffb640 10 client.15609 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 110.0 s, 3469.8 tps, lat 28.805 ms stddev 29.701 2024-04-25T17:38:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff37dffb640 1 -- 172.21.15.98:0/1060891290 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5566a29f95f0 con 0x5566a2b27d10 2024-04-25T17:38:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff37dffb640 10 client.15609 handle_command_reply: tid=0 2024-04-25T17:38:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 client.15609 shutdown 2024-04-25T17:38:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 -- 172.21.15.98:0/1060891290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 msgr2=0x5566a2b202f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b202f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff39002a920 tx=0x7ff39002a400 comp rx=0 tx=0).stop 2024-04-25T17:38:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 -- 172.21.15.98:0/1060891290 shutdown_connections 2024-04-25T17:38:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5566a2b27d10 0x5566a2b28120 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff37dffb640 0 client.15609 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5566a2b1b800 0x5566a2b25c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5566a2b10f40 0x5566a2b202f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 --2- 172.21.15.98:0/1060891290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5566a2b10620 0x5566a2b1fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 -- 172.21.15.98:0/1060891290 >> 172.21.15.98:0/1060891290 conn(0x5566a2b0cb30 msgr2=0x5566a2b0b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 -- 172.21.15.98:0/1060891290 shutdown_connections 2024-04-25T17:38:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.600+0000 7ff39f451740 1 -- 172.21.15.98:0/1060891290 wait complete. 2024-04-25T17:38:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39f451740 20 client.15609 trim_cache size 0 max 16384 2024-04-25T17:38:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:38:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:38:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:38:21.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:38:21.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 -- 172.21.15.98:0/2140923610 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff3800a3e40 msgr2=0x7ff3800a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:21.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff3800a3e40 0x7ff3800a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff38c0ce100 tx=0x7ff38c01eb40 comp rx=0 tx=0).stop 2024-04-25T17:38:21.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 -- 172.21.15.98:0/2140923610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff39419cff0 msgr2=0x7ff39419d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:21.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff39419cff0 0x7ff39419d4a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00eaa0 tx=0x7ff38c00ef60 comp rx=0 tx=0).stop 2024-04-25T17:38:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 -- 172.21.15.98:0/2140923610 shutdown_connections 2024-04-25T17:38:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff3800a3e40 0x7ff3800a6300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff39419cff0 0x7ff39419d4a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3941061d0 0x7ff394198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 --2- 172.21.15.98:0/2140923610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff394101330 0x7ff394198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 -- 172.21.15.98:0/2140923610 >> 172.21.15.98:0/2140923610 conn(0x7ff394077d30 msgr2=0x7ff3941008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 -- 172.21.15.98:0/2140923610 shutdown_connections 2024-04-25T17:38:21.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:21.601+0000 7ff39c736640 1 -- 172.21.15.98:0/2140923610 wait complete. 2024-04-25T17:38:22.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:22 smithi203 ceph-mon[32834]: pgmap v76: 97 pgs: 97 active+clean; 1.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.6 MiB/s wr, 737 op/s 2024-04-25T17:38:22.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:22 smithi173 ceph-mon[32856]: pgmap v76: 97 pgs: 97 active+clean; 1.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.6 MiB/s wr, 737 op/s 2024-04-25T17:38:22.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:22 smithi098 ceph-mon[29192]: pgmap v76: 97 pgs: 97 active+clean; 1.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.6 MiB/s wr, 737 op/s 2024-04-25T17:38:24.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:24 smithi203 ceph-mon[32834]: pgmap v77: 97 pgs: 97 active+clean; 1.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.7 MiB/s wr, 636 op/s 2024-04-25T17:38:24.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:24 smithi173 ceph-mon[32856]: pgmap v77: 97 pgs: 97 active+clean; 1.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.7 MiB/s wr, 636 op/s 2024-04-25T17:38:24.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:24 smithi098 ceph-mon[29192]: pgmap v77: 97 pgs: 97 active+clean; 1.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.7 MiB/s wr, 636 op/s 2024-04-25T17:38:26.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:26 smithi203 ceph-mon[32834]: pgmap v78: 97 pgs: 97 active+clean; 1.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.7 MiB/s wr, 636 op/s 2024-04-25T17:38:26.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:26 smithi173 ceph-mon[32856]: pgmap v78: 97 pgs: 97 active+clean; 1.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.7 MiB/s wr, 636 op/s 2024-04-25T17:38:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 115.0 s, 3683.4 tps, lat 27.115 ms stddev 27.129 2024-04-25T17:38:26.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:26 smithi098 ceph-mon[29192]: pgmap v78: 97 pgs: 97 active+clean; 1.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.7 MiB/s wr, 636 op/s 2024-04-25T17:38:28.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:28 smithi173 ceph-mon[32856]: pgmap v79: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 MiB/s wr, 966 op/s 2024-04-25T17:38:28.583 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:28 smithi098 ceph-mon[29192]: pgmap v79: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 MiB/s wr, 966 op/s 2024-04-25T17:38:28.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:28 smithi203 ceph-mon[32834]: pgmap v79: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 MiB/s wr, 966 op/s 2024-04-25T17:38:30.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:30 smithi173 ceph-mon[32856]: pgmap v80: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 641 op/s 2024-04-25T17:38:30.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:30 smithi098 ceph-mon[29192]: pgmap v80: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 641 op/s 2024-04-25T17:38:30.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:30 smithi203 ceph-mon[32834]: pgmap v80: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 641 op/s 2024-04-25T17:38:31.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 120.0 s, 3683.4 tps, lat 27.187 ms stddev 28.305 2024-04-25T17:38:32.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:32 smithi173 ceph-mon[32856]: pgmap v81: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 12 MiB/s wr, 960 op/s 2024-04-25T17:38:32.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:32 smithi098 ceph-mon[29192]: pgmap v81: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 12 MiB/s wr, 960 op/s 2024-04-25T17:38:32.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:32 smithi203 ceph-mon[32834]: pgmap v81: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 12 MiB/s wr, 960 op/s 2024-04-25T17:38:34.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:34 smithi173 ceph-mon[32856]: pgmap v82: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.6 MiB/s wr, 649 op/s 2024-04-25T17:38:34.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:34 smithi098 ceph-mon[29192]: pgmap v82: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.6 MiB/s wr, 649 op/s 2024-04-25T17:38:34.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:34 smithi203 ceph-mon[32834]: pgmap v82: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.6 MiB/s wr, 649 op/s 2024-04-25T17:38:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.723+0000 7f317ea6b640 1 -- 172.21.15.98:0/2361830657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3178108f40 msgr2=0x7f3178115400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.723+0000 7f317ea6b640 1 --2- 172.21.15.98:0/2361830657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3178108f40 0x7f3178115400 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3168009900 tx=0x7f316802a410 comp rx=0 tx=0).stop 2024-04-25T17:38:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.723+0000 7f317ea6b640 1 -- 172.21.15.98:0/2361830657 shutdown_connections 2024-04-25T17:38:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.723+0000 7f317ea6b640 1 --2- 172.21.15.98:0/2361830657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3178108f40 0x7f3178115400 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.723+0000 7f317ea6b640 1 --2- 172.21.15.98:0/2361830657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3178108580 0x7f3178108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.724+0000 7f317ea6b640 1 --2- 172.21.15.98:0/2361830657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f317810e640 0x7f317810ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.724+0000 7f317ea6b640 1 -- 172.21.15.98:0/2361830657 >> 172.21.15.98:0/2361830657 conn(0x7f31780764d0 msgr2=0x7f31780788f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.724+0000 7f317ea6b640 1 -- 172.21.15.98:0/2361830657 shutdown_connections 2024-04-25T17:38:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.724+0000 7f317ea6b640 1 -- 172.21.15.98:0/2361830657 wait complete. 2024-04-25T17:38:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.725+0000 7f317ea6b640 1 Processor -- start 2024-04-25T17:38:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.725+0000 7f317ea6b640 1 -- start start 2024-04-25T17:38:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.725+0000 7f317ea6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3178108580 0x7f317807c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.726 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317ea6b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3178108f40 0x7f317807ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.726 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317ea6b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f317810e640 0x7f3178080d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317c807640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3178108580 0x7f317807c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317c807640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3178108580 0x7f317807c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52802/0 (socket says 172.21.15.98:52802) 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317c807640 1 -- 172.21.15.98:0/2664884333 learned_addr learned my addr 172.21.15.98:0/2664884333 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f317807adb0 con 0x7f3178108580 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f317807ac30 con 0x7f317810e640 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f317807af30 con 0x7f3178108f40 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317d008640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f317810e640 0x7f3178080d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f3177fff640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3178108f40 0x7f317807ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317c807640 1 -- 172.21.15.98:0/2664884333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3178108f40 msgr2=0x7f317807ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317c807640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3178108f40 0x7f317807ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317c807640 1 -- 172.21.15.98:0/2664884333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f317810e640 msgr2=0x7f3178080d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317c807640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f317810e640 0x7f3178080d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f317c807640 1 -- 172.21.15.98:0/2664884333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31780814c0 con 0x7f3178108580 2024-04-25T17:38:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.726+0000 7f3177fff640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3178108f40 0x7f317807ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:38:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.727+0000 7f317c807640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3178108580 0x7f317807c500 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f316000d770 tx=0x7f316000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.727+0000 7f3175ffb640 1 -- 172.21.15.98:0/2664884333 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160017070 con 0x7f3178108580 2024-04-25T17:38:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.727+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31780817b0 con 0x7f3178108580 2024-04-25T17:38:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.727+0000 7f3175ffb640 1 -- 172.21.15.98:0/2664884333 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31600054d0 con 0x7f3178108580 2024-04-25T17:38:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.727+0000 7f3175ffb640 1 -- 172.21.15.98:0/2664884333 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160004c30 con 0x7f3178108580 2024-04-25T17:38:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.728+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31781b3fd0 con 0x7f3178108580 2024-04-25T17:38:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.731+0000 7f3175ffb640 1 -- 172.21.15.98:0/2664884333 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3160010460 con 0x7f3178108580 2024-04-25T17:38:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.731+0000 7f318177f740 1 Processor -- start 2024-04-25T17:38:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.731+0000 7f318177f740 1 -- start start 2024-04-25T17:38:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f318177f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f318177f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8c7e00 0x559ece8e32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f318177f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8e3810 0x559ece8e5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f318177f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559ece82c970 con 0x559ece8e3810 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f318177f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559ece7c94a0 con 0x559ece8c7e00 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f318177f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559ece828930 con 0x559ece8c74e0 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f3177fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8e3810 0x559ece8e5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f3177fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8e3810 0x559ece8e5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52818/0 (socket says 172.21.15.98:52818) 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f3177fff640 1 -- 172.21.15.98:0/720343659 learned_addr learned my addr 172.21.15.98:0/720343659 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f317d008640 1 --2- 172.21.15.98:0/720343659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.732+0000 7f317c807640 1 --2- 172.21.15.98:0/720343659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8c7e00 0x559ece8e32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f3177fff640 1 -- 172.21.15.98:0/720343659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 msgr2=0x559ece8c78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f3177fff640 1 --2- 172.21.15.98:0/720343659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8c78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f3177fff640 1 -- 172.21.15.98:0/720343659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8c7e00 msgr2=0x559ece8e32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f3177fff640 1 --2- 172.21.15.98:0/720343659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8c7e00 0x559ece8e32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f3177fff640 1 -- 172.21.15.98:0/720343659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ece53f210 con 0x559ece8e3810 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f3177fff640 1 --2- 172.21.15.98:0/720343659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8e3810 0x559ece8e5c00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3170009fd0 tx=0x7f31700328f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9784a7f9ae4fe33b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f316effd640 1 -- 172.21.15.98:0/720343659 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3170033770 con 0x559ece8e3810 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f3175ffb640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f31640a3e40 0x7f31640a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f316effd640 1 -- 172.21.15.98:0/720343659 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3170033dc0 con 0x559ece8e3810 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f316effd640 1 -- 172.21.15.98:0/720343659 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31700303d0 con 0x559ece8e3810 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f317d008640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f31640a3e40 0x7f31640a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.733+0000 7f3175ffb640 1 -- 172.21.15.98:0/2664884333 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f31600d0110 con 0x7f3178108580 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 -- 172.21.15.98:0/720343659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8e3810 msgr2=0x559ece8e5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f317d008640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f31640a3e40 0x7f31640a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3168005ec0 tx=0x7f31680046c0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 --2- 172.21.15.98:0/720343659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8e3810 0x559ece8e5c00 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3170009fd0 tx=0x7f31700328f0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 -- 172.21.15.98:0/720343659 shutdown_connections 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 --2- 172.21.15.98:0/720343659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8e3810 0x559ece8e5c00 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 --2- 172.21.15.98:0/720343659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8c7e00 0x559ece8e32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 --2- 172.21.15.98:0/720343659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8c78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 -- 172.21.15.98:0/720343659 >> 172.21.15.98:0/720343659 conn(0x559ece8c3b30 msgr2=0x559ece8c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 -- 172.21.15.98:0/720343659 shutdown_connections 2024-04-25T17:38:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 -- 172.21.15.98:0/720343659 wait complete. 2024-04-25T17:38:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 -- start start 2024-04-25T17:38:34.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.734+0000 7f318177f740 1 Processor -- start 2024-04-25T17:38:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f318177f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8d4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f318177f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f318177f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f318177f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559ece8ca4a0 con 0x559ece8c7e00 2024-04-25T17:38:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f318177f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559ece7c70b0 con 0x559ece8e3810 2024-04-25T17:38:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f318177f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559ece8ca620 con 0x559ece8c74e0 2024-04-25T17:38:34.741 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f318177f740 10 client.0 _ll_register_callbacks cb 0x559ece7b4d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3177fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3177fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52824/0 (socket says 172.21.15.98:52824) 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3177fff640 1 -- 172.21.15.98:0/2230448476 learned_addr learned my addr 172.21.15.98:0/2230448476 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f317d008640 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f317c807640 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8d4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3177fff640 1 -- 172.21.15.98:0/2230448476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 msgr2=0x559ece8d4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3177fff640 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8d4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3177fff640 1 -- 172.21.15.98:0/2230448476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 msgr2=0x559ece8cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3177fff640 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3177fff640 1 -- 172.21.15.98:0/2230448476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ece60b620 con 0x559ece8c7e00 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3177fff640 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d4c70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3170009600 tx=0x7f3170004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61cf3efb53245220 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3157fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.735+0000 7f3157fff640 1 -- 172.21.15.98:0/2230448476 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317004d060 con 0x559ece8c7e00 2024-04-25T17:38:34.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.736+0000 7f318177f740 1 -- 172.21.15.98:0/2230448476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3170033dc0 con 0x559ece8c7e00 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.736+0000 7f3157fff640 1 -- 172.21.15.98:0/2230448476 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f3170033780 con 0x559ece8c7e00 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.736+0000 7f3157fff640 1 -- 172.21.15.98:0/2230448476 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3170033980 con 0x559ece8c7e00 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.736+0000 7f318177f740 10 client.15621 fetch_fsmap learned FSMap version 29 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.736+0000 7f318177f740 1 -- 172.21.15.98:0/2230448476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559ece8de9d0 con 0x559ece8c7e00 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.736+0000 7f3157fff640 1 -- 172.21.15.98:0/2230448476 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3170033dc0 con 0x559ece8c7e00 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.736+0000 7f3157fff640 1 -- 172.21.15.98:0/2230448476 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f3170005720 con 0x559ece8c7e00 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.737+0000 7f318177f740 10 client.15621 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.737+0000 7f318177f740 10 client.15621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.737+0000 7f318177f740 20 client.15621 populate_metadata read hostname 'smithi098' 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.737+0000 7f318177f740 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559ece8dfd80 0x559ece8e21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.737+0000 7f318177f740 4 client.15621 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.737+0000 7f318177f740 1 -- 172.21.15.98:0/2230448476 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ece83a390 con 0x559ece8dfd80 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.737+0000 7f317c807640 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559ece8dfd80 0x559ece8e21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.739+0000 7f3157fff640 10 client.15621 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.739+0000 7f317c807640 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559ece8dfd80 0x559ece8e21a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29e01333c49995a3 server_cookie=eeee69d491161c25 in_seq=0 out_seq=0 2024-04-25T17:38:34.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f3157fff640 1 -- 172.21.15.98:0/2230448476 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ece83a390 con 0x559ece8dfd80 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f3157fff640 10 client.15621 handle_command_reply: tid=0 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 client.15621 shutdown 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 -- 172.21.15.98:0/2230448476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 msgr2=0x559ece8d4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d4c70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3170009600 tx=0x7f3170004c70 comp rx=0 tx=0).stop 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 -- 172.21.15.98:0/2230448476 shutdown_connections 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559ece8dfd80 0x559ece8e21a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d4c70 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f3157fff640 0 client.15621 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 --2- 172.21.15.98:0/2230448476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8d4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 -- 172.21.15.98:0/2230448476 >> 172.21.15.98:0/2230448476 conn(0x559ece8c3b30 msgr2=0x559ece8c98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 -- 172.21.15.98:0/2230448476 shutdown_connections 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 1 -- 172.21.15.98:0/2230448476 wait complete. 2024-04-25T17:38:34.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.740+0000 7f318177f740 20 client.15621 trim_cache size 0 max 16384 2024-04-25T17:38:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f318177f740 1 Processor -- start 2024-04-25T17:38:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f318177f740 1 -- start start 2024-04-25T17:38:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f318177f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c74e0 0x559ece8e2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f318177f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c7e00 0x559ece8eb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f318177f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8eb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f318177f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3158011010 con 0x559ece8c74e0 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f318177f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f315800e8b0 con 0x559ece8e3810 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f318177f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f315800c700 con 0x559ece8c7e00 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f317c807640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c74e0 0x559ece8e2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f317c807640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c74e0 0x559ece8e2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52834/0 (socket says 172.21.15.98:52834) 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f317c807640 1 -- 172.21.15.98:0/28606650 learned_addr learned my addr 172.21.15.98:0/28606650 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f317d008640 1 --2- 172.21.15.98:0/28606650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f3177fff640 1 --2- 172.21.15.98:0/28606650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c7e00 0x559ece8eb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f317c807640 1 -- 172.21.15.98:0/28606650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c7e00 msgr2=0x559ece8eb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f317c807640 1 --2- 172.21.15.98:0/28606650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c7e00 0x559ece8eb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f317c807640 1 -- 172.21.15.98:0/28606650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 msgr2=0x559ece8eb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.766+0000 7f317c807640 1 --2- 172.21.15.98:0/28606650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8eb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.767+0000 7f317c807640 1 -- 172.21.15.98:0/28606650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ece8d0fa0 con 0x559ece8c74e0 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.767+0000 7f317d008640 1 --2- 172.21.15.98:0/28606650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8eb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.767+0000 7f317c807640 1 --2- 172.21.15.98:0/28606650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c74e0 0x559ece8e2f30 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f316000df50 tx=0x7f31600d2560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1cff845722c6a3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.767+0000 7f316e7fc640 1 -- 172.21.15.98:0/28606650 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160020040 con 0x559ece8c74e0 2024-04-25T17:38:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.767+0000 7f316e7fc640 1 -- 172.21.15.98:0/28606650 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f316009c080 con 0x559ece8c74e0 2024-04-25T17:38:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.767+0000 7f316e7fc640 1 -- 172.21.15.98:0/28606650 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31600986f0 con 0x559ece8c74e0 2024-04-25T17:38:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.768+0000 7f318177f740 1 -- 172.21.15.98:0/28606650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c74e0 msgr2=0x559ece8e2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:34.769 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.768+0000 7f318177f740 1 --2- 172.21.15.98:0/28606650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c74e0 0x559ece8e2f30 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f316000df50 tx=0x7f31600d2560 comp rx=0 tx=0).stop 2024-04-25T17:38:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.768+0000 7f318177f740 1 -- 172.21.15.98:0/28606650 shutdown_connections 2024-04-25T17:38:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.768+0000 7f318177f740 1 --2- 172.21.15.98:0/28606650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8eb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.768+0000 7f318177f740 1 --2- 172.21.15.98:0/28606650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c7e00 0x559ece8eb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.768+0000 7f318177f740 1 --2- 172.21.15.98:0/28606650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c74e0 0x559ece8e2f30 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.768+0000 7f318177f740 1 -- 172.21.15.98:0/28606650 >> 172.21.15.98:0/28606650 conn(0x559ece8c3b30 msgr2=0x559ece8e2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.768+0000 7f318177f740 1 -- 172.21.15.98:0/28606650 shutdown_connections 2024-04-25T17:38:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.768+0000 7f318177f740 1 -- 172.21.15.98:0/28606650 wait complete. 2024-04-25T17:38:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.769+0000 7f318177f740 1 -- start start 2024-04-25T17:38:34.770 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.769+0000 7f318177f740 1 Processor -- start 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.770+0000 7f318177f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.770+0000 7f318177f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.770+0000 7f317c807640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.770+0000 7f317c807640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:56166/0 (socket says 172.21.15.98:56166) 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.770+0000 7f3177fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.770+0000 7f3177fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52836/0 (socket says 172.21.15.98:52836) 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.770+0000 7f318177f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8d78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f317c807640 1 -- 172.21.15.98:0/2074775747 learned_addr learned my addr 172.21.15.98:0/2074775747 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f317d008640 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8d78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f315800cd70 con 0x559ece8c7e00 2024-04-25T17:38:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f315800e1e0 con 0x559ece8e3810 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f315800d440 con 0x559ece8c74e0 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f318177f740 10 client.0 _ll_register_callbacks cb 0x7f317003d7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f317d008640 1 -- 172.21.15.98:0/2074775747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 msgr2=0x559ece8cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f317d008640 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f317d008640 1 -- 172.21.15.98:0/2074775747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 msgr2=0x559ece8d7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f317d008640 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.771+0000 7f317d008640 1 -- 172.21.15.98:0/2074775747 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ece8d80e0 con 0x559ece8e3810 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.772+0000 7f317d008640 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8d78b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f316802a930 tx=0x7f3168009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e22b9a2fb0872ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.772+0000 7f3157fff640 10 client.25214 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.772+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f316009c080 con 0x559ece8e3810 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.772+0000 7f3157fff640 1 -- 172.21.15.98:0/2074775747 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168035040 con 0x559ece8e3810 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.772+0000 7f3157fff640 1 -- 172.21.15.98:0/2074775747 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f316803d660 con 0x559ece8e3810 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.772+0000 7f3157fff640 1 -- 172.21.15.98:0/2074775747 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3168003e70 con 0x559ece8e3810 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.772+0000 7f318177f740 10 client.25214 fetch_fsmap learned FSMap version 29 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.772+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=29}) -- 0x559ece9489a0 con 0x559ece8e3810 2024-04-25T17:38:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.773+0000 7f3157fff640 1 -- 172.21.15.98:0/2074775747 <== mon.2 v2:172.21.15.173:3300/0 4 ==== fsmap(e 29) ==== 8350+0+0 (secure 0 0 0) 0x7f316803d860 con 0x559ece8e3810 2024-04-25T17:38:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.773+0000 7f318177f740 10 client.25214 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T17:38:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.773+0000 7f318177f740 10 client.25214 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:38:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.773+0000 7f318177f740 20 client.25214 populate_metadata read hostname 'smithi098' 2024-04-25T17:38:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.773+0000 7f318177f740 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559ece949050 0x559ece94b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.773+0000 7f318177f740 4 client.25214 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:38:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.773+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559ece834200 con 0x559ece949050 2024-04-25T17:38:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.773+0000 7f317c807640 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559ece949050 0x559ece94b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.773+0000 7f3157fff640 10 client.25214 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.773+0000 7f317c807640 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559ece949050 0x559ece94b470 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=937345fb1c240e58 server_cookie=d695dca8b73fb074 in_seq=0 out_seq=0 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.793+0000 7f3157fff640 1 -- 172.21.15.98:0/2074775747 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559ece834200 con 0x559ece949050 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.793+0000 7f3157fff640 10 client.25214 handle_command_reply: tid=0 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.793+0000 7f318177f740 1 client.25214 shutdown 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.793+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 msgr2=0x559ece8d78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.793+0000 7f318177f740 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8d78b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f316802a930 tx=0x7f3168009260 comp rx=0 tx=0).stop 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.794+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 shutdown_connections 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.794+0000 7f318177f740 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559ece949050 0x559ece94b470 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.794+0000 7f3157fff640 0 client.25214 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.794+0000 7f318177f740 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559ece8e3810 0x559ece8d78b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.794+0000 7f318177f740 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559ece8c7e00 0x559ece8d7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.794+0000 7f318177f740 1 --2- 172.21.15.98:0/2074775747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559ece8c74e0 0x559ece8cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.794+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 >> 172.21.15.98:0/2074775747 conn(0x559ece8c3b30 msgr2=0x559ece8c2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:34.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.794+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 shutdown_connections 2024-04-25T17:38:34.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.794+0000 7f318177f740 1 -- 172.21.15.98:0/2074775747 wait complete. 2024-04-25T17:38:34.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.795+0000 7f318177f740 20 client.25214 trim_cache size 0 max 16384 2024-04-25T17:38:34.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:38:34.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:38:34.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:38:34.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:38:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.795+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f31640a3e40 msgr2=0x7f31640a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f31640a3e40 0x7f31640a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3168005ec0 tx=0x7f31680046c0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3178108580 msgr2=0x7f317807c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3178108580 0x7f317807c500 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f316000d770 tx=0x7f316000dc30 comp rx=0 tx=0).stop 2024-04-25T17:38:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 shutdown_connections 2024-04-25T17:38:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f31640a3e40 0x7f31640a6300 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f317810e640 0x7f3178080d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3178108f40 0x7f317807ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 --2- 172.21.15.98:0/2664884333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3178108580 0x7f317807c500 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 >> 172.21.15.98:0/2664884333 conn(0x7f31780764d0 msgr2=0x7f3178077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 shutdown_connections 2024-04-25T17:38:34.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:34.796+0000 7f317ea6b640 1 -- 172.21.15.98:0/2664884333 wait complete. 2024-04-25T17:38:36.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:36 smithi173 ceph-mon[32856]: pgmap v83: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.6 MiB/s wr, 649 op/s 2024-04-25T17:38:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 125.0 s, 3772.8 tps, lat 26.565 ms stddev 27.394 2024-04-25T17:38:36.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:36 smithi098 ceph-mon[29192]: pgmap v83: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.6 MiB/s wr, 649 op/s 2024-04-25T17:38:36.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:36 smithi203 ceph-mon[32834]: pgmap v83: 97 pgs: 97 active+clean; 1.2 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.6 MiB/s wr, 649 op/s 2024-04-25T17:38:38.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:38 smithi173 ceph-mon[32856]: pgmap v84: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 11 MiB/s wr, 975 op/s 2024-04-25T17:38:38.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:38 smithi098 ceph-mon[29192]: pgmap v84: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 11 MiB/s wr, 975 op/s 2024-04-25T17:38:38.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:38 smithi203 ceph-mon[32834]: pgmap v84: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 11 MiB/s wr, 975 op/s 2024-04-25T17:38:39.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:39 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:38:39.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:39 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:38:39.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:39 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:38:40.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:40 smithi173 ceph-mon[32856]: pgmap v85: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 7.6 MiB/s wr, 645 op/s 2024-04-25T17:38:40.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:40 smithi098 ceph-mon[29192]: pgmap v85: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 7.6 MiB/s wr, 645 op/s 2024-04-25T17:38:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:40 smithi203 ceph-mon[32834]: pgmap v85: 97 pgs: 97 active+clean; 1.3 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 7.6 MiB/s wr, 645 op/s 2024-04-25T17:38:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 130.0 s, 3934.6 tps, lat 25.345 ms stddev 26.098 2024-04-25T17:38:42.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:42 smithi173 ceph-mon[32856]: pgmap v86: 97 pgs: 97 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 15 MiB/s wr, 1.03k op/s 2024-04-25T17:38:42.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:42 smithi098 ceph-mon[29192]: pgmap v86: 97 pgs: 97 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 15 MiB/s wr, 1.03k op/s 2024-04-25T17:38:42.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:42 smithi203 ceph-mon[32834]: pgmap v86: 97 pgs: 97 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 15 MiB/s wr, 1.03k op/s 2024-04-25T17:38:44.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:44 smithi173 ceph-mon[32856]: pgmap v87: 97 pgs: 97 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 714 op/s 2024-04-25T17:38:44.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:44 smithi098 ceph-mon[29192]: pgmap v87: 97 pgs: 97 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 714 op/s 2024-04-25T17:38:44.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:44 smithi203 ceph-mon[32834]: pgmap v87: 97 pgs: 97 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 714 op/s 2024-04-25T17:38:46.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:46 smithi098 ceph-mon[29192]: pgmap v88: 97 pgs: 97 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 714 op/s 2024-04-25T17:38:46.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:46 smithi173 ceph-mon[32856]: pgmap v88: 97 pgs: 97 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 714 op/s 2024-04-25T17:38:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 135.0 s, 3562.2 tps, lat 28.064 ms stddev 30.516 2024-04-25T17:38:46.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:46 smithi203 ceph-mon[32834]: pgmap v88: 97 pgs: 97 active+clean; 1.3 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 10 MiB/s wr, 714 op/s 2024-04-25T17:38:47.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:47 smithi098 ceph-mon[29192]: pgmap v89: 97 pgs: 97 active+clean; 1.3 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 1.04k op/s 2024-04-25T17:38:47.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:47 smithi203 ceph-mon[32834]: pgmap v89: 97 pgs: 97 active+clean; 1.3 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 1.04k op/s 2024-04-25T17:38:47.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:47 smithi173 ceph-mon[32856]: pgmap v89: 97 pgs: 97 active+clean; 1.3 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 1.04k op/s 2024-04-25T17:38:47.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.918+0000 7f85d59e2640 1 -- 172.21.15.98:0/1938859685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d0116fe0 msgr2=0x7f85d01193d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:47.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.918+0000 7f85d59e2640 1 --2- 172.21.15.98:0/1938859685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d0116fe0 0x7f85d01193d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009ca0 tx=0x7f85c802a8c0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.918+0000 7f85d59e2640 1 -- 172.21.15.98:0/1938859685 shutdown_connections 2024-04-25T17:38:47.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.918+0000 7f85d59e2640 1 --2- 172.21.15.98:0/1938859685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d0116fe0 0x7f85d01193d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.918+0000 7f85d59e2640 1 --2- 172.21.15.98:0/1938859685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85d0109520 0x7f85d0109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.918+0000 7f85d59e2640 1 --2- 172.21.15.98:0/1938859685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d010c6a0 0x7f85d0108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.918+0000 7f85d59e2640 1 -- 172.21.15.98:0/1938859685 >> 172.21.15.98:0/1938859685 conn(0x7f85d006d4b0 msgr2=0x7f85d006d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:47.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.919+0000 7f85d59e2640 1 -- 172.21.15.98:0/1938859685 shutdown_connections 2024-04-25T17:38:47.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.919+0000 7f85d59e2640 1 -- 172.21.15.98:0/1938859685 wait complete. 2024-04-25T17:38:47.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.919+0000 7f85d59e2640 1 Processor -- start 2024-04-25T17:38:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.919+0000 7f85d59e2640 1 -- start start 2024-04-25T17:38:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85d59e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0109520 0x7f85d010cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85d59e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d010c6a0 0x7f85d010d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85d59e2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85d0116fe0 0x7f85d010d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85d59e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85d011cbc0 con 0x7f85d0109520 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85d59e2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f85d011ca40 con 0x7f85d0116fe0 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85d59e2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85d011cd40 con 0x7f85d010c6a0 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85ce7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d010c6a0 0x7f85d010d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85ce7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d010c6a0 0x7f85d010d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:49714/0 (socket says 172.21.15.98:49714) 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85ce7fc640 1 -- 172.21.15.98:0/1425343077 learned_addr learned my addr 172.21.15.98:0/1425343077 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85d0116fe0 0x7f85d010d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.920+0000 7f85ceffd640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0109520 0x7f85d010cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85ce7fc640 1 -- 172.21.15.98:0/1425343077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85d0116fe0 msgr2=0x7f85d010d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85ce7fc640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85d0116fe0 0x7f85d010d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85ce7fc640 1 -- 172.21.15.98:0/1425343077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0109520 msgr2=0x7f85d010cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85ce7fc640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0109520 0x7f85d010cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85ce7fc640 1 -- 172.21.15.98:0/1425343077 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d01b3730 con 0x7f85d010c6a0 2024-04-25T17:38:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85ce7fc640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d010c6a0 0x7f85d010d110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f85c000e9f0 tx=0x7f85c000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85d49e0640 1 -- 172.21.15.98:0/1425343077 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c000cd80 con 0x7f85d010c6a0 2024-04-25T17:38:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85d49e0640 1 -- 172.21.15.98:0/1425343077 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85c00108e0 con 0x7f85d010c6a0 2024-04-25T17:38:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85d49e0640 1 -- 172.21.15.98:0/1425343077 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c00188b0 con 0x7f85d010c6a0 2024-04-25T17:38:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85d59e2640 1 -- 172.21.15.98:0/1425343077 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85d01b3a20 con 0x7f85d010c6a0 2024-04-25T17:38:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.921+0000 7f85d59e2640 1 -- 172.21.15.98:0/1425343077 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85d01b3fc0 con 0x7f85d010c6a0 2024-04-25T17:38:47.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.923+0000 7f85d49e0640 1 -- 172.21.15.98:0/1425343077 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f85c0010430 con 0x7f85d010c6a0 2024-04-25T17:38:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85d87d4740 1 Processor -- start 2024-04-25T17:38:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85d87d4740 1 -- start start 2024-04-25T17:38:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85d49e0640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85b80a3e40 0x7f85b80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239620 0x55fda4239a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239f40 0x55fda42442c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 0x55fda4246bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85d87d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fda419e970 con 0x55fda4239620 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85d87d4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fda413b4a0 con 0x55fda4239f40 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85d87d4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fda419a930 con 0x55fda4244800 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85b80a3e40 0x7f85b80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85d49e0640 1 -- 172.21.15.98:0/1425343077 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f85c00d0ac0 con 0x7f85d010c6a0 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85ceffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239620 0x55fda4239a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85ceffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239620 0x55fda4239a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33706/0 (socket says 172.21.15.98:33706) 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85ceffd640 1 -- 172.21.15.98:0/1978403805 learned_addr learned my addr 172.21.15.98:0/1978403805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/1978403805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 0x55fda4246bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.924+0000 7f85ce7fc640 1 --2- 172.21.15.98:0/1978403805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239f40 0x55fda42442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.925+0000 7f85ceffd640 1 -- 172.21.15.98:0/1978403805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 msgr2=0x55fda4246bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.925+0000 7f85ceffd640 1 --2- 172.21.15.98:0/1978403805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 0x55fda4246bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.925+0000 7f85ceffd640 1 -- 172.21.15.98:0/1978403805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239f40 msgr2=0x55fda42442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.925+0000 7f85ceffd640 1 --2- 172.21.15.98:0/1978403805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239f40 0x55fda42442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.925+0000 7f85ceffd640 1 -- 172.21.15.98:0/1978403805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda3eb1210 con 0x55fda4239620 2024-04-25T17:38:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.925+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85b80a3e40 0x7f85b80a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f85c8006010 tx=0x7f85c8031bd0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.925+0000 7f85ceffd640 1 --2- 172.21.15.98:0/1978403805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239620 0x55fda4239a00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f85c400bc50 tx=0x7f85c40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f228faacf83c2ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.925+0000 7f85b5ffb640 1 -- 172.21.15.98:0/1978403805 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c401c480 con 0x55fda4239620 2024-04-25T17:38:47.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.926+0000 7f85b5ffb640 1 -- 172.21.15.98:0/1978403805 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85c401cad0 con 0x55fda4239620 2024-04-25T17:38:47.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.926+0000 7f85b5ffb640 1 -- 172.21.15.98:0/1978403805 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4011b60 con 0x55fda4239620 2024-04-25T17:38:47.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.927+0000 7f85d87d4740 1 -- 172.21.15.98:0/1978403805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239620 msgr2=0x55fda4239a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:47.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.927+0000 7f85d87d4740 1 --2- 172.21.15.98:0/1978403805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239620 0x55fda4239a00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f85c400bc50 tx=0x7f85c40107a0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.927+0000 7f85d87d4740 1 -- 172.21.15.98:0/1978403805 shutdown_connections 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.927+0000 7f85d87d4740 1 --2- 172.21.15.98:0/1978403805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 0x55fda4246bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.927+0000 7f85d87d4740 1 --2- 172.21.15.98:0/1978403805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239f40 0x55fda42442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.927+0000 7f85d87d4740 1 --2- 172.21.15.98:0/1978403805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239620 0x55fda4239a00 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.927+0000 7f85d87d4740 1 -- 172.21.15.98:0/1978403805 >> 172.21.15.98:0/1978403805 conn(0x55fda4235b30 msgr2=0x55fda4235f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.927+0000 7f85d87d4740 1 -- 172.21.15.98:0/1978403805 shutdown_connections 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.927+0000 7f85d87d4740 1 -- 172.21.15.98:0/1978403805 wait complete. 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.927+0000 7f85d87d4740 1 -- start start 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85d87d4740 1 Processor -- start 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda4244180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 0x55fda4241470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 0x55fda42419f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85d87d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fda4247430 con 0x55fda4244800 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85d87d4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fda41390b0 con 0x55fda4239620 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85d87d4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fda42475b0 con 0x55fda4239f40 2024-04-25T17:38:47.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85d87d4740 10 client.0 _ll_register_callbacks cb 0x55fda4126d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 0x55fda42419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 0x55fda42419f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33718/0 (socket says 172.21.15.98:33718) 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85cf7fe640 1 -- 172.21.15.98:0/423582884 learned_addr learned my addr 172.21.15.98:0/423582884 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85ce7fc640 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 0x55fda4241470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85ceffd640 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda4244180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85cf7fe640 1 -- 172.21.15.98:0/423582884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 msgr2=0x55fda4241470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 0x55fda4241470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85cf7fe640 1 -- 172.21.15.98:0/423582884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 msgr2=0x55fda4244180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda4244180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.928+0000 7f85cf7fe640 1 -- 172.21.15.98:0/423582884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda3f7d620 con 0x55fda4244800 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85ceffd640 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda4244180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 0x55fda42419f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f85c8035000 tx=0x7f85c802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf46c6d32262f2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:38:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85aeffd640 1 -- 172.21.15.98:0/423582884 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c8031e60 con 0x55fda4244800 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85d87d4740 1 -- 172.21.15.98:0/423582884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85c401cad0 con 0x55fda4244800 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85aeffd640 1 -- 172.21.15.98:0/423582884 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f85c8004540 con 0x55fda4244800 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85aeffd640 1 -- 172.21.15.98:0/423582884 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f85c8059050 con 0x55fda4244800 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85d87d4740 10 client.15639 fetch_fsmap learned FSMap version 30 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85d87d4740 1 -- 172.21.15.98:0/423582884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fda4253380 con 0x55fda4244800 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85aeffd640 1 -- 172.21.15.98:0/423582884 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85c401cad0 con 0x55fda4244800 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.929+0000 7f85aeffd640 1 -- 172.21.15.98:0/423582884 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f85c80305c0 con 0x55fda4244800 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.930+0000 7f85d87d4740 10 client.15639 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.930+0000 7f85d87d4740 10 client.15639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.930+0000 7f85d87d4740 20 client.15639 populate_metadata read hostname 'smithi098' 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.930+0000 7f85d87d4740 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fda4254900 0x55fda4256d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.930+0000 7f85d87d4740 4 client.15639 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.930+0000 7f85d87d4740 1 -- 172.21.15.98:0/423582884 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fda41ac390 con 0x55fda4254900 2024-04-25T17:38:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.930+0000 7f85ceffd640 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fda4254900 0x55fda4256d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.931+0000 7f85aeffd640 10 client.15639 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.931+0000 7f85ceffd640 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fda4254900 0x55fda4256d50 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5ad03778cccf52e server_cookie=f7d34900c1726da2 in_seq=0 out_seq=0 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85aeffd640 1 -- 172.21.15.98:0/423582884 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fda41ac390 con 0x55fda4254900 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85aeffd640 10 client.15639 handle_command_reply: tid=0 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 client.15639 shutdown 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 -- 172.21.15.98:0/423582884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 msgr2=0x55fda42419f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 0x55fda42419f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f85c8035000 tx=0x7f85c802a8c0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 -- 172.21.15.98:0/423582884 shutdown_connections 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fda4254900 0x55fda4256d50 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85aeffd640 0 client.15639 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 0x55fda42419f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 0x55fda4241470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 --2- 172.21.15.98:0/423582884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda4244180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 -- 172.21.15.98:0/423582884 >> 172.21.15.98:0/423582884 conn(0x55fda4235b30 msgr2=0x55fda423b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:47.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 -- 172.21.15.98:0/423582884 shutdown_connections 2024-04-25T17:38:47.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 1 -- 172.21.15.98:0/423582884 wait complete. 2024-04-25T17:38:47.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.932+0000 7f85d87d4740 20 client.15639 trim_cache size 0 max 16384 2024-04-25T17:38:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85d87d4740 1 Processor -- start 2024-04-25T17:38:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85d87d4740 1 -- start start 2024-04-25T17:38:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda42502f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239f40 0x55fda4234b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 0x55fda425c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85d87d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01bf270 con 0x55fda4239f40 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85d87d4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01bcae0 con 0x55fda4239620 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85d87d4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01ba930 con 0x55fda4244800 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85ceffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda42502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 0x55fda425c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85cf7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 0x55fda425c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:49746/0 (socket says 172.21.15.98:49746) 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85ceffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda42502f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57082/0 (socket says 172.21.15.98:57082) 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85cf7fe640 1 -- 172.21.15.98:0/1190093724 learned_addr learned my addr 172.21.15.98:0/1190093724 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.959+0000 7f85ce7fc640 1 --2- 172.21.15.98:0/1190093724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239f40 0x55fda4234b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.960+0000 7f85cf7fe640 1 -- 172.21.15.98:0/1190093724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 msgr2=0x55fda42502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.960+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/1190093724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda42502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.960+0000 7f85cf7fe640 1 -- 172.21.15.98:0/1190093724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239f40 msgr2=0x55fda4234b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.960+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/1190093724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239f40 0x55fda4234b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.960+0000 7f85cf7fe640 1 -- 172.21.15.98:0/1190093724 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda4231500 con 0x55fda4244800 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.960+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/1190093724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 0x55fda425c660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009ba0 tx=0x7f85c802f450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75216b00f56c31ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.960+0000 7f85b57fa640 1 -- 172.21.15.98:0/1190093724 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c80305c0 con 0x55fda4244800 2024-04-25T17:38:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.960+0000 7f85b57fa640 1 -- 172.21.15.98:0/1190093724 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85c8062070 con 0x55fda4244800 2024-04-25T17:38:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.960+0000 7f85b57fa640 1 -- 172.21.15.98:0/1190093724 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c802f660 con 0x55fda4244800 2024-04-25T17:38:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.961+0000 7f85d87d4740 1 -- 172.21.15.98:0/1190093724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 msgr2=0x55fda425c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.961+0000 7f85d87d4740 1 --2- 172.21.15.98:0/1190093724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 0x55fda425c660 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009ba0 tx=0x7f85c802f450 comp rx=0 tx=0).stop 2024-04-25T17:38:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.961+0000 7f85d87d4740 1 -- 172.21.15.98:0/1190093724 shutdown_connections 2024-04-25T17:38:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.961+0000 7f85d87d4740 1 --2- 172.21.15.98:0/1190093724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4244800 0x55fda425c660 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.961+0000 7f85d87d4740 1 --2- 172.21.15.98:0/1190093724 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4239f40 0x55fda4234b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.961+0000 7f85d87d4740 1 --2- 172.21.15.98:0/1190093724 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda42502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.961+0000 7f85d87d4740 1 -- 172.21.15.98:0/1190093724 >> 172.21.15.98:0/1190093724 conn(0x55fda4235b30 msgr2=0x55fda4234420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.961+0000 7f85d87d4740 1 -- 172.21.15.98:0/1190093724 shutdown_connections 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.961+0000 7f85d87d4740 1 -- 172.21.15.98:0/1190093724 wait complete. 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.961+0000 7f85d87d4740 1 -- start start 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.962+0000 7f85d87d4740 1 Processor -- start 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.962+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda424a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.962+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 0x55fda424a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.962+0000 7f85d87d4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 0x55fda424ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.962+0000 7f85d87d4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01bafa0 con 0x55fda4244800 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.962+0000 7f85d87d4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01bc410 con 0x55fda4239620 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.962+0000 7f85d87d4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85d01bb670 con 0x55fda4239f40 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.962+0000 7f85d87d4740 10 client.0 _ll_register_callbacks cb 0x7f85c8031780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85ceffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda424a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85ceffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda424a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57084/0 (socket says 172.21.15.98:57084) 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85ceffd640 1 -- 172.21.15.98:0/926053178 learned_addr learned my addr 172.21.15.98:0/926053178 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85ce7fc640 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 0x55fda424a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85cf7fe640 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 0x55fda424ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85ceffd640 1 -- 172.21.15.98:0/926053178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 msgr2=0x55fda424a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85ceffd640 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 0x55fda424a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85ceffd640 1 -- 172.21.15.98:0/926053178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 msgr2=0x55fda424ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:38:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85ceffd640 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 0x55fda424ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85ceffd640 1 -- 172.21.15.98:0/926053178 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda42b9f30 con 0x55fda4239620 2024-04-25T17:38:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.963+0000 7f85ce7fc640 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 0x55fda424a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:38:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85ceffd640 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda424a2a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f85c4010ca0 tx=0x7f85c40058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8475dc3699c3deab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:38:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:38:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85aeffd640 1 -- 172.21.15.98:0/926053178 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4005030 con 0x55fda4239620 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85aeffd640 1 -- 172.21.15.98:0/926053178 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f85c40051d0 con 0x55fda4239620 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85d87d4740 1 -- 172.21.15.98:0/926053178 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85c8062070 con 0x55fda4239620 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85aeffd640 1 -- 172.21.15.98:0/926053178 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f85c4005410 con 0x55fda4239620 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85d87d4740 10 client.25232 fetch_fsmap learned FSMap version 30 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85d87d4740 1 -- 172.21.15.98:0/926053178 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fda42bb240 con 0x55fda4239620 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85aeffd640 1 -- 172.21.15.98:0/926053178 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85c8062070 con 0x55fda4239620 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85aeffd640 1 -- 172.21.15.98:0/926053178 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f85c40117e0 con 0x55fda4239620 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85d87d4740 10 client.25232 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85d87d4740 10 client.25232 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85d87d4740 20 client.25232 populate_metadata read hostname 'smithi098' 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85d87d4740 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fda42bb6f0 0x55fda42bdb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85d87d4740 4 client.25232 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.964+0000 7f85d87d4740 1 -- 172.21.15.98:0/926053178 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fda41a6200 con 0x55fda42bb6f0 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.965+0000 7f85ce7fc640 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fda42bb6f0 0x55fda42bdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:38:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.965+0000 7f85ce7fc640 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fda42bb6f0 0x55fda42bdb10 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc3fde9b9962fb62 server_cookie=2610da02206eb4ec in_seq=0 out_seq=0 2024-04-25T17:38:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.966+0000 7f85aeffd640 10 client.25232 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85aeffd640 1 -- 172.21.15.98:0/926053178 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fda41a6200 con 0x55fda42bb6f0 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85aeffd640 10 client.25232 handle_command_reply: tid=0 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 client.25232 shutdown 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 -- 172.21.15.98:0/926053178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 msgr2=0x55fda424a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda424a2a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f85c4010ca0 tx=0x7f85c40058e0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 -- 172.21.15.98:0/926053178 shutdown_connections 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fda42bb6f0 0x55fda42bdb10 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fda4244800 0x55fda424ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85aeffd640 0 client.25232 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fda4239f40 0x55fda424a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 --2- 172.21.15.98:0/926053178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fda4239620 0x55fda424a2a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 -- 172.21.15.98:0/926053178 >> 172.21.15.98:0/926053178 conn(0x55fda4235b30 msgr2=0x55fda4234cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 -- 172.21.15.98:0/926053178 shutdown_connections 2024-04-25T17:38:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 1 -- 172.21.15.98:0/926053178 wait complete. 2024-04-25T17:38:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.980+0000 7f85d87d4740 20 client.25232 trim_cache size 0 max 16384 2024-04-25T17:38:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:38:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:38:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:38:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 -- 172.21.15.98:0/1425343077 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85b80a3e40 msgr2=0x7f85b80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85b80a3e40 0x7f85b80a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f85c8006010 tx=0x7f85c8031bd0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 -- 172.21.15.98:0/1425343077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d010c6a0 msgr2=0x7f85d010d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d010c6a0 0x7f85d010d110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f85c000e9f0 tx=0x7f85c000eeb0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 -- 172.21.15.98:0/1425343077 shutdown_connections 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85b80a3e40 0x7f85b80a6300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85d0116fe0 0x7f85d010d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d010c6a0 0x7f85d010d110 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 --2- 172.21.15.98:0/1425343077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0109520 0x7f85d010cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 -- 172.21.15.98:0/1425343077 >> 172.21.15.98:0/1425343077 conn(0x7f85d006d4b0 msgr2=0x7f85d01173c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:38:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.982+0000 7f85d59e2640 1 -- 172.21.15.98:0/1425343077 shutdown_connections 2024-04-25T17:38:47.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:38:47.983+0000 7f85d59e2640 1 -- 172.21.15.98:0/1425343077 wait complete. 2024-04-25T17:38:49.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:49 smithi098 ceph-mon[29192]: pgmap v90: 97 pgs: 97 active+clean; 1.3 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 712 op/s 2024-04-25T17:38:49.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:49 smithi203 ceph-mon[32834]: pgmap v90: 97 pgs: 97 active+clean; 1.3 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 712 op/s 2024-04-25T17:38:49.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:49 smithi173 ceph-mon[32856]: pgmap v90: 97 pgs: 97 active+clean; 1.3 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 MiB/s wr, 712 op/s 2024-04-25T17:38:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 140.0 s, 3056.2 tps, lat 32.744 ms stddev 60.368 2024-04-25T17:38:51.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:51 smithi098 ceph-mon[29192]: pgmap v91: 97 pgs: 97 active+clean; 1.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 18 MiB/s wr, 983 op/s 2024-04-25T17:38:51.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:51 smithi203 ceph-mon[32834]: pgmap v91: 97 pgs: 97 active+clean; 1.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 18 MiB/s wr, 983 op/s 2024-04-25T17:38:51.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:51 smithi173 ceph-mon[32856]: pgmap v91: 97 pgs: 97 active+clean; 1.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 18 MiB/s wr, 983 op/s 2024-04-25T17:38:53.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:53 smithi098 ceph-mon[29192]: pgmap v92: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 33 MiB/s wr, 1.31k op/s 2024-04-25T17:38:53.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:53 smithi203 ceph-mon[32834]: pgmap v92: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 33 MiB/s wr, 1.31k op/s 2024-04-25T17:38:53.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:53 smithi173 ceph-mon[32856]: pgmap v92: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 33 MiB/s wr, 1.31k op/s 2024-04-25T17:38:55.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:55 smithi098 ceph-mon[29192]: pgmap v93: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 33 MiB/s wr, 1.31k op/s 2024-04-25T17:38:55.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:55 smithi203 ceph-mon[32834]: pgmap v93: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 33 MiB/s wr, 1.31k op/s 2024-04-25T17:38:55.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:55 smithi173 ceph-mon[32856]: pgmap v93: 97 pgs: 97 active+clean; 1.6 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 33 MiB/s wr, 1.31k op/s 2024-04-25T17:38:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 145.0 s, 3599.0 tps, lat 27.722 ms stddev 29.103 2024-04-25T17:38:57.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:57 smithi098 ceph-mon[29192]: pgmap v94: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 42 MiB/s wr, 1.77k op/s 2024-04-25T17:38:57.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:57 smithi203 ceph-mon[32834]: pgmap v94: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 42 MiB/s wr, 1.77k op/s 2024-04-25T17:38:57.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:57 smithi173 ceph-mon[32856]: pgmap v94: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 42 MiB/s wr, 1.77k op/s 2024-04-25T17:38:59.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:38:59 smithi098 ceph-mon[29192]: pgmap v95: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 39 MiB/s wr, 1.45k op/s 2024-04-25T17:38:59.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:38:59 smithi203 ceph-mon[32834]: pgmap v95: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 39 MiB/s wr, 1.45k op/s 2024-04-25T17:38:59.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:38:59 smithi173 ceph-mon[32856]: pgmap v95: 97 pgs: 97 active+clean; 1.6 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 39 MiB/s wr, 1.45k op/s 2024-04-25T17:39:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.110+0000 7f7c5efd6640 1 -- 172.21.15.98:0/2818919416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c5810c5b0 msgr2=0x7f7c5810e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.110+0000 7f7c5efd6640 1 --2- 172.21.15.98:0/2818919416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c5810c5b0 0x7f7c5810e9e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009ca0 tx=0x7f7c5002a950 comp rx=0 tx=0).stop 2024-04-25T17:39:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.110+0000 7f7c5efd6640 1 -- 172.21.15.98:0/2818919416 shutdown_connections 2024-04-25T17:39:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.110+0000 7f7c5efd6640 1 --2- 172.21.15.98:0/2818919416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c5810c5b0 0x7f7c5810e9e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.110+0000 7f7c5efd6640 1 --2- 172.21.15.98:0/2818919416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c58103b00 0x7f7c5810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.110+0000 7f7c5efd6640 1 --2- 172.21.15.98:0/2818919416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c581031e0 0x7f7c581035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.110+0000 7f7c5efd6640 1 -- 172.21.15.98:0/2818919416 >> 172.21.15.98:0/2818919416 conn(0x7f7c58077fa0 msgr2=0x7f7c580783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.110+0000 7f7c5efd6640 1 -- 172.21.15.98:0/2818919416 shutdown_connections 2024-04-25T17:39:01.113 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.110+0000 7f7c5efd6640 1 -- 172.21.15.98:0/2818919416 wait complete. 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.111+0000 7f7c5efd6640 1 Processor -- start 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.111+0000 7f7c5efd6640 1 -- start start 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.111+0000 7f7c5efd6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c581031e0 0x7f7c58198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.111+0000 7f7c5efd6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c58103b00 0x7f7c58198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.111+0000 7f7c5efd6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c5810c5b0 0x7f7c5819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.111+0000 7f7c5efd6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581121b0 con 0x7f7c58103b00 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.111+0000 7f7c5efd6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58112030 con 0x7f7c581031e0 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.111+0000 7f7c5efd6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58112330 con 0x7f7c5810c5b0 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c58103b00 0x7f7c58198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c58103b00 0x7f7c58198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44754/0 (socket says 172.21.15.98:44754) 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c5cd72640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c581031e0 0x7f7c58198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c57fff640 1 -- 172.21.15.98:0/625141272 learned_addr learned my addr 172.21.15.98:0/625141272 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c5d573640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c5810c5b0 0x7f7c5819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c57fff640 1 -- 172.21.15.98:0/625141272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c5810c5b0 msgr2=0x7f7c5819cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c57fff640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c5810c5b0 0x7f7c5819cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c57fff640 1 -- 172.21.15.98:0/625141272 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c581031e0 msgr2=0x7f7c58198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c57fff640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c581031e0 0x7f7c58198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c57fff640 1 -- 172.21.15.98:0/625141272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5819d6d0 con 0x7f7c58103b00 2024-04-25T17:39:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c57fff640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c58103b00 0x7f7c58198c70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400ea60 tx=0x7f7c4400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c55ffb640 1 -- 172.21.15.98:0/625141272 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400cde0 con 0x7f7c58103b00 2024-04-25T17:39:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c55ffb640 1 -- 172.21.15.98:0/625141272 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c440035f0 con 0x7f7c58103b00 2024-04-25T17:39:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c5efd6640 1 -- 172.21.15.98:0/625141272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c5819d9c0 con 0x7f7c58103b00 2024-04-25T17:39:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.112+0000 7f7c55ffb640 1 -- 172.21.15.98:0/625141272 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44010690 con 0x7f7c58103b00 2024-04-25T17:39:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.113+0000 7f7c5efd6640 1 -- 172.21.15.98:0/625141272 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c581a5250 con 0x7f7c58103b00 2024-04-25T17:39:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.115+0000 7f7c55ffb640 1 -- 172.21.15.98:0/625141272 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7c440037a0 con 0x7f7c58103b00 2024-04-25T17:39:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.115+0000 7f7c61dc5740 1 Processor -- start 2024-04-25T17:39:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.115+0000 7f7c61dc5740 1 -- start start 2024-04-25T17:39:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.115+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccb2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.115+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccbd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.115+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 0x55af3ccbfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.115+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af3cc17970 con 0x55af3ccb2f40 2024-04-25T17:39:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.115+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55af3cbb44a0 con 0x55af3ccbd800 2024-04-25T17:39:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55af3cc13930 con 0x55af3ccb2620 2024-04-25T17:39:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5cd72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccbd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c57fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 0x55af3ccbfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5cd72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccbd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44764/0 (socket says 172.21.15.98:44764) 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5cd72640 1 -- 172.21.15.98:0/2333473518 learned_addr learned my addr 172.21.15.98:0/2333473518 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c55ffb640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7c480a3e40 0x7f7c480a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5d573640 1 --2- 172.21.15.98:0/2333473518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccb2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5cd72640 1 -- 172.21.15.98:0/2333473518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 msgr2=0x55af3ccb2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5cd72640 1 --2- 172.21.15.98:0/2333473518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccb2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5d573640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7c480a3e40 0x7f7c480a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5cd72640 1 -- 172.21.15.98:0/2333473518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 msgr2=0x55af3ccbfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c55ffb640 1 -- 172.21.15.98:0/625141272 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f7c440d09b0 con 0x7f7c58103b00 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5cd72640 1 --2- 172.21.15.98:0/2333473518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 0x55af3ccbfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5cd72640 1 -- 172.21.15.98:0/2333473518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af3c92a210 con 0x55af3ccb2f40 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c57fff640 1 --2- 172.21.15.98:0/2333473518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 0x55af3ccbfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.116+0000 7f7c5cd72640 1 --2- 172.21.15.98:0/2333473518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccbd2c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009cf0 tx=0x7f7c4c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6379f16a27cab4ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.117+0000 7f7c42ffd640 1 -- 172.21.15.98:0/2333473518 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c01e070 con 0x55af3ccb2f40 2024-04-25T17:39:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.117+0000 7f7c42ffd640 1 -- 172.21.15.98:0/2333473518 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c4c019ae0 con 0x55af3ccb2f40 2024-04-25T17:39:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.117+0000 7f7c42ffd640 1 -- 172.21.15.98:0/2333473518 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c00e930 con 0x55af3ccb2f40 2024-04-25T17:39:01.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.117+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2333473518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 msgr2=0x55af3ccbd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.117+0000 7f7c5d573640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7c480a3e40 0x7f7c480a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009430 tx=0x7f7c50005c50 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.117+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2333473518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccbd2c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c009cf0 tx=0x7f7c4c011810 comp rx=0 tx=0).stop 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.118+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2333473518 shutdown_connections 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.118+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2333473518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 0x55af3ccbfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.118+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2333473518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccbd2c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.118+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2333473518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccb2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.118+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2333473518 >> 172.21.15.98:0/2333473518 conn(0x55af3ccaeb30 msgr2=0x55af3ccaef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.118+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2333473518 shutdown_connections 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.118+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2333473518 wait complete. 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.118+0000 7f7c61dc5740 1 -- start start 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.118+0000 7f7c61dc5740 1 Processor -- start 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccbd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccba450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 0x55af3ccba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55af3ccd1440 con 0x55af3ccb2f40 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55af3cbb20b0 con 0x55af3ccbd800 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55af3ccd15c0 con 0x55af3ccb2620 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c61dc5740 10 client.0 _ll_register_callbacks cb 0x55af3cb9fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccba450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c57fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccba450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44776/0 (socket says 172.21.15.98:44776) 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c57fff640 1 -- 172.21.15.98:0/2389570365 learned_addr learned my addr 172.21.15.98:0/2389570365 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c5cd72640 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccbd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c5d573640 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 0x55af3ccba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c57fff640 1 -- 172.21.15.98:0/2389570365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 msgr2=0x55af3ccbd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c57fff640 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccbd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c57fff640 1 -- 172.21.15.98:0/2389570365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 msgr2=0x55af3ccba9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c57fff640 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 0x55af3ccba9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c57fff640 1 -- 172.21.15.98:0/2389570365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af3c9f6620 con 0x55af3ccb2f40 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c57fff640 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccba450 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7c440ce130 tx=0x7f7c44005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c975a657724acf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.119+0000 7f7c37fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.120+0000 7f7c37fff640 1 -- 172.21.15.98:0/2389570365 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44085e60 con 0x55af3ccb2f40 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.120+0000 7f7c37fff640 1 -- 172.21.15.98:0/2389570365 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f7c440ce350 con 0x55af3ccb2f40 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.120+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2389570365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c4c019ae0 con 0x55af3ccb2f40 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.120+0000 7f7c37fff640 1 -- 172.21.15.98:0/2389570365 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7c4409c280 con 0x55af3ccb2f40 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.120+0000 7f7c61dc5740 10 client.15657 fetch_fsmap learned FSMap version 30 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.120+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2389570365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55af3ccd6590 con 0x55af3ccb2f40 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.120+0000 7f7c37fff640 1 -- 172.21.15.98:0/2389570365 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c4c019ae0 con 0x55af3ccb2f40 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.120+0000 7f7c37fff640 1 -- 172.21.15.98:0/2389570365 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f7c440870b0 con 0x55af3ccb2f40 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.121+0000 7f7c61dc5740 10 client.15657 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.121+0000 7f7c61dc5740 10 client.15657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:39:01.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.121+0000 7f7c61dc5740 20 client.15657 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:01.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.121+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55af3ccd7a50 0x55af3ccd9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.121+0000 7f7c61dc5740 4 client.15657 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:39:01.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.121+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2389570365 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af3cc25390 con 0x55af3ccd7a50 2024-04-25T17:39:01.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.121+0000 7f7c5cd72640 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55af3ccd7a50 0x55af3ccd9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.121+0000 7f7c37fff640 10 client.15657 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:01.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.121+0000 7f7c5cd72640 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55af3ccd7a50 0x55af3ccd9ed0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb2e024b067cce0d server_cookie=abab13872a74774d in_seq=0 out_seq=0 2024-04-25T17:39:01.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c37fff640 1 -- 172.21.15.98:0/2389570365 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55af3cc25390 con 0x55af3ccd7a50 2024-04-25T17:39:01.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c37fff640 10 client.15657 handle_command_reply: tid=0 2024-04-25T17:39:01.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 client.15657 shutdown 2024-04-25T17:39:01.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2389570365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 msgr2=0x55af3ccba450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:01.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccba450 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7c440ce130 tx=0x7f7c44005790 comp rx=0 tx=0).stop 2024-04-25T17:39:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2389570365 shutdown_connections 2024-04-25T17:39:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55af3ccd7a50 0x55af3ccd9ed0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccbd800 0x55af3ccba9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2f40 0x55af3ccba450 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2389570365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccbd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2389570365 >> 172.21.15.98:0/2389570365 conn(0x55af3ccaeb30 msgr2=0x55af3ccb4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c37fff640 0 client.15657 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2389570365 shutdown_connections 2024-04-25T17:39:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2389570365 wait complete. 2024-04-25T17:39:01.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.122+0000 7f7c61dc5740 20 client.15657 trim_cache size 0 max 16384 2024-04-25T17:39:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c61dc5740 1 Processor -- start 2024-04-25T17:39:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c61dc5740 1 -- start start 2024-04-25T17:39:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccba410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccb2f40 0x55af3ccbcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccbd800 0x55af3ccf9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c581a5a60 con 0x55af3ccbd800 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58071fc0 con 0x55af3ccb2f40 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5806fe10 con 0x55af3ccb2620 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c5d573640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccbd800 0x55af3ccf9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c5d573640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccbd800 0x55af3ccf9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44786/0 (socket says 172.21.15.98:44786) 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c5d573640 1 -- 172.21.15.98:0/2763193387 learned_addr learned my addr 172.21.15.98:0/2763193387 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c57fff640 1 --2- 172.21.15.98:0/2763193387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccb2f40 0x55af3ccbcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.140+0000 7f7c5cd72640 1 --2- 172.21.15.98:0/2763193387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccba410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.141+0000 7f7c5d573640 1 -- 172.21.15.98:0/2763193387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 msgr2=0x55af3ccba410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.141+0000 7f7c5d573640 1 --2- 172.21.15.98:0/2763193387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccba410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.141+0000 7f7c5d573640 1 -- 172.21.15.98:0/2763193387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccb2f40 msgr2=0x55af3ccbcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.141+0000 7f7c5d573640 1 --2- 172.21.15.98:0/2763193387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccb2f40 0x55af3ccbcc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.141+0000 7f7c5d573640 1 -- 172.21.15.98:0/2763193387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af3ccbad20 con 0x55af3ccbd800 2024-04-25T17:39:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.141+0000 7f7c5d573640 1 --2- 172.21.15.98:0/2763193387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccbd800 0x55af3ccf9ed0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009ca0 tx=0x7f7c50041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6b4db993ccc1c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.141+0000 7f7c427fc640 1 -- 172.21.15.98:0/2763193387 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50034040 con 0x55af3ccbd800 2024-04-25T17:39:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.141+0000 7f7c427fc640 1 -- 172.21.15.98:0/2763193387 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c50003e20 con 0x55af3ccbd800 2024-04-25T17:39:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.141+0000 7f7c427fc640 1 -- 172.21.15.98:0/2763193387 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50004750 con 0x55af3ccbd800 2024-04-25T17:39:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.142+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2763193387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccbd800 msgr2=0x55af3ccf9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.142+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2763193387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccbd800 0x55af3ccf9ed0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009ca0 tx=0x7f7c50041000 comp rx=0 tx=0).stop 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.142+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2763193387 shutdown_connections 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.142+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2763193387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccbd800 0x55af3ccf9ed0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.142+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2763193387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccb2f40 0x55af3ccbcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.142+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/2763193387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccb2620 0x55af3ccba410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.142+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2763193387 >> 172.21.15.98:0/2763193387 conn(0x55af3ccaeb30 msgr2=0x55af3ccaef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.142+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2763193387 shutdown_connections 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.142+0000 7f7c61dc5740 1 -- 172.21.15.98:0/2763193387 wait complete. 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c61dc5740 1 -- start start 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c61dc5740 1 Processor -- start 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2620 0x55af3ccdfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccb2f40 0x55af3ccdbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c61dc5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccbd800 0x55af3ccdc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58070480 con 0x55af3ccb2620 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7c580718f0 con 0x55af3ccb2f40 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c61dc5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58070b50 con 0x55af3ccbd800 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c61dc5740 10 client.0 _ll_register_callbacks cb 0x7f7c44004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c5cd72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2620 0x55af3ccdfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c5cd72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2620 0x55af3ccdfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44790/0 (socket says 172.21.15.98:44790) 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c5d573640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccbd800 0x55af3ccdc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c57fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccb2f40 0x55af3ccdbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.143+0000 7f7c5cd72640 1 -- 172.21.15.98:0/294721312 learned_addr learned my addr 172.21.15.98:0/294721312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c5cd72640 1 -- 172.21.15.98:0/294721312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccbd800 msgr2=0x55af3ccdc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c5cd72640 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccbd800 0x55af3ccdc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c5cd72640 1 -- 172.21.15.98:0/294721312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccb2f40 msgr2=0x55af3ccdbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c5cd72640 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccb2f40 0x55af3ccdbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c5cd72640 1 -- 172.21.15.98:0/294721312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af3ccb2d70 con 0x55af3ccb2620 2024-04-25T17:39:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c5cd72640 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2620 0x55af3ccdfb80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00e5a0 tx=0x7f7c4c009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca29c487cfdb943 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c37fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c37fff640 1 -- 172.21.15.98:0/294721312 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c029e60 con 0x55af3ccb2620 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c61dc5740 1 -- 172.21.15.98:0/294721312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c50003e20 con 0x55af3ccb2620 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c37fff640 1 -- 172.21.15.98:0/294721312 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8469+0+0 (secure 0 0 0) 0x7f7c4c0054c0 con 0x55af3ccb2620 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c37fff640 1 -- 172.21.15.98:0/294721312 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7c4c037050 con 0x55af3ccb2620 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c61dc5740 10 client.15663 fetch_fsmap learned FSMap version 30 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c61dc5740 1 -- 172.21.15.98:0/294721312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55af3cd55590 con 0x55af3ccb2620 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.144+0000 7f7c37fff640 1 -- 172.21.15.98:0/294721312 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c4c02e070 con 0x55af3ccb2620 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.145+0000 7f7c37fff640 1 -- 172.21.15.98:0/294721312 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f7c4c00e6d0 con 0x55af3ccb2620 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.145+0000 7f7c61dc5740 10 client.15663 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.145+0000 7f7c61dc5740 10 client.15663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.145+0000 7f7c61dc5740 20 client.15663 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.145+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55af3cd55b70 0x55af3cd57f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:01.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.145+0000 7f7c61dc5740 4 client.15663 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:39:01.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.145+0000 7f7c61dc5740 1 -- 172.21.15.98:0/294721312 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7c50003e20 con 0x55af3cd55b70 2024-04-25T17:39:01.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.145+0000 7f7c57fff640 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55af3cd55b70 0x55af3cd57f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:01.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.145+0000 7f7c37fff640 10 client.15663 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:01.150 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:01.145+0000 7f7c57fff640 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55af3cd55b70 0x55af3cd57f90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff9e98280bb88d8c server_cookie=7ce8d98f42a51a5a in_seq=0 out_seq=0 2024-04-25T17:39:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 150.0 s, 3240.4 tps, lat 30.667 ms stddev 33.646 2024-04-25T17:39:01.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:01 smithi098 ceph-mon[29192]: pgmap v96: 97 pgs: 97 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 40 MiB/s wr, 1.55k op/s 2024-04-25T17:39:01.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:01 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:39:01.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:01 smithi203 ceph-mon[32834]: pgmap v96: 97 pgs: 97 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 40 MiB/s wr, 1.55k op/s 2024-04-25T17:39:01.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:01 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:39:01.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:01 smithi173 ceph-mon[32856]: pgmap v96: 97 pgs: 97 active+clean; 1.7 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 40 MiB/s wr, 1.55k op/s 2024-04-25T17:39:01.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:01 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:39:02.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:02 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:39:02.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:02 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:39:02.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:02 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:39:02.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:02 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:39:02.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:02 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:39:02.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:02 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:39:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c37fff640 1 -- 172.21.15.98:0/294721312 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7c50003e20 con 0x55af3cd55b70 2024-04-25T17:39:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c37fff640 10 client.15663 handle_command_reply: tid=0 2024-04-25T17:39:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c61dc5740 1 client.15663 shutdown 2024-04-25T17:39:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c61dc5740 1 -- 172.21.15.98:0/294721312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2620 msgr2=0x55af3ccdfb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2620 0x55af3ccdfb80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00e5a0 tx=0x7f7c4c009ef0 comp rx=0 tx=0).stop 2024-04-25T17:39:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c61dc5740 1 -- 172.21.15.98:0/294721312 shutdown_connections 2024-04-25T17:39:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55af3cd55b70 0x55af3cd57f90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55af3ccbd800 0x55af3ccdc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55af3ccb2f40 0x55af3ccdbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c37fff640 0 client.15663 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c61dc5740 1 --2- 172.21.15.98:0/294721312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55af3ccb2620 0x55af3ccdfb80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.744+0000 7f7c61dc5740 1 -- 172.21.15.98:0/294721312 >> 172.21.15.98:0/294721312 conn(0x55af3ccaeb30 msgr2=0x55af3cce2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c61dc5740 1 -- 172.21.15.98:0/294721312 shutdown_connections 2024-04-25T17:39:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c61dc5740 1 -- 172.21.15.98:0/294721312 wait complete. 2024-04-25T17:39:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c61dc5740 20 client.15663 trim_cache size 0 max 16384 2024-04-25T17:39:02.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:39:02.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:39:02.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:39:02.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c5efd6640 1 -- 172.21.15.98:0/625141272 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7c480a3e40 msgr2=0x7f7c480a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c5efd6640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7c480a3e40 0x7f7c480a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009430 tx=0x7f7c50005c50 comp rx=0 tx=0).stop 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c5efd6640 1 -- 172.21.15.98:0/625141272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c58103b00 msgr2=0x7f7c58198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c5efd6640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c58103b00 0x7f7c58198c70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400ea60 tx=0x7f7c4400ef20 comp rx=0 tx=0).stop 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c5efd6640 1 -- 172.21.15.98:0/625141272 shutdown_connections 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c5efd6640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7c480a3e40 0x7f7c480a6300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c5efd6640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c5810c5b0 0x7f7c5819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c5efd6640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c58103b00 0x7f7c58198c70 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c5efd6640 1 --2- 172.21.15.98:0/625141272 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c581031e0 0x7f7c58198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.745+0000 7f7c5efd6640 1 -- 172.21.15.98:0/625141272 >> 172.21.15.98:0/625141272 conn(0x7f7c58077fa0 msgr2=0x7f7c580ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.746+0000 7f7c5efd6640 1 -- 172.21.15.98:0/625141272 shutdown_connections 2024-04-25T17:39:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:02.746+0000 7f7c5efd6640 1 -- 172.21.15.98:0/625141272 wait complete. 2024-04-25T17:39:02.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:02 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:39:02.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:02 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:39:02.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:02 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:39:03.458 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: *** Caught signal (Segmentation fault) ** 2024-04-25T17:39:03.458 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: in thread 7fbd4dda8700 thread_name:md_log_replay 2024-04-25T17:39:03.458 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T17:39:03.458 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbd5f169d20] 2024-04-25T17:39:03.458 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2: /usr/bin/ceph-mds(+0x549372) [0x564746edb372] 2024-04-25T17:39:03.458 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x564746edeb23] 2024-04-25T17:39:03.458 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 4: (MDLog::_replay_thread()+0x617) [0x564746f39647] 2024-04-25T17:39:03.458 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 5: (MDLog::ReplayThread::entry()+0x11) [0x564746f42d31] 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fbd5f15f1ca] 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 7: clone() 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2024-04-25T17:39:03.049+0000 7fbd4dda8700 -1 *** Caught signal (Segmentation fault) ** 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: in thread 7fbd4dda8700 thread_name:md_log_replay 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbd5f169d20] 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2: /usr/bin/ceph-mds(+0x549372) [0x564746edb372] 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x564746edeb23] 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 4: (MDLog::_replay_thread()+0x617) [0x564746f39647] 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 5: (MDLog::ReplayThread::entry()+0x11) [0x564746f42d31] 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fbd5f15f1ca] 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 7: clone() 2024-04-25T17:39:03.459 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T17:39:03.460 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2024-04-25T17:39:03.460 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 0> 2024-04-25T17:39:03.049+0000 7fbd4dda8700 -1 *** Caught signal (Segmentation fault) ** 2024-04-25T17:39:03.460 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: in thread 7fbd4dda8700 thread_name:md_log_replay 2024-04-25T17:39:03.460 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2024-04-25T17:39:03.460 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T17:39:03.460 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbd5f169d20] 2024-04-25T17:39:03.460 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2: /usr/bin/ceph-mds(+0x549372) [0x564746edb372] 2024-04-25T17:39:03.460 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x564746edeb23] 2024-04-25T17:39:03.460 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 4: (MDLog::_replay_thread()+0x617) [0x564746f39647] 2024-04-25T17:39:03.460 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 5: (MDLog::ReplayThread::entry()+0x11) [0x564746f42d31] 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fbd5f15f1ca] 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 7: clone() 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 0> 2024-04-25T17:39:03.049+0000 7fbd4dda8700 -1 *** Caught signal (Segmentation fault) ** 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: in thread 7fbd4dda8700 thread_name:md_log_replay 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbd5f169d20] 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2: /usr/bin/ceph-mds(+0x549372) [0x564746edb372] 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x564746edeb23] 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 4: (MDLog::_replay_thread()+0x617) [0x564746f39647] 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 5: (MDLog::ReplayThread::entry()+0x11) [0x564746f42d31] 2024-04-25T17:39:03.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fbd5f15f1ca] 2024-04-25T17:39:03.462 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 7: clone() 2024-04-25T17:39:03.462 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T17:39:03.462 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[63562]: 2024-04-25T17:39:03.462 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 podman[67952]: 2024-04-25 17:39:03.114579578 +0000 UTC m=+0.023789302 container died 9af6cac92cbc05fb57f0aa3fe60af037ab0a0fcc89d7e27cc53cb2d5a2c00bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-25T17:39:03.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:03 smithi203 ceph-mon[32834]: pgmap v97: 97 pgs: 97 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 MiB/s wr, 1.50k op/s 2024-04-25T17:39:03.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:03 smithi098 ceph-mon[29192]: pgmap v97: 97 pgs: 97 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 MiB/s wr, 1.50k op/s 2024-04-25T17:39:04.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:03 smithi173 ceph-mon[32856]: pgmap v97: 97 pgs: 97 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 MiB/s wr, 1.50k op/s 2024-04-25T17:39:04.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 podman[67952]: 2024-04-25 17:39:03.884783811 +0000 UTC m=+0.793993533 container remove 9af6cac92cbc05fb57f0aa3fe60af037ab0a0fcc89d7e27cc53cb2d5a2c00bcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T17:39:04.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:03 smithi203 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-04-25T17:39:04.712 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:04 smithi203 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-25T17:39:04.712 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:04 smithi203 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.909s CPU time. 2024-04-25T17:39:05.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:05 smithi098 ceph-mon[29192]: pgmap v98: 97 pgs: 97 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 782 op/s 2024-04-25T17:39:05.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:05 smithi203 ceph-mon[32834]: pgmap v98: 97 pgs: 97 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 782 op/s 2024-04-25T17:39:06.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:05 smithi173 ceph-mon[32856]: pgmap v98: 97 pgs: 97 active+clean; 1.7 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 13 MiB/s wr, 782 op/s 2024-04-25T17:39:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 155.0 s, 1735.6 tps, lat 58.101 ms stddev 83.720 2024-04-25T17:39:07.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:07 smithi098 ceph-mon[29192]: pgmap v99: 97 pgs: 97 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 31 MiB/s wr, 2.27k op/s 2024-04-25T17:39:07.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:07 smithi203 ceph-mon[32834]: pgmap v99: 97 pgs: 97 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 31 MiB/s wr, 2.27k op/s 2024-04-25T17:39:08.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:07 smithi173 ceph-mon[32856]: pgmap v99: 97 pgs: 97 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 31 MiB/s wr, 2.27k op/s 2024-04-25T17:39:09.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:09 smithi098 ceph-mon[29192]: pgmap v100: 97 pgs: 97 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 22 MiB/s wr, 1.81k op/s 2024-04-25T17:39:09.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:09 smithi203 ceph-mon[32834]: pgmap v100: 97 pgs: 97 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 22 MiB/s wr, 1.81k op/s 2024-04-25T17:39:10.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:09 smithi173 ceph-mon[32856]: pgmap v100: 97 pgs: 97 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 22 MiB/s wr, 1.81k op/s 2024-04-25T17:39:11.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 160.0 s, 3011.2 tps, lat 33.128 ms stddev 34.626 2024-04-25T17:39:11.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:11 smithi098 ceph-mon[29192]: pgmap v101: 97 pgs: 97 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 MiB/s wr, 1.86k op/s 2024-04-25T17:39:11.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:11 smithi203 ceph-mon[32834]: pgmap v101: 97 pgs: 97 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 MiB/s wr, 1.86k op/s 2024-04-25T17:39:12.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:11 smithi173 ceph-mon[32856]: pgmap v101: 97 pgs: 97 active+clean; 1.7 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 23 MiB/s wr, 1.86k op/s 2024-04-25T17:39:13.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:13 smithi098 ceph-mon[29192]: pgmap v102: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 24 MiB/s wr, 2.03k op/s 2024-04-25T17:39:13.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:13 smithi203 ceph-mon[32834]: pgmap v102: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 24 MiB/s wr, 2.03k op/s 2024-04-25T17:39:14.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:13 smithi173 ceph-mon[32856]: pgmap v102: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 24 MiB/s wr, 2.03k op/s 2024-04-25T17:39:14.726 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:14 smithi203 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-04-25T17:39:14.727 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:14 smithi203 systemd[1]: Stopped Ceph mds.c for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:39:14.727 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:14 smithi203 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.909s CPU time. 2024-04-25T17:39:14.727 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:14 smithi203 systemd[1]: Starting Ceph mds.c for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:39:14.727 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:14 smithi203 podman[68044]: 2024-04-25 17:39:14.607858992 +0000 UTC m=+0.019408356 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:39:14.999 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:14 smithi203 podman[68044]: 2024-04-25 17:39:14.727099491 +0000 UTC m=+0.138648828 container create 905b117a93741a29c8d62f9ed84da14c1a550e4c42da7e2706c829a3c8aca1f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:39:15.000 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:14 smithi203 podman[68044]: 2024-04-25 17:39:14.97203933 +0000 UTC m=+0.383588668 container init 905b117a93741a29c8d62f9ed84da14c1a550e4c42da7e2706c829a3c8aca1f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T17:39:15.000 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:14 smithi203 podman[68044]: 2024-04-25 17:39:14.975676557 +0000 UTC m=+0.387225899 container start 905b117a93741a29c8d62f9ed84da14c1a550e4c42da7e2706c829a3c8aca1f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:39:15.396 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:15 smithi203 ceph-mds[68062]: -- 172.21.15.203:0/2426240237 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556324c71a40 con 0x556324c44800 2024-04-25T17:39:15.397 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:15 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: starting mds.c at 2024-04-25T17:39:15.397 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:15 smithi203 bash[68044]: 905b117a93741a29c8d62f9ed84da14c1a550e4c42da7e2706c829a3c8aca1f8 2024-04-25T17:39:15.397 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:39:15 smithi203 systemd[1]: Started Ceph mds.c for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:39:15.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:15 smithi203 ceph-mon[32834]: pgmap v103: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 MiB/s wr, 1.81k op/s 2024-04-25T17:39:15.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:15 smithi203 ceph-mon[32834]: daemon mds.c restarted 2024-04-25T17:39:15.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:15 smithi203 ceph-mon[32834]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T17:39:15.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:15 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.203:6836/1401608926,v1:172.21.15.203:6837/1401608926] up:boot 2024-04-25T17:39:15.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:15 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:39:15.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:15 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:39:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.864+0000 7fb6b0005640 1 -- 172.21.15.98:0/698951857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a8108f50 msgr2=0x7fb6a8109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.864+0000 7fb6b0005640 1 --2- 172.21.15.98:0/698951857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a8108f50 0x7fb6a8109330 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4009900 tx=0x7fb6a402a480 comp rx=0 tx=0).stop 2024-04-25T17:39:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.864+0000 7fb6b0005640 1 -- 172.21.15.98:0/698951857 shutdown_connections 2024-04-25T17:39:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.864+0000 7fb6b0005640 1 --2- 172.21.15.98:0/698951857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6a81150a0 0x7fb6a8117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.864+0000 7fb6b0005640 1 --2- 172.21.15.98:0/698951857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6a8109870 0x7fb6a8114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.864+0000 7fb6b0005640 1 --2- 172.21.15.98:0/698951857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a8108f50 0x7fb6a8109330 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.864+0000 7fb6b0005640 1 -- 172.21.15.98:0/698951857 >> 172.21.15.98:0/698951857 conn(0x7fb6a806d020 msgr2=0x7fb6a806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.864+0000 7fb6b0005640 1 -- 172.21.15.98:0/698951857 shutdown_connections 2024-04-25T17:39:15.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.864+0000 7fb6b0005640 1 -- 172.21.15.98:0/698951857 wait complete. 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6b0005640 1 Processor -- start 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6b0005640 1 -- start start 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6b0005640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6a8108f50 0x7fb6a81a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6b0005640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a8109870 0x7fb6a81a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6b0005640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6a81150a0 0x7fb6a81a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6b0005640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a811aa20 con 0x7fb6a8109870 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6b0005640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a811a8a0 con 0x7fb6a81150a0 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6b0005640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a811aba0 con 0x7fb6a8108f50 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6ad5a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a8109870 0x7fb6a81a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6ad5a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a8109870 0x7fb6a81a1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53868/0 (socket says 172.21.15.98:53868) 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.865+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3283131590 learned_addr learned my addr 172.21.15.98:0/3283131590 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.866+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3283131590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6a8108f50 msgr2=0x7fb6a81a0ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.866+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6a8108f50 0x7fb6a81a0ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.866+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3283131590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6a81150a0 msgr2=0x7fb6a81a57a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T17:39:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.866+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6a81150a0 0x7fb6a81a57a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.866+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3283131590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a81a5ed0 con 0x7fb6a8109870 2024-04-25T17:39:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.866+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a8109870 0x7fb6a81a1420 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb69800e9f0 tx=0x7fb69800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.866+0000 7fb69effd640 1 -- 172.21.15.98:0/3283131590 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69800cd80 con 0x7fb6a8109870 2024-04-25T17:39:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.866+0000 7fb69effd640 1 -- 172.21.15.98:0/3283131590 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6980108e0 con 0x7fb6a8109870 2024-04-25T17:39:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.866+0000 7fb6b0005640 1 -- 172.21.15.98:0/3283131590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a81a61c0 con 0x7fb6a8109870 2024-04-25T17:39:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.867+0000 7fb69effd640 1 -- 172.21.15.98:0/3283131590 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6980188e0 con 0x7fb6a8109870 2024-04-25T17:39:15.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.867+0000 7fb6b0005640 1 -- 172.21.15.98:0/3283131590 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a81ada50 con 0x7fb6a8109870 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.869+0000 7fb69effd640 1 -- 172.21.15.98:0/3283131590 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb698010430 con 0x7fb6a8109870 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.869+0000 7fb6b2df7740 1 Processor -- start 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6b2df7740 1 -- start start 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb69effd640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6940a3e40 0x7fb6940a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913520 0x55dc6f913900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913e40 0x55dc6f9226d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f922c10 0x55dc6f925000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dc6f878970 con 0x55dc6f913520 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dc6f8154a0 con 0x55dc6f922c10 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dc6f874930 con 0x55dc6f913e40 2024-04-25T17:39:15.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb69effd640 1 -- 172.21.15.98:0/3283131590 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fb6980d0c00 con 0x7fb6a8109870 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6adda1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913520 0x55dc6f913900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6ae5a2640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6940a3e40 0x7fb6940a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6adda1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913520 0x55dc6f913900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53876/0 (socket says 172.21.15.98:53876) 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6adda1640 1 -- 172.21.15.98:0/284639407 learned_addr learned my addr 172.21.15.98:0/284639407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/284639407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913e40 0x55dc6f9226d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6ae5a2640 1 --2- 172.21.15.98:0/284639407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f922c10 0x55dc6f925000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6adda1640 1 -- 172.21.15.98:0/284639407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913e40 msgr2=0x55dc6f9226d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6adda1640 1 --2- 172.21.15.98:0/284639407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913e40 0x55dc6f9226d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6adda1640 1 -- 172.21.15.98:0/284639407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f922c10 msgr2=0x55dc6f925000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6adda1640 1 --2- 172.21.15.98:0/284639407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f922c10 0x55dc6f925000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6adda1640 1 -- 172.21.15.98:0/284639407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc6f58b210 con 0x55dc6f913520 2024-04-25T17:39:15.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.870+0000 7fb6ae5a2640 1 --2- 172.21.15.98:0/284639407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f922c10 0x55dc6f925000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:39:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.871+0000 7fb6adda1640 1 --2- 172.21.15.98:0/284639407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913520 0x55dc6f913900 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4009230 tx=0x7fb6a4005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276b1e97359ec87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.871+0000 7fb693fff640 1 -- 172.21.15.98:0/284639407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a403e030 con 0x55dc6f913520 2024-04-25T17:39:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.871+0000 7fb693fff640 1 -- 172.21.15.98:0/284639407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6a40035f0 con 0x55dc6f913520 2024-04-25T17:39:15.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.872+0000 7fb693fff640 1 -- 172.21.15.98:0/284639407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4035040 con 0x55dc6f913520 2024-04-25T17:39:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.872+0000 7fb6b2df7740 1 -- 172.21.15.98:0/284639407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913520 msgr2=0x55dc6f913900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.872+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/284639407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913520 0x55dc6f913900 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4009230 tx=0x7fb6a4005e60 comp rx=0 tx=0).stop 2024-04-25T17:39:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.873+0000 7fb6ae5a2640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6940a3e40 0x7fb6940a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55dc6f9150e0 tx=0x7fb6a0009920 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:15.877 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.873+0000 7fb6b2df7740 1 -- 172.21.15.98:0/284639407 shutdown_connections 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.873+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/284639407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f922c10 0x55dc6f925000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.873+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/284639407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913e40 0x55dc6f9226d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.873+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/284639407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913520 0x55dc6f913900 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.873+0000 7fb6b2df7740 1 -- 172.21.15.98:0/284639407 >> 172.21.15.98:0/284639407 conn(0x55dc6f90fb30 msgr2=0x55dc6f90ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.874+0000 7fb6b2df7740 1 -- 172.21.15.98:0/284639407 shutdown_connections 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.874+0000 7fb6b2df7740 1 -- 172.21.15.98:0/284639407 wait complete. 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.874+0000 7fb6b2df7740 1 -- start start 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.874+0000 7fb6b2df7740 1 Processor -- start 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.874+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 0x55dc6f9224b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913e40 0x55dc6f91f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 0x55dc6f91fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55dc6f9257e0 con 0x55dc6f922c10 2024-04-25T17:39:15.878 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55dc6f8130b0 con 0x55dc6f913520 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55dc6f925960 con 0x55dc6f913e40 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6adda1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 0x55dc6f9224b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6b2df7740 10 client.0 _ll_register_callbacks cb 0x55dc6f800d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6ad5a0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913e40 0x55dc6f91f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6adda1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 0x55dc6f9224b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:55840/0 (socket says 172.21.15.98:55840) 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6adda1640 1 -- 172.21.15.98:0/2844186516 learned_addr learned my addr 172.21.15.98:0/2844186516 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6ae5a2640 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 0x55dc6f91fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6adda1640 1 -- 172.21.15.98:0/2844186516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913e40 msgr2=0x55dc6f91f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6adda1640 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913e40 0x55dc6f91f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6adda1640 1 -- 172.21.15.98:0/2844186516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 msgr2=0x55dc6f91fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6adda1640 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 0x55dc6f91fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6adda1640 1 -- 172.21.15.98:0/2844186516 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc6f657620 con 0x55dc6f913520 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6adda1640 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 0x55dc6f9224b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb6a402a9a0 tx=0x7fb6a4009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f63eee15e209e315 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:15.879 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.875+0000 7fb6917fa640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6917fa640 1 -- 172.21.15.98:0/2844186516 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4035040 con 0x55dc6f913520 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6917fa640 1 -- 172.21.15.98:0/2844186516 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fb6a4003bb0 con 0x55dc6f913520 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6b2df7740 1 -- 172.21.15.98:0/2844186516 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6a40035f0 con 0x55dc6f913520 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6917fa640 1 -- 172.21.15.98:0/2844186516 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb6a40314a0 con 0x55dc6f913520 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6b2df7740 10 client.25262 fetch_fsmap learned FSMap version 31 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6b2df7740 1 -- 172.21.15.98:0/2844186516 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dc6f931650 con 0x55dc6f913520 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6917fa640 1 -- 172.21.15.98:0/2844186516 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6a40035f0 con 0x55dc6f913520 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6917fa640 1 -- 172.21.15.98:0/2844186516 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb6a4043510 con 0x55dc6f913520 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6b2df7740 10 client.25262 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6b2df7740 10 client.25262 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6b2df7740 20 client.25262 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55dc6f9329a0 0x55dc6f934d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6b2df7740 4 client.25262 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:39:15.880 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.876+0000 7fb6b2df7740 1 -- 172.21.15.98:0/2844186516 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc6f886390 con 0x55dc6f9329a0 2024-04-25T17:39:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.877+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55dc6f9329a0 0x55dc6f934d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.877+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55dc6f9329a0 0x55dc6f934d90 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dd89e8039ba5562 server_cookie=f94e657c2e9bdd5a in_seq=0 out_seq=0 2024-04-25T17:39:15.881 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.877+0000 7fb6917fa640 10 client.25262 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6917fa640 1 -- 172.21.15.98:0/2844186516 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dc6f886390 con 0x55dc6f9329a0 2024-04-25T17:39:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6917fa640 10 client.25262 handle_command_reply: tid=0 2024-04-25T17:39:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6b2df7740 1 client.25262 shutdown 2024-04-25T17:39:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6b2df7740 1 -- 172.21.15.98:0/2844186516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 msgr2=0x55dc6f9224b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 0x55dc6f9224b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb6a402a9a0 tx=0x7fb6a4009640 comp rx=0 tx=0).stop 2024-04-25T17:39:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6b2df7740 1 -- 172.21.15.98:0/2844186516 shutdown_connections 2024-04-25T17:39:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55dc6f9329a0 0x55dc6f934d90 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 0x55dc6f91fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913e40 0x55dc6f91f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/2844186516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 0x55dc6f9224b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6917fa640 0 client.25262 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6b2df7740 1 -- 172.21.15.98:0/2844186516 >> 172.21.15.98:0/2844186516 conn(0x55dc6f90fb30 msgr2=0x55dc6f9159a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.878+0000 7fb6b2df7740 1 -- 172.21.15.98:0/2844186516 shutdown_connections 2024-04-25T17:39:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.879+0000 7fb6b2df7740 1 -- 172.21.15.98:0/2844186516 wait complete. 2024-04-25T17:39:15.883 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.879+0000 7fb6b2df7740 20 client.25262 trim_cache size 0 max 16384 2024-04-25T17:39:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.898+0000 7fb6b2df7740 1 Processor -- start 2024-04-25T17:39:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.898+0000 7fb6b2df7740 1 -- start start 2024-04-25T17:39:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.898+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913520 0x55dc6f92e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.898+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913e40 0x55dc6f91af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.898+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 0x55dc6f91e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.898+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a81b6ba0 con 0x55dc6f922c10 2024-04-25T17:39:15.899 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.898+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a81b43b0 con 0x55dc6f913e40 2024-04-25T17:39:15.900 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.898+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a81b2200 con 0x55dc6f913520 2024-04-25T17:39:15.900 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.899+0000 7fb6ae5a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 0x55dc6f91e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.900 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.899+0000 7fb6ad5a0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913e40 0x55dc6f91af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.900 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.899+0000 7fb6adda1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913520 0x55dc6f92e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.900 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.899+0000 7fb6ae5a2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 0x55dc6f91e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53890/0 (socket says 172.21.15.98:53890) 2024-04-25T17:39:15.900 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.899+0000 7fb6ae5a2640 1 -- 172.21.15.98:0/3254966246 learned_addr learned my addr 172.21.15.98:0/3254966246 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:15.900 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.899+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3254966246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913520 msgr2=0x55dc6f92e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:15.900 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.899+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/3254966246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913520 0x55dc6f92e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.901 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.899+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3254966246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 msgr2=0x55dc6f91e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:15.901 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.899+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/3254966246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 0x55dc6f91e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.901 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.899+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3254966246 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc6f914380 con 0x55dc6f913e40 2024-04-25T17:39:15.901 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.900+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/3254966246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913e40 0x55dc6f91af00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb6980ce380 tx=0x7fb698053040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6c9ceb01b1c978f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:15.901 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.900+0000 7fb6937fe640 1 -- 172.21.15.98:0/3254966246 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69809cbd0 con 0x55dc6f913e40 2024-04-25T17:39:15.901 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.900+0000 7fb6937fe640 1 -- 172.21.15.98:0/3254966246 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6980a0f80 con 0x55dc6f913e40 2024-04-25T17:39:15.901 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.900+0000 7fb6937fe640 1 -- 172.21.15.98:0/3254966246 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698099240 con 0x55dc6f913e40 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3254966246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913e40 msgr2=0x55dc6f91af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/3254966246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913e40 0x55dc6f91af00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb6980ce380 tx=0x7fb698053040 comp rx=0 tx=0).stop 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3254966246 shutdown_connections 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/3254966246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f922c10 0x55dc6f91e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/3254966246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913e40 0x55dc6f91af00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/3254966246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f913520 0x55dc6f92e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3254966246 >> 172.21.15.98:0/3254966246 conn(0x55dc6f90fb30 msgr2=0x55dc6f91e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3254966246 shutdown_connections 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3254966246 wait complete. 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 -- start start 2024-04-25T17:39:15.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.901+0000 7fb6b2df7740 1 Processor -- start 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 0x55dc6f92ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913e40 0x55dc6f91bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6b2df7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f922c10 0x55dc6f91c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a81b2870 con 0x55dc6f913e40 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a81b3ce0 con 0x55dc6f913520 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6b2df7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a81b2f40 con 0x55dc6f922c10 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6b2df7740 10 client.0 _ll_register_callbacks cb 0x7fb6a4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6ad5a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913e40 0x55dc6f91bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6adda1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 0x55dc6f92ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6ad5a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913e40 0x55dc6f91bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53898/0 (socket says 172.21.15.98:53898) 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3813870441 learned_addr learned my addr 172.21.15.98:0/3813870441 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6ae5a2640 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f922c10 0x55dc6f91c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3813870441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f922c10 msgr2=0x55dc6f91c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:15.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f922c10 0x55dc6f91c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3813870441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 msgr2=0x55dc6f92ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 0x55dc6f92ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.902+0000 7fb6ad5a0640 1 -- 172.21.15.98:0/3813870441 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc6f90f960 con 0x55dc6f913e40 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6ad5a0640 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913e40 0x55dc6f91bfd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb698060ee0 tx=0x7fb698098f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ecb3b80a647bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6917fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6917fa640 1 -- 172.21.15.98:0/3813870441 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69809cbd0 con 0x55dc6f913e40 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6917fa640 1 -- 172.21.15.98:0/3813870441 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fb698099240 con 0x55dc6f913e40 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3813870441 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6980a0f80 con 0x55dc6f913e40 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6917fa640 1 -- 172.21.15.98:0/3813870441 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb6980b6150 con 0x55dc6f913e40 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6b2df7740 10 client.15684 fetch_fsmap learned FSMap version 31 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3813870441 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dc6f9163a0 con 0x55dc6f913e40 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6917fa640 1 -- 172.21.15.98:0/3813870441 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6980a0f80 con 0x55dc6f913e40 2024-04-25T17:39:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6917fa640 1 -- 172.21.15.98:0/3813870441 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb698050b20 con 0x55dc6f913e40 2024-04-25T17:39:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6b2df7740 10 client.15684 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:39:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6b2df7740 10 client.15684 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:39:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6b2df7740 20 client.15684 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55dc6f916b00 0x55dc6f918f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6b2df7740 4 client.15684 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:39:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.903+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3813870441 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dc6f920070 con 0x55dc6f916b00 2024-04-25T17:39:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.904+0000 7fb6adda1640 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55dc6f916b00 0x55dc6f918f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.904+0000 7fb6adda1640 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55dc6f916b00 0x55dc6f918f20 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b939e92a2d17dd8 server_cookie=4bf5ee0fcdea137e in_seq=0 out_seq=0 2024-04-25T17:39:15.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.904+0000 7fb6917fa640 10 client.15684 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6917fa640 1 -- 172.21.15.98:0/3813870441 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dc6f920070 con 0x55dc6f916b00 2024-04-25T17:39:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6917fa640 10 client.15684 handle_command_reply: tid=0 2024-04-25T17:39:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6b2df7740 1 client.15684 shutdown 2024-04-25T17:39:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3813870441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913e40 msgr2=0x55dc6f91bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913e40 0x55dc6f91bfd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb698060ee0 tx=0x7fb698098f20 comp rx=0 tx=0).stop 2024-04-25T17:39:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3813870441 shutdown_connections 2024-04-25T17:39:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55dc6f916b00 0x55dc6f918f20 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55dc6f922c10 0x55dc6f91c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dc6f913e40 0x55dc6f91bfd0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6917fa640 0 client.15684 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6b2df7740 1 --2- 172.21.15.98:0/3813870441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55dc6f913520 0x55dc6f92ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.920+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3813870441 >> 172.21.15.98:0/3813870441 conn(0x55dc6f90fb30 msgr2=0x55dc6f91e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.921+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3813870441 shutdown_connections 2024-04-25T17:39:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.921+0000 7fb6b2df7740 1 -- 172.21.15.98:0/3813870441 wait complete. 2024-04-25T17:39:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.921+0000 7fb6b2df7740 20 client.15684 trim_cache size 0 max 16384 2024-04-25T17:39:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:39:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:39:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:39:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:39:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 -- 172.21.15.98:0/3283131590 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6940a3e40 msgr2=0x7fb6940a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6940a3e40 0x7fb6940a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55dc6f9150e0 tx=0x7fb6a0009920 comp rx=0 tx=0).stop 2024-04-25T17:39:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 -- 172.21.15.98:0/3283131590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a8109870 msgr2=0x7fb6a81a1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a8109870 0x7fb6a81a1420 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb69800e9f0 tx=0x7fb69800eeb0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 -- 172.21.15.98:0/3283131590 shutdown_connections 2024-04-25T17:39:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6940a3e40 0x7fb6940a6300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6a81150a0 0x7fb6a81a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6a8109870 0x7fb6a81a1420 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 --2- 172.21.15.98:0/3283131590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6a8108f50 0x7fb6a81a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 -- 172.21.15.98:0/3283131590 >> 172.21.15.98:0/3283131590 conn(0x7fb6a806d020 msgr2=0x7fb6a8070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 -- 172.21.15.98:0/3283131590 shutdown_connections 2024-04-25T17:39:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:15.922+0000 7fb6b0005640 1 -- 172.21.15.98:0/3283131590 wait complete. 2024-04-25T17:39:15.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:15 smithi098 ceph-mon[29192]: pgmap v103: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 MiB/s wr, 1.81k op/s 2024-04-25T17:39:15.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:15 smithi098 ceph-mon[29192]: daemon mds.c restarted 2024-04-25T17:39:15.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:15 smithi098 ceph-mon[29192]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T17:39:15.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:15 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.203:6836/1401608926,v1:172.21.15.203:6837/1401608926] up:boot 2024-04-25T17:39:15.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:15 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:39:15.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:15 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:39:16.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:15 smithi173 ceph-mon[32856]: pgmap v103: 97 pgs: 97 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 21 MiB/s wr, 1.81k op/s 2024-04-25T17:39:16.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:15 smithi173 ceph-mon[32856]: daemon mds.c restarted 2024-04-25T17:39:16.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:15 smithi173 ceph-mon[32856]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T17:39:16.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:15 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.203:6836/1401608926,v1:172.21.15.203:6837/1401608926] up:boot 2024-04-25T17:39:16.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:15 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:39:16.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:15 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:39:16.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 165.0 s, 3401.4 tps, lat 29.436 ms stddev 30.157 2024-04-25T17:39:17.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:17 smithi098 ceph-mon[29192]: pgmap v105: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 894 op/s 2024-04-25T17:39:17.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:17 smithi203 ceph-mon[32834]: pgmap v105: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 894 op/s 2024-04-25T17:39:18.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:17 smithi173 ceph-mon[32856]: pgmap v105: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 894 op/s 2024-04-25T17:39:19.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:19 smithi098 ceph-mon[29192]: pgmap v106: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 894 op/s 2024-04-25T17:39:19.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:19 smithi203 ceph-mon[32834]: pgmap v106: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 894 op/s 2024-04-25T17:39:20.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:19 smithi173 ceph-mon[32856]: pgmap v106: 97 pgs: 97 active+clean; 1.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 11 MiB/s wr, 894 op/s 2024-04-25T17:39:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 170.0 s, 3216.6 tps, lat 31.075 ms stddev 32.664 2024-04-25T17:39:21.823 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:21 smithi173 ceph-mon[32856]: pgmap v107: 97 pgs: 97 active+clean; 1.8 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 898 op/s 2024-04-25T17:39:21.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:21 smithi098 ceph-mon[29192]: pgmap v107: 97 pgs: 97 active+clean; 1.8 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 898 op/s 2024-04-25T17:39:21.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:21 smithi203 ceph-mon[32834]: pgmap v107: 97 pgs: 97 active+clean; 1.8 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 898 op/s 2024-04-25T17:39:23.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:23 smithi098 ceph-mon[29192]: pgmap v108: 97 pgs: 97 active+clean; 1.8 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 886 op/s 2024-04-25T17:39:23.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:23 smithi203 ceph-mon[32834]: pgmap v108: 97 pgs: 97 active+clean; 1.8 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 886 op/s 2024-04-25T17:39:24.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:23 smithi173 ceph-mon[32856]: pgmap v108: 97 pgs: 97 active+clean; 1.8 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 886 op/s 2024-04-25T17:39:25.884 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:25 smithi203 ceph-mon[32834]: pgmap v109: 97 pgs: 97 active+clean; 1.8 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 886 op/s 2024-04-25T17:39:25.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:25 smithi098 ceph-mon[29192]: pgmap v109: 97 pgs: 97 active+clean; 1.8 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 886 op/s 2024-04-25T17:39:26.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:25 smithi173 ceph-mon[32856]: pgmap v109: 97 pgs: 97 active+clean; 1.8 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 886 op/s 2024-04-25T17:39:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 175.0 s, 3316.2 tps, lat 30.107 ms stddev 31.133 2024-04-25T17:39:27.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:27 smithi098 ceph-mon[29192]: pgmap v110: 97 pgs: 97 active+clean; 1.8 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 14 MiB/s wr, 1.10k op/s 2024-04-25T17:39:27.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:27 smithi203 ceph-mon[32834]: pgmap v110: 97 pgs: 97 active+clean; 1.8 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 14 MiB/s wr, 1.10k op/s 2024-04-25T17:39:28.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:27 smithi173 ceph-mon[32856]: pgmap v110: 97 pgs: 97 active+clean; 1.8 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 14 MiB/s wr, 1.10k op/s 2024-04-25T17:39:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.042+0000 7f4d80479640 1 -- 172.21.15.98:0/1191918646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d78108f40 msgr2=0x7f4d78115400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:29.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.042+0000 7f4d80479640 1 --2- 172.21.15.98:0/1191918646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d78108f40 0x7f4d78115400 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4d70009ca0 tx=0x7f4d7002a8c0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.042+0000 7f4d80479640 1 -- 172.21.15.98:0/1191918646 shutdown_connections 2024-04-25T17:39:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.042+0000 7f4d80479640 1 --2- 172.21.15.98:0/1191918646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d78108f40 0x7f4d78115400 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.042+0000 7f4d80479640 1 --2- 172.21.15.98:0/1191918646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d78108580 0x7f4d78108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.042+0000 7f4d80479640 1 --2- 172.21.15.98:0/1191918646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d7810e640 0x7f4d7810ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.042+0000 7f4d80479640 1 -- 172.21.15.98:0/1191918646 >> 172.21.15.98:0/1191918646 conn(0x7f4d78076500 msgr2=0x7f4d78078920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.042+0000 7f4d80479640 1 -- 172.21.15.98:0/1191918646 shutdown_connections 2024-04-25T17:39:29.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.042+0000 7f4d80479640 1 -- 172.21.15.98:0/1191918646 wait complete. 2024-04-25T17:39:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d80479640 1 Processor -- start 2024-04-25T17:39:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d80479640 1 -- start start 2024-04-25T17:39:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d80479640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d78108580 0x7f4d781a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d80479640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d78108f40 0x7f4d781a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d80479640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d7810e640 0x7f4d7819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d80479640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d78118820 con 0x7f4d7810e640 2024-04-25T17:39:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d80479640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d781186a0 con 0x7f4d78108f40 2024-04-25T17:39:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d80479640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d781189a0 con 0x7f4d78108580 2024-04-25T17:39:29.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d7e215640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d78108580 0x7f4d781a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d7e215640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d78108580 0x7f4d781a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:44380/0 (socket says 172.21.15.98:44380) 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d7e215640 1 -- 172.21.15.98:0/540866307 learned_addr learned my addr 172.21.15.98:0/540866307 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d7ea16640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d7810e640 0x7f4d7819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.043+0000 7f4d7da14640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d78108f40 0x7f4d781a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.044+0000 7f4d7e215640 1 -- 172.21.15.98:0/540866307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d78108f40 msgr2=0x7f4d781a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.044+0000 7f4d7e215640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d78108f40 0x7f4d781a4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.044+0000 7f4d7e215640 1 -- 172.21.15.98:0/540866307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d7810e640 msgr2=0x7f4d7819e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.044+0000 7f4d7e215640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d7810e640 0x7f4d7819e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.044+0000 7f4d7e215640 1 -- 172.21.15.98:0/540866307 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7819eff0 con 0x7f4d78108580 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.044+0000 7f4d7e215640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d78108580 0x7f4d781a46b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400e9f0 tx=0x7f4d7400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.045+0000 7f4d6f7fe640 1 -- 172.21.15.98:0/540866307 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7400cd80 con 0x7f4d78108580 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.045+0000 7f4d6f7fe640 1 -- 172.21.15.98:0/540866307 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d740054d0 con 0x7f4d78108580 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.045+0000 7f4d6f7fe640 1 -- 172.21.15.98:0/540866307 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74010690 con 0x7f4d78108580 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.045+0000 7f4d80479640 1 -- 172.21.15.98:0/540866307 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7819f2e0 con 0x7f4d78108580 2024-04-25T17:39:29.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.045+0000 7f4d80479640 1 -- 172.21.15.98:0/540866307 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d781ab500 con 0x7f4d78108580 2024-04-25T17:39:29.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.049+0000 7f4d8318e740 1 Processor -- start 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.049+0000 7f4d8318e740 1 -- start start 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3cbf40 0x55853e3d62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3d6800 0x55853e3d8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d8318e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55853e330970 con 0x55853e3cb620 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d8318e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55853e2cd4a0 con 0x55853e3cbf40 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d8318e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55853e32c930 con 0x55853e3d6800 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d7da14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3d6800 0x55853e3d8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d7ea16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d6f7fe640 1 -- 172.21.15.98:0/540866307 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4d74004e90 con 0x7f4d78108580 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d7da14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3d6800 0x55853e3d8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:44392/0 (socket says 172.21.15.98:44392) 2024-04-25T17:39:29.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d7da14640 1 -- 172.21.15.98:0/1741442313 learned_addr learned my addr 172.21.15.98:0/1741442313 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d7e215640 1 --2- 172.21.15.98:0/1741442313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3cbf40 0x55853e3d62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.050+0000 7f4d7da14640 1 -- 172.21.15.98:0/1741442313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3cbf40 msgr2=0x55853e3d62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:29.051 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.051+0000 7f4d6f7fe640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d640a3e40 0x7f4d640a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.051+0000 7f4d7da14640 1 --2- 172.21.15.98:0/1741442313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3cbf40 0x55853e3d62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.051+0000 7f4d7da14640 1 -- 172.21.15.98:0/1741442313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 msgr2=0x55853e3cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.051+0000 7f4d7da14640 1 --2- 172.21.15.98:0/1741442313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.051+0000 7f4d7da14640 1 -- 172.21.15.98:0/1741442313 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55853e043210 con 0x55853e3d6800 2024-04-25T17:39:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.051+0000 7f4d6f7fe640 1 -- 172.21.15.98:0/540866307 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f4d740d0b50 con 0x7f4d78108580 2024-04-25T17:39:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.051+0000 7f4d7ea16640 1 --2- 172.21.15.98:0/1741442313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.051+0000 7f4d7e215640 1 --2- 172.21.15.98:0/1741442313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3cbf40 0x55853e3d62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:39:29.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.052+0000 7f4d7ea16640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d640a3e40 0x7f4d640a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.052+0000 7f4d7da14640 1 --2- 172.21.15.98:0/1741442313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3d6800 0x55853e3d8bf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009c60 tx=0x7f4d68011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20f2c45540a4e4aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:29.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.052+0000 7f4d6cbfe640 1 -- 172.21.15.98:0/1741442313 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68004230 con 0x55853e3d6800 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.052+0000 7f4d6cbfe640 1 -- 172.21.15.98:0/1741442313 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d680043d0 con 0x55853e3d6800 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.052+0000 7f4d7ea16640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d640a3e40 0x7f4d640a6300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4d70005fa0 tx=0x7f4d70005fd0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.052+0000 7f4d6cbfe640 1 -- 172.21.15.98:0/1741442313 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68004c60 con 0x55853e3d6800 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 -- 172.21.15.98:0/1741442313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3d6800 msgr2=0x55853e3d8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 --2- 172.21.15.98:0/1741442313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3d6800 0x55853e3d8bf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009c60 tx=0x7f4d68011810 comp rx=0 tx=0).stop 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 -- 172.21.15.98:0/1741442313 shutdown_connections 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 --2- 172.21.15.98:0/1741442313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3d6800 0x55853e3d8bf0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 --2- 172.21.15.98:0/1741442313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3cbf40 0x55853e3d62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 --2- 172.21.15.98:0/1741442313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 -- 172.21.15.98:0/1741442313 >> 172.21.15.98:0/1741442313 conn(0x55853e3c7b30 msgr2=0x55853e3c7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 -- 172.21.15.98:0/1741442313 shutdown_connections 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 -- 172.21.15.98:0/1741442313 wait complete. 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 -- start start 2024-04-25T17:39:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.053+0000 7f4d8318e740 1 Processor -- start 2024-04-25T17:39:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.054+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3cb620 0x55853e3d6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.054+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.054+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3d6800 0x55853e3d3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.054+0000 7f4d7da14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.054+0000 7f4d7da14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:44400/0 (socket says 172.21.15.98:44400) 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.054+0000 7f4d7da14640 1 -- 172.21.15.98:0/3145627548 learned_addr learned my addr 172.21.15.98:0/3145627548 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.054+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55853e3d9430 con 0x55853e3d6800 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.054+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55853e2cb0b0 con 0x55853e3cb620 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.054+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55853e3d95b0 con 0x55853e3cbf40 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d7ea16640 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3d6800 0x55853e3d3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d7da14640 1 -- 172.21.15.98:0/3145627548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3cb620 msgr2=0x55853e3d6030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d7da14640 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3cb620 0x55853e3d6030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d7da14640 1 -- 172.21.15.98:0/3145627548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3d6800 msgr2=0x55853e3d3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d7da14640 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3d6800 0x55853e3d3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d7da14640 1 -- 172.21.15.98:0/3145627548 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55853e10f620 con 0x55853e3cbf40 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d8318e740 10 client.0 _ll_register_callbacks cb 0x55853e2b8d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d7da14640 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d3320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4d68000c00 tx=0x7f4d68019b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2157dd4999a60b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:39:29.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d61ffb640 1 -- 172.21.15.98:0/3145627548 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68028020 con 0x55853e3cbf40 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d61ffb640 1 -- 172.21.15.98:0/3145627548 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f4d68024620 con 0x55853e3cbf40 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.055+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d680043d0 con 0x55853e3cbf40 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.056+0000 7f4d61ffb640 1 -- 172.21.15.98:0/3145627548 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4d68033050 con 0x55853e3cbf40 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.056+0000 7f4d8318e740 10 client.25222 fetch_fsmap learned FSMap version 31 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.056+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55853e3e5040 con 0x55853e3cbf40 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.056+0000 7f4d61ffb640 1 -- 172.21.15.98:0/3145627548 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f4d68018070 con 0x55853e3cbf40 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.056+0000 7f4d8318e740 10 client.25222 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.056+0000 7f4d8318e740 10 client.25222 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.056+0000 7f4d8318e740 20 client.25222 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.056+0000 7f4d8318e740 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55853e3e6390 0x55853e3e8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.056+0000 7f4d8318e740 4 client.25222 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.056+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4d680043d0 con 0x55853e3e6390 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.057+0000 7f4d7e215640 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55853e3e6390 0x55853e3e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.057+0000 7f4d7e215640 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55853e3e6390 0x55853e3e8780 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6193a5cbc286bba8 server_cookie=9752818bf6267d3b in_seq=0 out_seq=0 2024-04-25T17:39:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.058+0000 7f4d61ffb640 10 client.25222 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.058+0000 7f4d61ffb640 1 -- 172.21.15.98:0/3145627548 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f4d680043d0 con 0x55853e3e6390 2024-04-25T17:39:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.058+0000 7f4d61ffb640 10 client.25222 handle_command_reply: tid=0 2024-04-25T17:39:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.058+0000 7f4d8318e740 1 client.25222 shutdown 2024-04-25T17:39:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 msgr2=0x55853e3d3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d8318e740 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d3320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4d68000c00 tx=0x7f4d68019b70 comp rx=0 tx=0).stop 2024-04-25T17:39:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 shutdown_connections 2024-04-25T17:39:29.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d8318e740 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55853e3e6390 0x55853e3e8780 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d8318e740 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3d6800 0x55853e3d3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d61ffb640 0 client.25222 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d8318e740 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d3320 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d8318e740 1 --2- 172.21.15.98:0/3145627548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3cb620 0x55853e3d6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 >> 172.21.15.98:0/3145627548 conn(0x55853e3c7b30 msgr2=0x55853e3cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 shutdown_connections 2024-04-25T17:39:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.059+0000 7f4d8318e740 1 -- 172.21.15.98:0/3145627548 wait complete. 2024-04-25T17:39:29.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.060+0000 7f4d8318e740 20 client.25222 trim_cache size 0 max 16384 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.080+0000 7f4d8318e740 1 Processor -- start 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.080+0000 7f4d8318e740 1 -- start start 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.080+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3e22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.080+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.080+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 0x55853e3edfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.080+0000 7f4d8318e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d781b67c0 con 0x55853e3cb620 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.080+0000 7f4d8318e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d781b3fd0 con 0x55853e3d6800 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.080+0000 7f4d8318e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d781b1e20 con 0x55853e3cbf40 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.080+0000 7f4d7e215640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3e22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7e215640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3e22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37002/0 (socket says 172.21.15.98:37002) 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7da14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7e215640 1 -- 172.21.15.98:0/1241315385 learned_addr learned my addr 172.21.15.98:0/1241315385 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7ea16640 1 --2- 172.21.15.98:0/1241315385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 0x55853e3edfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7da14640 1 -- 172.21.15.98:0/1241315385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 msgr2=0x55853e3edfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7da14640 1 --2- 172.21.15.98:0/1241315385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 0x55853e3edfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7da14640 1 -- 172.21.15.98:0/1241315385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 msgr2=0x55853e3e22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7da14640 1 --2- 172.21.15.98:0/1241315385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3e22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7da14640 1 -- 172.21.15.98:0/1241315385 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55853e3cbd70 con 0x55853e3cbf40 2024-04-25T17:39:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7e215640 1 --2- 172.21.15.98:0/1241315385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3e22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d7da14640 1 --2- 172.21.15.98:0/1241315385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3c6b30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55853e3d2b80 tx=0x7f4d6801e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d64807d0cced396 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d63fff640 1 -- 172.21.15.98:0/1241315385 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d680097a0 con 0x55853e3cbf40 2024-04-25T17:39:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d63fff640 1 -- 172.21.15.98:0/1241315385 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d6801f040 con 0x55853e3cbf40 2024-04-25T17:39:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.081+0000 7f4d63fff640 1 -- 172.21.15.98:0/1241315385 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6802d3f0 con 0x55853e3cbf40 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.082+0000 7f4d8318e740 1 -- 172.21.15.98:0/1241315385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 msgr2=0x55853e3c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.082+0000 7f4d8318e740 1 --2- 172.21.15.98:0/1241315385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3c6b30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55853e3d2b80 tx=0x7f4d6801e000 comp rx=0 tx=0).stop 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.082+0000 7f4d8318e740 1 -- 172.21.15.98:0/1241315385 shutdown_connections 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.082+0000 7f4d8318e740 1 --2- 172.21.15.98:0/1241315385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 0x55853e3edfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.082+0000 7f4d8318e740 1 --2- 172.21.15.98:0/1241315385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3c6b30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.082+0000 7f4d8318e740 1 --2- 172.21.15.98:0/1241315385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3e22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.082+0000 7f4d8318e740 1 -- 172.21.15.98:0/1241315385 >> 172.21.15.98:0/1241315385 conn(0x55853e3c7b30 msgr2=0x55853e3c6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.083+0000 7f4d8318e740 1 -- 172.21.15.98:0/1241315385 shutdown_connections 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.083+0000 7f4d8318e740 1 -- 172.21.15.98:0/1241315385 wait complete. 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d8318e740 1 -- start start 2024-04-25T17:39:29.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d8318e740 1 Processor -- start 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3de940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d8318e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 0x55853e3da300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d8318e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d781b2490 con 0x55853e3cb620 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d7e215640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3de940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d7da14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d7e215640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3de940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37008/0 (socket says 172.21.15.98:37008) 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d7e215640 1 -- 172.21.15.98:0/251079863 learned_addr learned my addr 172.21.15.98:0/251079863 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d7da14640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:44404/0 (socket says 172.21.15.98:44404) 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.084+0000 7f4d7ea16640 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 0x55853e3da300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d8318e740 1 -- 172.21.15.98:0/251079863 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d781b3900 con 0x55853e3d6800 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d8318e740 1 -- 172.21.15.98:0/251079863 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d781b2b60 con 0x55853e3cbf40 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d8318e740 10 client.0 _ll_register_callbacks cb 0x7f4d68019800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d7e215640 1 -- 172.21.15.98:0/251079863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 msgr2=0x55853e3d9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d7e215640 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d7e215640 1 -- 172.21.15.98:0/251079863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 msgr2=0x55853e3da300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d7e215640 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 0x55853e3da300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d7e215640 1 -- 172.21.15.98:0/251079863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55853e3da870 con 0x55853e3cb620 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d7ea16640 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 0x55853e3da300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d7e215640 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3de940 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4d74098f80 tx=0x7f4d7409c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de7cca382b9948f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d61ffb640 1 -- 172.21.15.98:0/251079863 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d740b6060 con 0x55853e3cb620 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d61ffb640 1 -- 172.21.15.98:0/251079863 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f4d740046c0 con 0x55853e3cb620 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d8318e740 1 -- 172.21.15.98:0/251079863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d6801f040 con 0x55853e3cb620 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d61ffb640 1 -- 172.21.15.98:0/251079863 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4d7409cb20 con 0x55853e3cb620 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d8318e740 10 client.15702 fetch_fsmap learned FSMap version 31 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.085+0000 7f4d8318e740 1 -- 172.21.15.98:0/251079863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55853e44c5b0 con 0x55853e3cb620 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d61ffb640 1 -- 172.21.15.98:0/251079863 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d6801f040 con 0x55853e3cb620 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d61ffb640 1 -- 172.21.15.98:0/251079863 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f4d740bbce0 con 0x55853e3cb620 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d8318e740 10 client.15702 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:39:29.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d8318e740 10 client.15702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:39:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d8318e740 20 client.15702 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d8318e740 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55853e3e6390 0x55853e44ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d8318e740 4 client.15702 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:39:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d8318e740 1 -- 172.21.15.98:0/251079863 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55853e33e390 con 0x55853e3e6390 2024-04-25T17:39:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d7da14640 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55853e3e6390 0x55853e44ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d61ffb640 10 client.15702 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.086+0000 7f4d7da14640 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55853e3e6390 0x55853e44ecc0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58e1ce653de4fb9c server_cookie=4d997ab8782b3907 in_seq=0 out_seq=0 2024-04-25T17:39:29.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d61ffb640 1 -- 172.21.15.98:0/251079863 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55853e33e390 con 0x55853e3e6390 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d61ffb640 10 client.15702 handle_command_reply: tid=0 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 client.15702 shutdown 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 -- 172.21.15.98:0/251079863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 msgr2=0x55853e3de940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3de940 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4d74098f80 tx=0x7f4d7409c540 comp rx=0 tx=0).stop 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 -- 172.21.15.98:0/251079863 shutdown_connections 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55853e3e6390 0x55853e44ecc0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55853e3d6800 0x55853e3da300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d61ffb640 0 client.15702 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55853e3cbf40 0x55853e3d9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 --2- 172.21.15.98:0/251079863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55853e3cb620 0x55853e3de940 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 -- 172.21.15.98:0/251079863 >> 172.21.15.98:0/251079863 conn(0x55853e3c7b30 msgr2=0x55853e3c6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 -- 172.21.15.98:0/251079863 shutdown_connections 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 1 -- 172.21.15.98:0/251079863 wait complete. 2024-04-25T17:39:29.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.101+0000 7f4d8318e740 20 client.15702 trim_cache size 0 max 16384 2024-04-25T17:39:29.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:39:29.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:39:29.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:39:29.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:39:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 -- 172.21.15.98:0/540866307 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d640a3e40 msgr2=0x7f4d640a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d640a3e40 0x7f4d640a6300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4d70005fa0 tx=0x7f4d70005fd0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 -- 172.21.15.98:0/540866307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d78108580 msgr2=0x7f4d781a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d78108580 0x7f4d781a46b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400e9f0 tx=0x7f4d7400edf0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 -- 172.21.15.98:0/540866307 shutdown_connections 2024-04-25T17:39:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d640a3e40 0x7f4d640a6300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d7810e640 0x7f4d7819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d78108f40 0x7f4d781a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 --2- 172.21.15.98:0/540866307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d78108580 0x7f4d781a46b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 -- 172.21.15.98:0/540866307 >> 172.21.15.98:0/540866307 conn(0x7f4d78076500 msgr2=0x7f4d7810f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 -- 172.21.15.98:0/540866307 shutdown_connections 2024-04-25T17:39:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:29.102+0000 7f4d80479640 1 -- 172.21.15.98:0/540866307 wait complete. 2024-04-25T17:39:29.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:29 smithi098 ceph-mon[29192]: pgmap v111: 97 pgs: 97 active+clean; 1.8 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.3 MiB/s wr, 615 op/s 2024-04-25T17:39:29.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:29 smithi203 ceph-mon[32834]: pgmap v111: 97 pgs: 97 active+clean; 1.8 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.3 MiB/s wr, 615 op/s 2024-04-25T17:39:30.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:29 smithi173 ceph-mon[32856]: pgmap v111: 97 pgs: 97 active+clean; 1.8 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.3 MiB/s wr, 615 op/s 2024-04-25T17:39:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 180.0 s, 3303.4 tps, lat 30.340 ms stddev 31.644 2024-04-25T17:39:31.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:31 smithi098 ceph-mon[29192]: pgmap v112: 97 pgs: 97 active+clean; 1.8 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.6 MiB/s wr, 659 op/s 2024-04-25T17:39:31.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:31 smithi203 ceph-mon[32834]: pgmap v112: 97 pgs: 97 active+clean; 1.8 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.6 MiB/s wr, 659 op/s 2024-04-25T17:39:32.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:31 smithi173 ceph-mon[32856]: pgmap v112: 97 pgs: 97 active+clean; 1.8 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.6 MiB/s wr, 659 op/s 2024-04-25T17:39:33.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:33 smithi098 ceph-mon[29192]: pgmap v113: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 10 MiB/s wr, 875 op/s 2024-04-25T17:39:33.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:33 smithi203 ceph-mon[32834]: pgmap v113: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 10 MiB/s wr, 875 op/s 2024-04-25T17:39:34.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:33 smithi173 ceph-mon[32856]: pgmap v113: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 10 MiB/s wr, 875 op/s 2024-04-25T17:39:35.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:35 smithi098 ceph-mon[29192]: pgmap v114: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.3 MiB/s wr, 610 op/s 2024-04-25T17:39:35.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:35 smithi203 ceph-mon[32834]: pgmap v114: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.3 MiB/s wr, 610 op/s 2024-04-25T17:39:36.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:35 smithi173 ceph-mon[32856]: pgmap v114: 97 pgs: 97 active+clean; 1.8 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.3 MiB/s wr, 610 op/s 2024-04-25T17:39:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 185.0 s, 3054.6 tps, lat 32.654 ms stddev 35.927 2024-04-25T17:39:37.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:37 smithi098 ceph-mon[29192]: pgmap v115: 97 pgs: 97 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 916 op/s 2024-04-25T17:39:37.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:37 smithi203 ceph-mon[32834]: pgmap v115: 97 pgs: 97 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 916 op/s 2024-04-25T17:39:38.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:37 smithi173 ceph-mon[32856]: pgmap v115: 97 pgs: 97 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 12 MiB/s wr, 916 op/s 2024-04-25T17:39:39.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:39 smithi098 ceph-mon[29192]: pgmap v116: 97 pgs: 97 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.4 MiB/s wr, 615 op/s 2024-04-25T17:39:39.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:39 smithi203 ceph-mon[32834]: pgmap v116: 97 pgs: 97 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.4 MiB/s wr, 615 op/s 2024-04-25T17:39:40.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:39 smithi173 ceph-mon[32856]: pgmap v116: 97 pgs: 97 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.4 MiB/s wr, 615 op/s 2024-04-25T17:39:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 190.0 s, 3293.4 tps, lat 30.354 ms stddev 31.297 2024-04-25T17:39:41.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:41 smithi098 ceph-mon[29192]: pgmap v117: 97 pgs: 97 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 8.4 MiB/s wr, 714 op/s 2024-04-25T17:39:41.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:41 smithi203 ceph-mon[32834]: pgmap v117: 97 pgs: 97 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 8.4 MiB/s wr, 714 op/s 2024-04-25T17:39:42.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:41 smithi173 ceph-mon[32856]: pgmap v117: 97 pgs: 97 active+clean; 1.9 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 8.4 MiB/s wr, 714 op/s 2024-04-25T17:39:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.230+0000 7f4d7538e640 1 -- 172.21.15.98:0/3725173041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d70106680 msgr2=0x7f4d70106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.230+0000 7f4d7538e640 1 --2- 172.21.15.98:0/3725173041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d70106680 0x7f4d70106a60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400aa10 tx=0x7f4d6402b760 comp rx=0 tx=0).stop 2024-04-25T17:39:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.230+0000 7f4d7538e640 1 -- 172.21.15.98:0/3725173041 shutdown_connections 2024-04-25T17:39:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.230+0000 7f4d7538e640 1 --2- 172.21.15.98:0/3725173041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d70100f80 0x7f4d7010d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.230+0000 7f4d7538e640 1 --2- 172.21.15.98:0/3725173041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d701005c0 0x7f4d70100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.230+0000 7f4d7538e640 1 --2- 172.21.15.98:0/3725173041 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d70106680 0x7f4d70106a60 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.230+0000 7f4d7538e640 1 -- 172.21.15.98:0/3725173041 >> 172.21.15.98:0/3725173041 conn(0x7f4d700773e0 msgr2=0x7f4d700fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.234+0000 7f4d7538e640 1 -- 172.21.15.98:0/3725173041 shutdown_connections 2024-04-25T17:39:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.235+0000 7f4d7538e640 1 -- 172.21.15.98:0/3725173041 wait complete. 2024-04-25T17:39:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.236+0000 7f4d7538e640 1 Processor -- start 2024-04-25T17:39:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.237+0000 7f4d7538e640 1 -- start start 2024-04-25T17:39:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.238+0000 7f4d7538e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d701005c0 0x7f4d70073660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.238+0000 7f4d7538e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d70100f80 0x7f4d70073ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.238+0000 7f4d7538e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d70106680 0x7f4d7006c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.238+0000 7f4d7538e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7010fe30 con 0x7f4d70100f80 2024-04-25T17:39:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.238+0000 7f4d7538e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7010fcb0 con 0x7f4d701005c0 2024-04-25T17:39:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.238+0000 7f4d7538e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7010ffb0 con 0x7f4d70106680 2024-04-25T17:39:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.238+0000 7f4d6f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d70106680 0x7f4d7006c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.238+0000 7f4d6e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d70100f80 0x7f4d70073ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.239+0000 7f4d6f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d70106680 0x7f4d7006c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:59648/0 (socket says 172.21.15.98:59648) 2024-04-25T17:39:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.239+0000 7f4d6e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d70100f80 0x7f4d70073ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34460/0 (socket says 172.21.15.98:34460) 2024-04-25T17:39:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.239+0000 7f4d6e7fc640 1 -- 172.21.15.98:0/1214183355 learned_addr learned my addr 172.21.15.98:0/1214183355 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.240+0000 7f4d6e7fc640 1 -- 172.21.15.98:0/1214183355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d70106680 msgr2=0x7f4d7006c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.240+0000 7f4d6e7fc640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d70106680 0x7f4d7006c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.240+0000 7f4d6e7fc640 1 -- 172.21.15.98:0/1214183355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d701005c0 msgr2=0x7f4d70073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.240+0000 7f4d6effd640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d701005c0 0x7f4d70073660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.240+0000 7f4d6e7fc640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d701005c0 0x7f4d70073660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.240+0000 7f4d6e7fc640 1 -- 172.21.15.98:0/1214183355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7006d180 con 0x7f4d70100f80 2024-04-25T17:39:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.240+0000 7f4d6f7fe640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d70106680 0x7f4d7006c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.240+0000 7f4d6e7fc640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d70100f80 0x7f4d70073ba0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800ba50 tx=0x7f4d6800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.242+0000 7f4d57fff640 1 -- 172.21.15.98:0/1214183355 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68019070 con 0x7f4d70100f80 2024-04-25T17:39:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.242+0000 7f4d57fff640 1 -- 172.21.15.98:0/1214183355 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d6800cbe0 con 0x7f4d70100f80 2024-04-25T17:39:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.242+0000 7f4d7538e640 1 -- 172.21.15.98:0/1214183355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7006d470 con 0x7f4d70100f80 2024-04-25T17:39:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.242+0000 7f4d7538e640 1 -- 172.21.15.98:0/1214183355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d700713b0 con 0x7f4d70100f80 2024-04-25T17:39:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.242+0000 7f4d57fff640 1 -- 172.21.15.98:0/1214183355 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d680075d0 con 0x7f4d70100f80 2024-04-25T17:39:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.245+0000 7f4d57fff640 1 -- 172.21.15.98:0/1214183355 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4d6801c070 con 0x7f4d70100f80 2024-04-25T17:39:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.245+0000 7f4d57fff640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d580a3f20 0x7f4d580a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.245+0000 7f4d7817e740 1 Processor -- start 2024-04-25T17:39:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.245+0000 7f4d7817e740 1 -- start start 2024-04-25T17:39:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.245+0000 7f4d57fff640 1 -- 172.21.15.98:0/1214183355 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f4d680cfb70 con 0x7f4d70100f80 2024-04-25T17:39:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfaa620 0x56007dfaaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfb52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfb5800 0x56007dfb7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d7817e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56007df0f970 con 0x56007dfaa620 2024-04-25T17:39:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d7817e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56007deac4a0 con 0x56007dfaaf40 2024-04-25T17:39:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d7817e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56007df0b930 con 0x56007dfb5800 2024-04-25T17:39:42.247 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d6e7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfb52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d6e7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfb52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52394/0 (socket says 172.21.15.98:52394) 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d6e7fc640 1 -- 172.21.15.98:0/1785360394 learned_addr learned my addr 172.21.15.98:0/1785360394 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d6f7fe640 1 --2- 172.21.15.98:0/1785360394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfb5800 0x56007dfb7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d6e7fc640 1 -- 172.21.15.98:0/1785360394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfb5800 msgr2=0x56007dfb7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d6e7fc640 1 --2- 172.21.15.98:0/1785360394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfb5800 0x56007dfb7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d6e7fc640 1 -- 172.21.15.98:0/1785360394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfaa620 msgr2=0x56007dfaaa00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d6e7fc640 1 --2- 172.21.15.98:0/1785360394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfaa620 0x56007dfaaa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.246+0000 7f4d6e7fc640 1 -- 172.21.15.98:0/1785360394 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56007dc22210 con 0x56007dfaaf40 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.247+0000 7f4d6effd640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d580a3f20 0x7f4d580a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.247+0000 7f4d6e7fc640 1 --2- 172.21.15.98:0/1785360394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfb52c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d680986d0 tx=0x7f4d68014a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b692e7bbf324835 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.247+0000 7f4d557fa640 1 -- 172.21.15.98:0/1785360394 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d680a0000 con 0x56007dfaaf40 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.247+0000 7f4d557fa640 1 -- 172.21.15.98:0/1785360394 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d680a48c0 con 0x56007dfaaf40 2024-04-25T17:39:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.247+0000 7f4d557fa640 1 -- 172.21.15.98:0/1785360394 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d680bc2d0 con 0x56007dfaaf40 2024-04-25T17:39:42.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.248+0000 7f4d7817e740 1 -- 172.21.15.98:0/1785360394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 msgr2=0x56007dfb52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:42.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.248+0000 7f4d7817e740 1 --2- 172.21.15.98:0/1785360394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfb52c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d680986d0 tx=0x7f4d68014a80 comp rx=0 tx=0).stop 2024-04-25T17:39:42.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.248+0000 7f4d6effd640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d580a3f20 0x7f4d580a63e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d64005ec0 tx=0x7f4d64005d10 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:42.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.248+0000 7f4d7817e740 1 -- 172.21.15.98:0/1785360394 shutdown_connections 2024-04-25T17:39:42.250 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.248+0000 7f4d7817e740 1 --2- 172.21.15.98:0/1785360394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfb5800 0x56007dfb7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.248+0000 7f4d7817e740 1 --2- 172.21.15.98:0/1785360394 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfb52c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.248+0000 7f4d7817e740 1 --2- 172.21.15.98:0/1785360394 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfaa620 0x56007dfaaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.248+0000 7f4d7817e740 1 -- 172.21.15.98:0/1785360394 >> 172.21.15.98:0/1785360394 conn(0x56007dfa6b30 msgr2=0x56007dfa6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.248+0000 7f4d7817e740 1 -- 172.21.15.98:0/1785360394 shutdown_connections 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.248+0000 7f4d7817e740 1 -- 172.21.15.98:0/1785360394 wait complete. 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d7817e740 1 -- start start 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d7817e740 1 Processor -- start 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfaaf40 0x56007dfb2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfb5800 0x56007dfb2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d7817e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56007dfb83d0 con 0x56007dfaaf40 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d7817e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56007deaa0b0 con 0x56007dfb5800 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d7817e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56007dfb8550 con 0x56007dfaa620 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d6effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d6effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:59654/0 (socket says 172.21.15.98:59654) 2024-04-25T17:39:42.251 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.249+0000 7f4d6effd640 1 -- 172.21.15.98:0/3072969491 learned_addr learned my addr 172.21.15.98:0/3072969491 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:42.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.250+0000 7f4d7817e740 10 client.0 _ll_register_callbacks cb 0x56007de97d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:42.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.250+0000 7f4d6f7fe640 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfb5800 0x56007dfb2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.250+0000 7f4d6effd640 1 -- 172.21.15.98:0/3072969491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfb5800 msgr2=0x56007dfb2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:42.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.250+0000 7f4d6effd640 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfb5800 0x56007dfb2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.250+0000 7f4d6effd640 1 -- 172.21.15.98:0/3072969491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfaaf40 msgr2=0x56007dfb2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:42.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.250+0000 7f4d6effd640 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfaaf40 0x56007dfb2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.250+0000 7f4d6effd640 1 -- 172.21.15.98:0/3072969491 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56007dcee620 con 0x56007dfaa620 2024-04-25T17:39:42.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.250+0000 7f4d6f7fe640 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfb5800 0x56007dfb2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:39:42.252 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.250+0000 7f4d6effd640 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb5110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4d64004560 tx=0x7f4d6400a370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43702ae8e462ac7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.251+0000 7f4d52ff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:39:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.251+0000 7f4d52ff5640 1 -- 172.21.15.98:0/3072969491 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d64042070 con 0x56007dfaa620 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.251+0000 7f4d7817e740 1 -- 172.21.15.98:0/3072969491 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d680a48c0 con 0x56007dfaa620 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.251+0000 7f4d52ff5640 1 -- 172.21.15.98:0/3072969491 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f4d64030bd0 con 0x56007dfaa620 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.251+0000 7f4d52ff5640 1 -- 172.21.15.98:0/3072969491 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4d64057050 con 0x56007dfaa620 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.251+0000 7f4d7817e740 10 client.25237 fetch_fsmap learned FSMap version 31 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.251+0000 7f4d7817e740 1 -- 172.21.15.98:0/3072969491 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56007dfc4320 con 0x56007dfaa620 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.251+0000 7f4d52ff5640 1 -- 172.21.15.98:0/3072969491 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d680a48c0 con 0x56007dfaa620 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.252+0000 7f4d52ff5640 1 -- 172.21.15.98:0/3072969491 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f4d6400b040 con 0x56007dfaa620 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.252+0000 7f4d7817e740 10 client.25237 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.252+0000 7f4d7817e740 10 client.25237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.252+0000 7f4d7817e740 20 client.25237 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.252+0000 7f4d7817e740 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56007dfc5d00 0x56007dfc81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.252+0000 7f4d7817e740 4 client.25237 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.252+0000 7f4d7817e740 1 -- 172.21.15.98:0/3072969491 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56007df1d390 con 0x56007dfc5d00 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.252+0000 7f4d6e7fc640 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56007dfc5d00 0x56007dfc81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.255 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.252+0000 7f4d6e7fc640 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56007dfc5d00 0x56007dfc81c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8891f7662c07c6a server_cookie=8b824c5580bc2bdc in_seq=0 out_seq=0 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.252+0000 7f4d52ff5640 10 client.25237 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d52ff5640 1 -- 172.21.15.98:0/3072969491 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56007df1d390 con 0x56007dfc5d00 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d52ff5640 10 client.25237 handle_command_reply: tid=0 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 client.25237 shutdown 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 -- 172.21.15.98:0/3072969491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 msgr2=0x56007dfb5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb5110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4d64004560 tx=0x7f4d6400a370 comp rx=0 tx=0).stop 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 -- 172.21.15.98:0/3072969491 shutdown_connections 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56007dfc5d00 0x56007dfc81c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfb5800 0x56007dfb2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d52ff5640 0 client.25237 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfaaf40 0x56007dfb2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 --2- 172.21.15.98:0/3072969491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb5110 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 -- 172.21.15.98:0/3072969491 >> 172.21.15.98:0/3072969491 conn(0x56007dfa6b30 msgr2=0x56007dfac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:42.256 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 -- 172.21.15.98:0/3072969491 shutdown_connections 2024-04-25T17:39:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 1 -- 172.21.15.98:0/3072969491 wait complete. 2024-04-25T17:39:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.253+0000 7f4d7817e740 20 client.25237 trim_cache size 0 max 16384 2024-04-25T17:39:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d7817e740 1 Processor -- start 2024-04-25T17:39:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d7817e740 1 -- start start 2024-04-25T17:39:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfcdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 0x56007dfce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d7817e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c011110 con 0x56007dfb5800 2024-04-25T17:39:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d7817e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c00e920 con 0x56007dfaaf40 2024-04-25T17:39:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d7817e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c00c770 con 0x56007dfaa620 2024-04-25T17:39:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d6effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d6e7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfcdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d6effd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:59662/0 (socket says 172.21.15.98:59662) 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 0x56007dfce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.279+0000 7f4d6effd640 1 -- 172.21.15.98:0/3788349598 learned_addr learned my addr 172.21.15.98:0/3788349598 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d6effd640 1 -- 172.21.15.98:0/3788349598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 msgr2=0x56007dfcdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d6effd640 1 --2- 172.21.15.98:0/3788349598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfcdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d6effd640 1 -- 172.21.15.98:0/3788349598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 msgr2=0x56007dfce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d6effd640 1 --2- 172.21.15.98:0/3788349598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 0x56007dfce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d6effd640 1 -- 172.21.15.98:0/3788349598 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56007dfb2ca0 con 0x56007dfaa620 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d6e7fc640 1 --2- 172.21.15.98:0/3788349598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfcdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d6f7fe640 1 --2- 172.21.15.98:0/3788349598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 0x56007dfce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d6effd640 1 --2- 172.21.15.98:0/3788349598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb4c40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d64000c00 tx=0x7f4d64005990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c82b446380840d13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d54ff9640 1 -- 172.21.15.98:0/3788349598 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6405f020 con 0x56007dfaa620 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d54ff9640 1 -- 172.21.15.98:0/3788349598 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d6404e070 con 0x56007dfaa620 2024-04-25T17:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.280+0000 7f4d54ff9640 1 -- 172.21.15.98:0/3788349598 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d64042040 con 0x56007dfaa620 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 -- 172.21.15.98:0/3788349598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 msgr2=0x56007dfb4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 --2- 172.21.15.98:0/3788349598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb4c40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4d64000c00 tx=0x7f4d64005990 comp rx=0 tx=0).stop 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 -- 172.21.15.98:0/3788349598 shutdown_connections 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 --2- 172.21.15.98:0/3788349598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 0x56007dfce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 --2- 172.21.15.98:0/3788349598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaaf40 0x56007dfcdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 --2- 172.21.15.98:0/3788349598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaa620 0x56007dfb4c40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 -- 172.21.15.98:0/3788349598 >> 172.21.15.98:0/3788349598 conn(0x56007dfa6b30 msgr2=0x56007dfa5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 -- 172.21.15.98:0/3788349598 shutdown_connections 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 -- 172.21.15.98:0/3788349598 wait complete. 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 -- start start 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 Processor -- start 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaa620 0x56007e0246b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.281+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaaf40 0x56007e024bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.282+0000 7f4d7817e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 0x56007dfbabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.282+0000 7f4d7817e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c00cde0 con 0x56007dfb5800 2024-04-25T17:39:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.282+0000 7f4d7817e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c00e250 con 0x56007dfaa620 2024-04-25T17:39:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.282+0000 7f4d6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 0x56007dfbabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.282+0000 7f4d6e7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaaf40 0x56007e024bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.282+0000 7f4d6effd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaa620 0x56007e0246b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.282+0000 7f4d6e7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaaf40 0x56007e024bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:59676/0 (socket says 172.21.15.98:59676) 2024-04-25T17:39:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.282+0000 7f4d6e7fc640 1 -- 172.21.15.98:0/2780942686 learned_addr learned my addr 172.21.15.98:0/2780942686 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.282+0000 7f4d7817e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5c00d4b0 con 0x56007dfaaf40 2024-04-25T17:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.282+0000 7f4d7817e740 10 client.0 _ll_register_callbacks cb 0x7f4d64004190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.283+0000 7f4d6f7fe640 1 -- 172.21.15.98:0/2780942686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaaf40 msgr2=0x56007e024bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.283+0000 7f4d6f7fe640 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaaf40 0x56007e024bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.283+0000 7f4d6f7fe640 1 -- 172.21.15.98:0/2780942686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaa620 msgr2=0x56007e0246b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.283+0000 7f4d6f7fe640 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaa620 0x56007e0246b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.283+0000 7f4d6f7fe640 1 -- 172.21.15.98:0/2780942686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56007dfbb290 con 0x56007dfb5800 2024-04-25T17:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.283+0000 7f4d6f7fe640 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 0x56007dfbabd0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x56007dfb57d0 tx=0x7f4d60015d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9857c682f766ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.283+0000 7f4d52ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.283+0000 7f4d52ff5640 1 -- 172.21.15.98:0/2780942686 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d60013c80 con 0x56007dfb5800 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.283+0000 7f4d7817e740 1 -- 172.21.15.98:0/2780942686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d6404e070 con 0x56007dfb5800 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.283+0000 7f4d52ff5640 1 -- 172.21.15.98:0/2780942686 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f4d60013e20 con 0x56007dfb5800 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d52ff5640 1 -- 172.21.15.98:0/2780942686 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4d60016430 con 0x56007dfb5800 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d7817e740 10 client.15714 fetch_fsmap learned FSMap version 31 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d7817e740 1 -- 172.21.15.98:0/2780942686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56007e02bed0 con 0x56007dfb5800 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d52ff5640 1 -- 172.21.15.98:0/2780942686 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d6404e070 con 0x56007dfb5800 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d52ff5640 1 -- 172.21.15.98:0/2780942686 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f4d6001e6c0 con 0x56007dfb5800 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d7817e740 10 client.15714 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d7817e740 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d7817e740 20 client.15714 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d7817e740 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56007e02c680 0x56007e02eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d7817e740 4 client.15714 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.284+0000 7f4d7817e740 1 -- 172.21.15.98:0/2780942686 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56007df17200 con 0x56007e02c680 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.285+0000 7f4d6effd640 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56007e02c680 0x56007e02eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:42.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.286+0000 7f4d6effd640 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56007e02c680 0x56007e02eaa0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e559dc1b47b57da server_cookie=69c76d5e209e6beb in_seq=0 out_seq=0 2024-04-25T17:39:42.289 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.286+0000 7f4d52ff5640 10 client.15714 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:42.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d52ff5640 1 -- 172.21.15.98:0/2780942686 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56007df17200 con 0x56007e02c680 2024-04-25T17:39:42.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d52ff5640 10 client.15714 handle_command_reply: tid=0 2024-04-25T17:39:42.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 client.15714 shutdown 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 -- 172.21.15.98:0/2780942686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 msgr2=0x56007dfbabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 0x56007dfbabd0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x56007dfb57d0 tx=0x7f4d60015d80 comp rx=0 tx=0).stop 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 -- 172.21.15.98:0/2780942686 shutdown_connections 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56007e02c680 0x56007e02eaa0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56007dfb5800 0x56007dfbabd0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56007dfaaf40 0x56007e024bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 --2- 172.21.15.98:0/2780942686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56007dfaa620 0x56007e0246b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 -- 172.21.15.98:0/2780942686 >> 172.21.15.98:0/2780942686 conn(0x56007dfa6b30 msgr2=0x56007dfb1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d52ff5640 0 client.15714 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 -- 172.21.15.98:0/2780942686 shutdown_connections 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 1 -- 172.21.15.98:0/2780942686 wait complete. 2024-04-25T17:39:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.301+0000 7f4d7817e740 20 client.15714 trim_cache size 0 max 16384 2024-04-25T17:39:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:39:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:39:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:39:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:39:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 -- 172.21.15.98:0/1214183355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d580a3f20 msgr2=0x7f4d580a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d580a3f20 0x7f4d580a63e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d64005ec0 tx=0x7f4d64005d10 comp rx=0 tx=0).stop 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 -- 172.21.15.98:0/1214183355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d70100f80 msgr2=0x7f4d70073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d70100f80 0x7f4d70073ba0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800ba50 tx=0x7f4d6800bf10 comp rx=0 tx=0).stop 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 -- 172.21.15.98:0/1214183355 shutdown_connections 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4d580a3f20 0x7f4d580a63e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d70106680 0x7f4d7006c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d70100f80 0x7f4d70073ba0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 --2- 172.21.15.98:0/1214183355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d701005c0 0x7f4d70073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 -- 172.21.15.98:0/1214183355 >> 172.21.15.98:0/1214183355 conn(0x7f4d700773e0 msgr2=0x7f4d700fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 -- 172.21.15.98:0/1214183355 shutdown_connections 2024-04-25T17:39:42.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:42.302+0000 7f4d7538e640 1 -- 172.21.15.98:0/1214183355 wait complete. 2024-04-25T17:39:43.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:43 smithi098 ceph-mon[29192]: pgmap v118: 97 pgs: 97 active+clean; 1.9 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 10 MiB/s wr, 875 op/s 2024-04-25T17:39:43.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:43 smithi203 ceph-mon[32834]: pgmap v118: 97 pgs: 97 active+clean; 1.9 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 10 MiB/s wr, 875 op/s 2024-04-25T17:39:44.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:43 smithi173 ceph-mon[32856]: pgmap v118: 97 pgs: 97 active+clean; 1.9 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 10 MiB/s wr, 875 op/s 2024-04-25T17:39:45.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:45 smithi098 ceph-mon[29192]: pgmap v119: 97 pgs: 97 active+clean; 1.9 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.3 MiB/s wr, 610 op/s 2024-04-25T17:39:45.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:45 smithi203 ceph-mon[32834]: pgmap v119: 97 pgs: 97 active+clean; 1.9 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.3 MiB/s wr, 610 op/s 2024-04-25T17:39:46.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:45 smithi173 ceph-mon[32856]: pgmap v119: 97 pgs: 97 active+clean; 1.9 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 7.3 MiB/s wr, 610 op/s 2024-04-25T17:39:46.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 195.0 s, 3972.8 tps, lat 25.242 ms stddev 27.618 2024-04-25T17:39:47.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:47 smithi098 ceph-mon[29192]: pgmap v120: 97 pgs: 97 active+clean; 1.9 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 14 MiB/s wr, 979 op/s 2024-04-25T17:39:47.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:47 smithi203 ceph-mon[32834]: pgmap v120: 97 pgs: 97 active+clean; 1.9 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 14 MiB/s wr, 979 op/s 2024-04-25T17:39:48.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:47 smithi173 ceph-mon[32856]: pgmap v120: 97 pgs: 97 active+clean; 1.9 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 14 MiB/s wr, 979 op/s 2024-04-25T17:39:49.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:49 smithi098 ceph-mon[29192]: pgmap v121: 97 pgs: 97 active+clean; 1.9 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.3 MiB/s wr, 673 op/s 2024-04-25T17:39:49.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:49 smithi203 ceph-mon[32834]: pgmap v121: 97 pgs: 97 active+clean; 1.9 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.3 MiB/s wr, 673 op/s 2024-04-25T17:39:50.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:49 smithi173 ceph-mon[32856]: pgmap v121: 97 pgs: 97 active+clean; 1.9 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 9.3 MiB/s wr, 673 op/s 2024-04-25T17:39:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 200.0 s, 3307.0 tps, lat 30.257 ms stddev 31.667 2024-04-25T17:39:51.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:51 smithi098 ceph-mon[29192]: pgmap v122: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 777 op/s 2024-04-25T17:39:51.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:51 smithi203 ceph-mon[32834]: pgmap v122: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 777 op/s 2024-04-25T17:39:52.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:51 smithi173 ceph-mon[32856]: pgmap v122: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 777 op/s 2024-04-25T17:39:52.089 INFO:tasks.quiescer.fs.[cephfs]:Total duration (50.5) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-25T17:39:52.090 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.25 2024-04-25T17:39:52.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:39:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:39:53.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:53 smithi098 ceph-mon[29192]: pgmap v123: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 885 op/s 2024-04-25T17:39:53.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:53 smithi203 ceph-mon[32834]: pgmap v123: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 885 op/s 2024-04-25T17:39:54.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:53 smithi173 ceph-mon[32856]: pgmap v123: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 MiB/s wr, 885 op/s 2024-04-25T17:39:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.754+0000 7f29f24aa700 1 -- 172.21.15.98:0/2636450190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29ec101890 msgr2=0x7f29ec101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.754+0000 7f29f24aa700 1 --2- 172.21.15.98:0/2636450190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29ec101890 0x7f29ec101d10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0098f0 tx=0x7f29dc009c00 comp rx=0 tx=0).stop 2024-04-25T17:39:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.754+0000 7f29f24aa700 1 -- 172.21.15.98:0/2636450190 shutdown_connections 2024-04-25T17:39:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.754+0000 7f29f24aa700 1 --2- 172.21.15.98:0/2636450190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29ec101890 0x7f29ec101d10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.754+0000 7f29f24aa700 1 --2- 172.21.15.98:0/2636450190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29ec1066f0 0x7f29ec106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.754+0000 7f29f24aa700 1 --2- 172.21.15.98:0/2636450190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ec1070a0 0x7f29ec101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.754+0000 7f29f24aa700 1 -- 172.21.15.98:0/2636450190 >> 172.21.15.98:0/2636450190 conn(0x7f29ec076bf0 msgr2=0x7f29ec077000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.755+0000 7f29f24aa700 1 -- 172.21.15.98:0/2636450190 shutdown_connections 2024-04-25T17:39:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.755+0000 7f29f24aa700 1 -- 172.21.15.98:0/2636450190 wait complete. 2024-04-25T17:39:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.755+0000 7f29f24aa700 1 Processor -- start 2024-04-25T17:39:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.756+0000 7f29f24aa700 1 -- start start 2024-04-25T17:39:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.756+0000 7f29f24aa700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29ec101890 0x7f29ec0734d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.756+0000 7f29f24aa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ec1066f0 0x7f29ec073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.756+0000 7f29f24aa700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29ec1070a0 0x7f29ec116610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.756+0000 7f29f24aa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec10aae0 con 0x7f29ec1066f0 2024-04-25T17:39:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.756+0000 7f29f24aa700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec10a960 con 0x7f29ec1070a0 2024-04-25T17:39:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.756+0000 7f29f24aa700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec10ac60 con 0x7f29ec101890 2024-04-25T17:39:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.756+0000 7f29f14a8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29ec101890 0x7f29ec0734d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.756+0000 7f29f14a8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29ec101890 0x7f29ec0734d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:43714/0 (socket says 172.21.15.98:43714) 2024-04-25T17:39:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.756+0000 7f29f14a8700 1 -- 172.21.15.98:0/2145960237 learned_addr learned my addr 172.21.15.98:0/2145960237 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.757+0000 7f29f0ca7700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ec1066f0 0x7f29ec073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.757+0000 7f29f14a8700 1 -- 172.21.15.98:0/2145960237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29ec1070a0 msgr2=0x7f29ec116610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.757+0000 7f29f14a8700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29ec1070a0 0x7f29ec116610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.757+0000 7f29f14a8700 1 -- 172.21.15.98:0/2145960237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ec1066f0 msgr2=0x7f29ec073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.757+0000 7f29f14a8700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ec1066f0 0x7f29ec073a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.757+0000 7f29f14a8700 1 -- 172.21.15.98:0/2145960237 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec116d10 con 0x7f29ec101890 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.757+0000 7f29f14a8700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29ec101890 0x7f29ec0734d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f29e801ea30 tx=0x7f29e801edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.758+0000 7f29e27fc700 1 -- 172.21.15.98:0/2145960237 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e8028020 con 0x7f29ec101890 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.758+0000 7f29f24aa700 1 -- 172.21.15.98:0/2145960237 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ec117000 con 0x7f29ec101890 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.758+0000 7f29f24aa700 1 -- 172.21.15.98:0/2145960237 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ec10e6a0 con 0x7f29ec101890 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.758+0000 7f29e27fc700 1 -- 172.21.15.98:0/2145960237 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29e8021470 con 0x7f29ec101890 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.758+0000 7f29e27fc700 1 -- 172.21.15.98:0/2145960237 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e8020630 con 0x7f29ec101890 2024-04-25T17:39:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.759+0000 7f29f24aa700 1 -- 172.21.15.98:0/2145960237 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d0005330 con 0x7f29ec101890 2024-04-25T17:39:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.761+0000 7f29e27fc700 1 -- 172.21.15.98:0/2145960237 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f29e802b020 con 0x7f29ec101890 2024-04-25T17:39:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.761+0000 7f29e27fc700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f29d80a3e30 0x7f29d80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.761+0000 7f29f0ca7700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f29d80a3e30 0x7f29d80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.761+0000 7f29e27fc700 1 -- 172.21.15.98:0/2145960237 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f29e80e1440 con 0x7f29ec101890 2024-04-25T17:39:54.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.762+0000 7f29f0ca7700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f29d80a3e30 0x7f29d80a62f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009da0 tx=0x7f29e400b040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:54.766+0000 7f29e27fc700 1 -- 172.21.15.98:0/2145960237 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29e8032020 con 0x7f29ec101890 2024-04-25T17:39:55.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.168+0000 7f29f24aa700 1 -- 172.21.15.98:0/2145960237 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29d00058f0 con 0x7f29ec101890 2024-04-25T17:39:55.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.169+0000 7f29e27fc700 1 -- 172.21.15.98:0/2145960237 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11883 (secure 0 0 0) 0x7f29ec117000 con 0x7f29ec101890 2024-04-25T17:39:55.169 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:39:55.169 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:39:15.018336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.171+0000 7f29f24aa700 1 -- 172.21.15.98:0/2145960237 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f29d80a3e30 msgr2=0x7f29d80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.171+0000 7f29f24aa700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f29d80a3e30 0x7f29d80a62f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009da0 tx=0x7f29e400b040 comp rx=0 tx=0).stop 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.171+0000 7f29f24aa700 1 -- 172.21.15.98:0/2145960237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29ec101890 msgr2=0x7f29ec0734d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.171+0000 7f29f24aa700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29ec101890 0x7f29ec0734d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f29e801ea30 tx=0x7f29e801edf0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.171+0000 7f29f24aa700 1 -- 172.21.15.98:0/2145960237 shutdown_connections 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.171+0000 7f29f24aa700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f29d80a3e30 0x7f29d80a62f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.171+0000 7f29f24aa700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29ec101890 0x7f29ec0734d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.171+0000 7f29f24aa700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29ec1070a0 0x7f29ec116610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.171+0000 7f29f24aa700 1 --2- 172.21.15.98:0/2145960237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29ec1066f0 0x7f29ec073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.171+0000 7f29f24aa700 1 -- 172.21.15.98:0/2145960237 >> 172.21.15.98:0/2145960237 conn(0x7f29ec076bf0 msgr2=0x7f29ec102400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.172+0000 7f29f24aa700 1 -- 172.21.15.98:0/2145960237 shutdown_connections 2024-04-25T17:39:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:55.172+0000 7f29f24aa700 1 -- 172.21.15.98:0/2145960237 wait complete. 2024-04-25T17:39:55.173 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 31 2024-04-25T17:39:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.444+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1889126946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bc8101330 msgr2=0x7f2bc8101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.444+0000 7f2bcdce8640 1 --2- 172.21.15.98:0/1889126946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bc8101330 0x7f2bc8101710 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800d910 tx=0x7f2bb802c380 comp rx=0 tx=0).stop 2024-04-25T17:39:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.447+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1889126946 shutdown_connections 2024-04-25T17:39:55.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.447+0000 7f2bcdce8640 1 --2- 172.21.15.98:0/1889126946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bc81061d0 0x7f2bc810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.447+0000 7f2bcdce8640 1 --2- 172.21.15.98:0/1889126946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc8101ce0 0x7f2bc8105af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.447+0000 7f2bcdce8640 1 --2- 172.21.15.98:0/1889126946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bc8101330 0x7f2bc8101710 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.447+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1889126946 >> 172.21.15.98:0/1889126946 conn(0x7f2bc8077d30 msgr2=0x7f2bc8078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.447+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1889126946 shutdown_connections 2024-04-25T17:39:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.447+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1889126946 wait complete. 2024-04-25T17:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.449+0000 7f2bcdce8640 1 Processor -- start 2024-04-25T17:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.449+0000 7f2bcdce8640 1 -- start start 2024-04-25T17:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bcdce8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bc8101330 0x7f2bc81a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bcdce8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc81061d0 0x7f2bc81a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bcdce8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bc819eb00 0x7f2bc819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bc8101330 0x7f2bc81a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bcdce8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc8110050 con 0x7f2bc8101330 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bc8101330 0x7f2bc81a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48202/0 (socket says 172.21.15.98:48202) 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc77fe640 1 -- 172.21.15.98:0/1747269596 learned_addr learned my addr 172.21.15.98:0/1747269596 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1747269596 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc810fed0 con 0x7f2bc819eb00 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1747269596 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc81101d0 con 0x7f2bc81061d0 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc81061d0 0x7f2bc81a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc7fff640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bc819eb00 0x7f2bc819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc77fe640 1 -- 172.21.15.98:0/1747269596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc81061d0 msgr2=0x7f2bc81a4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc77fe640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc81061d0 0x7f2bc81a4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc77fe640 1 -- 172.21.15.98:0/1747269596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bc819eb00 msgr2=0x7f2bc819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc77fe640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bc819eb00 0x7f2bc819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc77fe640 1 -- 172.21.15.98:0/1747269596 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc819f620 con 0x7f2bc8101330 2024-04-25T17:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc7fff640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bc819eb00 0x7f2bc819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.450+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc81061d0 0x7f2bc81a4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.451+0000 7f2bc77fe640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bc8101330 0x7f2bc81a4950 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8000c00 tx=0x7f2bb8034a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.453+0000 7f2bc4ff9640 1 -- 172.21.15.98:0/1747269596 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8034e40 con 0x7f2bc8101330 2024-04-25T17:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.453+0000 7f2bc4ff9640 1 -- 172.21.15.98:0/1747269596 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bb80035f0 con 0x7f2bc8101330 2024-04-25T17:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.453+0000 7f2bc4ff9640 1 -- 172.21.15.98:0/1747269596 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb80044e0 con 0x7f2bc8101330 2024-04-25T17:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.453+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1747269596 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc819f8b0 con 0x7f2bc8101330 2024-04-25T17:39:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.454+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1747269596 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc81aba50 con 0x7f2bc8101330 2024-04-25T17:39:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.457+0000 7f2bd0a02740 1 Processor -- start 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.457+0000 7f2bd0a02740 1 -- start start 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38d620 0x55b0fa38da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38df40 0x55b0fa3982c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa398800 0x55b0fa39abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b0fa2f2970 con 0x55b0fa38df40 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b0fa28f4a0 con 0x55b0fa38d620 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b0fa2ee930 con 0x55b0fa398800 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa398800 0x55b0fa39abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc4ff9640 1 -- 172.21.15.98:0/1747269596 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2bb800c050 con 0x7f2bc8101330 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa398800 0x55b0fa39abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:43744/0 (socket says 172.21.15.98:43744) 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/193508241 learned_addr learned my addr 172.21.15.98:0/193508241 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc7fff640 1 --2- 172.21.15.98:0/193508241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38d620 0x55b0fa38da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/193508241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38d620 msgr2=0x55b0fa38da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/193508241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38d620 0x55b0fa38da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/193508241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38df40 msgr2=0x55b0fa3982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/193508241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38df40 0x55b0fa3982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/193508241 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0fa005210 con 0x55b0fa398800 2024-04-25T17:39:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.458+0000 7f2bc7fff640 1 --2- 172.21.15.98:0/193508241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38d620 0x55b0fa38da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:39:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.459+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/193508241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa398800 0x55b0fa39abf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009fd0 tx=0x7f2bbc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b2ea9701c33a4af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.459+0000 7f2badffb640 1 -- 172.21.15.98:0/193508241 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc004030 con 0x55b0fa398800 2024-04-25T17:39:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.459+0000 7f2badffb640 1 -- 172.21.15.98:0/193508241 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc0054d0 con 0x55b0fa398800 2024-04-25T17:39:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.459+0000 7f2bc4ff9640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2bb00a3f20 0x7f2bb00a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.459+0000 7f2badffb640 1 -- 172.21.15.98:0/193508241 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc004c40 con 0x55b0fa398800 2024-04-25T17:39:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.459+0000 7f2bc4ff9640 1 -- 172.21.15.98:0/1747269596 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f2bb80eff20 con 0x7f2bc8101330 2024-04-25T17:39:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.460+0000 7f2bd0a02740 1 -- 172.21.15.98:0/193508241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa398800 msgr2=0x55b0fa39abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.460+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/193508241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa398800 0x55b0fa39abf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009fd0 tx=0x7f2bbc0118d0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.460+0000 7f2bd0a02740 1 -- 172.21.15.98:0/193508241 shutdown_connections 2024-04-25T17:39:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.460+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/193508241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa398800 0x55b0fa39abf0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009fd0 tx=0x7f2bbc0118d0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.460+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/193508241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38df40 0x55b0fa3982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.460+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/193508241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38d620 0x55b0fa38da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.460+0000 7f2bd0a02740 1 -- 172.21.15.98:0/193508241 >> 172.21.15.98:0/193508241 conn(0x55b0fa389b30 msgr2=0x55b0fa389f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.461+0000 7f2bc7fff640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2bb00a3f20 0x7f2bb00a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.461+0000 7f2bc7fff640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2bb00a3f20 0x7f2bb00a63e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2bc00092f0 tx=0x7f2bc0009280 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:55.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.462+0000 7f2bd0a02740 1 -- 172.21.15.98:0/193508241 shutdown_connections 2024-04-25T17:39:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.463+0000 7f2bd0a02740 1 -- 172.21.15.98:0/193508241 wait complete. 2024-04-25T17:39:55.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.463+0000 7f2bd0a02740 1 -- start start 2024-04-25T17:39:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.463+0000 7f2bd0a02740 1 Processor -- start 2024-04-25T17:39:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38d620 0x55b0fa395670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa395bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa3960f0 0x55b0fa3b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b0fa3ac440 con 0x55b0fa3960f0 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b0fa28d0b0 con 0x55b0fa38df40 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa395bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa3960f0 0x55b0fa3b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa395bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:36678/0 (socket says 172.21.15.98:36678) 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/4220513933 learned_addr learned my addr 172.21.15.98:0/4220513933 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc77fe640 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38d620 0x55b0fa395670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bd0a02740 1 -- 172.21.15.98:0/4220513933 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b0fa3ac5c0 con 0x55b0fa38d620 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bd0a02740 10 client.0 _ll_register_callbacks cb 0x55b0fa27ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/4220513933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38d620 msgr2=0x55b0fa395670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38d620 0x55b0fa395670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/4220513933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa3960f0 msgr2=0x55b0fa3b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa3960f0 0x55b0fa3b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/4220513933 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0fa0d1620 con 0x55b0fa38df40 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.464+0000 7f2bc77fe640 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38d620 0x55b0fa395670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa395bb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009fa0 tx=0x7f2bbc00fcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1ef665af097f19b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2ba6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/4220513933 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc029020 con 0x55b0fa38df40 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/4220513933 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f2bbc025680 con 0x55b0fa38df40 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2bd0a02740 1 -- 172.21.15.98:0/4220513933 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bbc0054d0 con 0x55b0fa38df40 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/4220513933 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2bbc034050 con 0x55b0fa38df40 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2bd0a02740 10 client.25304 fetch_fsmap learned FSMap version 31 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2bd0a02740 1 -- 172.21.15.98:0/4220513933 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b0fa3b8970 con 0x55b0fa38df40 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/4220513933 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc0054d0 con 0x55b0fa38df40 2024-04-25T17:39:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/4220513933 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f2bbc004b00 con 0x55b0fa38df40 2024-04-25T17:39:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2bd0a02740 10 client.25304 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:39:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2bd0a02740 10 client.25304 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:39:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2bd0a02740 20 client.25304 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b0fa3b9d80 0x55b0fa3bc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2bd0a02740 4 client.25304 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:39:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.465+0000 7f2bd0a02740 1 -- 172.21.15.98:0/4220513933 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0fa300390 con 0x55b0fa3b9d80 2024-04-25T17:39:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.466+0000 7f2bc77fe640 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b0fa3b9d80 0x55b0fa3bc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.467+0000 7f2ba6ffd640 10 client.25304 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.467+0000 7f2bc77fe640 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b0fa3b9d80 0x55b0fa3bc170 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c65c72bd25733d97 server_cookie=177168ca4f6843bc in_seq=0 out_seq=0 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.467+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/4220513933 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0fa300390 con 0x55b0fa3b9d80 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.467+0000 7f2ba6ffd640 10 client.25304 handle_command_reply: tid=0 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.467+0000 7f2bd0a02740 1 client.25304 shutdown 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 1 -- 172.21.15.98:0/4220513933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 msgr2=0x55b0fa395bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa395bb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009fa0 tx=0x7f2bbc00fcf0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 1 -- 172.21.15.98:0/4220513933 shutdown_connections 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b0fa3b9d80 0x55b0fa3bc170 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa3960f0 0x55b0fa3b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2ba6ffd640 0 client.25304 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa395bb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/4220513933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38d620 0x55b0fa395670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 1 -- 172.21.15.98:0/4220513933 >> 172.21.15.98:0/4220513933 conn(0x55b0fa389b30 msgr2=0x55b0fa38f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 1 -- 172.21.15.98:0/4220513933 shutdown_connections 2024-04-25T17:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 1 -- 172.21.15.98:0/4220513933 wait complete. 2024-04-25T17:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.468+0000 7f2bd0a02740 20 client.25304 trim_cache size 0 max 16384 2024-04-25T17:39:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.491+0000 7f2bd0a02740 1 Processor -- start 2024-04-25T17:39:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.491+0000 7f2bd0a02740 1 -- start start 2024-04-25T17:39:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 0x55b0fa390970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38df40 0x55b0fa3c1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa3960f0 0x55b0fa3c1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc81ae0d0 con 0x55b0fa38d620 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc8073cf0 con 0x55b0fa3960f0 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc8071b60 con 0x55b0fa38df40 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38df40 0x55b0fa3c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 0x55b0fa390970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38df40 0x55b0fa3c1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:43758/0 (socket says 172.21.15.98:43758) 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/3034201366 learned_addr learned my addr 172.21.15.98:0/3034201366 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc7fff640 1 --2- 172.21.15.98:0/3034201366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa3960f0 0x55b0fa3c1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/3034201366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa3960f0 msgr2=0x55b0fa3c1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/3034201366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa3960f0 0x55b0fa3c1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/3034201366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 msgr2=0x55b0fa390970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/3034201366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 0x55b0fa390970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc6ffd640 1 -- 172.21.15.98:0/3034201366 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0fa38dd60 con 0x55b0fa38df40 2024-04-25T17:39:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.492+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/3034201366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38df40 0x55b0fa3c1930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00a010 tx=0x7f2bbc005f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=160fe11180c35bd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.493+0000 7f2bad7fa640 1 -- 172.21.15.98:0/3034201366 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc03d070 con 0x55b0fa38df40 2024-04-25T17:39:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.493+0000 7f2bad7fa640 1 -- 172.21.15.98:0/3034201366 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc016030 con 0x55b0fa38df40 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.493+0000 7f2bad7fa640 1 -- 172.21.15.98:0/3034201366 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc029040 con 0x55b0fa38df40 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.494+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3034201366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38df40 msgr2=0x55b0fa3c1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.494+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/3034201366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38df40 0x55b0fa3c1930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00a010 tx=0x7f2bbc005f50 comp rx=0 tx=0).stop 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.494+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3034201366 shutdown_connections 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.494+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/3034201366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa3960f0 0x55b0fa3c1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.494+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/3034201366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa38df40 0x55b0fa3c1930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.494+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/3034201366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 0x55b0fa390970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.494+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3034201366 >> 172.21.15.98:0/3034201366 conn(0x55b0fa389b30 msgr2=0x55b0fa390260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.494+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3034201366 shutdown_connections 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.494+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3034201366 wait complete. 2024-04-25T17:39:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bd0a02740 1 -- start start 2024-04-25T17:39:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bd0a02740 1 Processor -- start 2024-04-25T17:39:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 0x55b0fa418580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa418ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bd0a02740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa3960f0 0x55b0fa3acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc80721b0 con 0x55b0fa38d620 2024-04-25T17:39:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc8073620 con 0x55b0fa38df40 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bd0a02740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc8072880 con 0x55b0fa3960f0 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bd0a02740 10 client.0 _ll_register_callbacks cb 0x7f2bbc00f980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bc77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 0x55b0fa418580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bc77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 0x55b0fa418580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48244/0 (socket says 172.21.15.98:48244) 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.495+0000 7f2bc77fe640 1 -- 172.21.15.98:0/3112522853 learned_addr learned my addr 172.21.15.98:0/3112522853 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bc7fff640 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa3960f0 0x55b0fa3acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa418ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bc77fe640 1 -- 172.21.15.98:0/3112522853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa3960f0 msgr2=0x55b0fa3acbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bc77fe640 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa3960f0 0x55b0fa3acbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bc77fe640 1 -- 172.21.15.98:0/3112522853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 msgr2=0x55b0fa418ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bc77fe640 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa418ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bc77fe640 1 -- 172.21.15.98:0/3112522853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0fa3ad2d0 con 0x55b0fa38d620 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa418ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bc7fff640 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa3960f0 0x55b0fa3acbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bc77fe640 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 0x55b0fa418580 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80b89d0 tx=0x7f2bb802ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ef310a493cf30c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2ba6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/3112522853 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb80c4c70 con 0x55b0fa38d620 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/3112522853 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f2bb80ba300 con 0x55b0fa38d620 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3112522853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bbc016030 con 0x55b0fa38d620 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/3112522853 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2bb80037a0 con 0x55b0fa38d620 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bd0a02740 10 client.15735 fetch_fsmap learned FSMap version 31 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3112522853 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b0fa41fce0 con 0x55b0fa38d620 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.496+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/3112522853 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc016030 con 0x55b0fa38d620 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.497+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/3112522853 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f2bb8003aa0 con 0x55b0fa38d620 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.497+0000 7f2bd0a02740 10 client.15735 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.497+0000 7f2bd0a02740 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.497+0000 7f2bd0a02740 20 client.15735 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.497+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b0fa3b9d80 0x55b0fa422410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.497+0000 7f2bd0a02740 4 client.15735 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.497+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3112522853 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b0fa395f40 con 0x55b0fa3b9d80 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.497+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b0fa3b9d80 0x55b0fa422410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.497+0000 7f2ba6ffd640 10 client.15735 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.497+0000 7f2bc6ffd640 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b0fa3b9d80 0x55b0fa422410 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=933771896d2290cc server_cookie=6c5dea73e220d8ef in_seq=0 out_seq=0 2024-04-25T17:39:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.511+0000 7f2ba6ffd640 1 -- 172.21.15.98:0/3112522853 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b0fa395f40 con 0x55b0fa3b9d80 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.511+0000 7f2ba6ffd640 10 client.15735 handle_command_reply: tid=0 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.511+0000 7f2bd0a02740 1 client.15735 shutdown 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3112522853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 msgr2=0x55b0fa418580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 0x55b0fa418580 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80b89d0 tx=0x7f2bb802ca00 comp rx=0 tx=0).stop 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3112522853 shutdown_connections 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b0fa3b9d80 0x55b0fa422410 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b0fa3960f0 0x55b0fa3acbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b0fa38df40 0x55b0fa418ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2bd0a02740 1 --2- 172.21.15.98:0/3112522853 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b0fa38d620 0x55b0fa418580 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3112522853 >> 172.21.15.98:0/3112522853 conn(0x55b0fa389b30 msgr2=0x55b0fa388cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2ba6ffd640 0 client.15735 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3112522853 shutdown_connections 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.512+0000 7f2bd0a02740 1 -- 172.21.15.98:0/3112522853 wait complete. 2024-04-25T17:39:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.513+0000 7f2bd0a02740 20 client.15735 trim_cache size 0 max 16384 2024-04-25T17:39:55.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:39:55.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:39:55.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:39:55.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:39:55.514 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.513+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1747269596 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2bb00a3f20 msgr2=0x7f2bb00a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.513+0000 7f2bcdce8640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2bb00a3f20 0x7f2bb00a63e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2bc00092f0 tx=0x7f2bc0009280 comp rx=0 tx=0).stop 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.513+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1747269596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bc8101330 msgr2=0x7f2bc81a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.513+0000 7f2bcdce8640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bc8101330 0x7f2bc81a4950 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8000c00 tx=0x7f2bb8034a40 comp rx=0 tx=0).stop 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.513+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1747269596 shutdown_connections 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.513+0000 7f2bcdce8640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2bb00a3f20 0x7f2bb00a63e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.513+0000 7f2bcdce8640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bc819eb00 0x7f2bc819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.513+0000 7f2bcdce8640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc81061d0 0x7f2bc81a4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.514+0000 7f2bcdce8640 1 --2- 172.21.15.98:0/1747269596 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bc8101330 0x7f2bc81a4950 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.514+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1747269596 >> 172.21.15.98:0/1747269596 conn(0x7f2bc8077d30 msgr2=0x7f2bc8100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.514+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1747269596 shutdown_connections 2024-04-25T17:39:55.515 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:39:55.514+0000 7f2bcdce8640 1 -- 172.21.15.98:0/1747269596 wait complete. 2024-04-25T17:39:55.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:55 smithi098 ceph-mon[29192]: pgmap v124: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.5 MiB/s wr, 680 op/s 2024-04-25T17:39:55.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:55 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2145960237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:39:55.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:55 smithi203 ceph-mon[32834]: pgmap v124: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.5 MiB/s wr, 680 op/s 2024-04-25T17:39:55.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:55 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2145960237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:39:56.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:55 smithi173 ceph-mon[32856]: pgmap v124: 97 pgs: 97 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.5 MiB/s wr, 680 op/s 2024-04-25T17:39:56.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:55 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2145960237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:39:56.182 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 85.25 --await-for 42.0' 2024-04-25T17:39:56.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 85.25 --await-for 42.0 2024-04-25T17:39:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:39:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 205.0 s, 2829.2 tps, lat 35.276 ms stddev 40.306 2024-04-25T17:39:57.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:57 smithi098 ceph-mon[29192]: pgmap v125: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 14 MiB/s wr, 983 op/s 2024-04-25T17:39:57.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:57 smithi203 ceph-mon[32834]: pgmap v125: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 14 MiB/s wr, 983 op/s 2024-04-25T17:39:58.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:57 smithi173 ceph-mon[32856]: pgmap v125: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 14 MiB/s wr, 983 op/s 2024-04-25T17:39:58.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.780+0000 7f6c4052a700 1 -- 172.21.15.98:0/3844619031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c38108a60 msgr2=0x7f6c38108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:58.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.780+0000 7f6c4052a700 1 --2- 172.21.15.98:0/3844619031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c38108a60 0x7f6c38108e40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009990 tx=0x7f6c34009ca0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.781+0000 7f6c4052a700 1 -- 172.21.15.98:0/3844619031 shutdown_connections 2024-04-25T17:39:58.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.781+0000 7f6c4052a700 1 --2- 172.21.15.98:0/3844619031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c38103480 0x7f6c3810f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.781+0000 7f6c4052a700 1 --2- 172.21.15.98:0/3844619031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c38108a60 0x7f6c38108e40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.781+0000 7f6c4052a700 1 --2- 172.21.15.98:0/3844619031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38102ac0 0x7f6c38102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.781+0000 7f6c4052a700 1 -- 172.21.15.98:0/3844619031 >> 172.21.15.98:0/3844619031 conn(0x7f6c38079b60 msgr2=0x7f6c38079f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.781+0000 7f6c4052a700 1 -- 172.21.15.98:0/3844619031 shutdown_connections 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.781+0000 7f6c4052a700 1 -- 172.21.15.98:0/3844619031 wait complete. 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.782+0000 7f6c4052a700 1 Processor -- start 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.782+0000 7f6c4052a700 1 -- start start 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.782+0000 7f6c4052a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c38102ac0 0x7f6c38198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.782+0000 7f6c4052a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38103480 0x7f6c38199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.782+0000 7f6c4052a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c38108a60 0x7f6c3819d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.782+0000 7f6c4052a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38112cb0 con 0x7f6c38108a60 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.782+0000 7f6c4052a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38112b30 con 0x7f6c38102ac0 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.782+0000 7f6c4052a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38112e30 con 0x7f6c38103480 2024-04-25T17:39:58.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.782+0000 7f6c3e2c6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c38102ac0 0x7f6c38198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.782+0000 7f6c3dac5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38103480 0x7f6c38199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.783+0000 7f6c3eac7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c38108a60 0x7f6c3819d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.783+0000 7f6c3dac5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38103480 0x7f6c38199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:43790/0 (socket says 172.21.15.98:43790) 2024-04-25T17:39:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.783+0000 7f6c3e2c6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c38102ac0 0x7f6c38198f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:36696/0 (socket says 172.21.15.98:36696) 2024-04-25T17:39:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.783+0000 7f6c3dac5700 1 -- 172.21.15.98:0/2891960798 learned_addr learned my addr 172.21.15.98:0/2891960798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.783+0000 7f6c3dac5700 1 -- 172.21.15.98:0/2891960798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c38102ac0 msgr2=0x7f6c38198f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.783+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c38102ac0 0x7f6c38198f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.783+0000 7f6c3dac5700 1 -- 172.21.15.98:0/2891960798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c38108a60 msgr2=0x7f6c3819d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.783+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c38108a60 0x7f6c3819d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.783+0000 7f6c3dac5700 1 -- 172.21.15.98:0/2891960798 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3819df50 con 0x7f6c38103480 2024-04-25T17:39:58.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.784+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38103480 0x7f6c38199480 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6c2800e9c0 tx=0x7f6c2800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:58.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.785+0000 7f6c2f7fe700 1 -- 172.21.15.98:0/2891960798 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2800cbd0 con 0x7f6c38103480 2024-04-25T17:39:58.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.785+0000 7f6c4052a700 1 -- 172.21.15.98:0/2891960798 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3819e240 con 0x7f6c38103480 2024-04-25T17:39:58.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.785+0000 7f6c2f7fe700 1 -- 172.21.15.98:0/2891960798 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c2800ee50 con 0x7f6c38103480 2024-04-25T17:39:58.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.785+0000 7f6c2f7fe700 1 -- 172.21.15.98:0/2891960798 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c28005550 con 0x7f6c38103480 2024-04-25T17:39:58.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.785+0000 7f6c4052a700 1 -- 172.21.15.98:0/2891960798 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c381a5ac0 con 0x7f6c38103480 2024-04-25T17:39:58.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.788+0000 7f6c2f7fe700 1 -- 172.21.15.98:0/2891960798 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6c28021440 con 0x7f6c38103480 2024-04-25T17:39:58.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.788+0000 7f6c2f7fe700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6c240a3e50 0x7f6c240a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.789+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6c240a3e50 0x7f6c240a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.789+0000 7f6c2f7fe700 1 -- 172.21.15.98:0/2891960798 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f6c280d0a90 con 0x7f6c38103480 2024-04-25T17:39:58.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.789+0000 7f6c4adb3b80 1 Processor -- start 2024-04-25T17:39:58.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.790+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6c240a3e50 0x7f6c240a6310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c34006010 tx=0x7f6c3400b280 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:58.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.791+0000 7f6c4adb3b80 1 -- start start 2024-04-25T17:39:58.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.791+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 0x559857678e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.791+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598576793a0 0x559857683d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.791+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 0x559857686660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.791+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5598575cbec0 con 0x5598576793a0 2024-04-25T17:39:58.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.791+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559857572af0 con 0x559857684270 2024-04-25T17:39:58.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.791+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5598575c7e80 con 0x559857678a80 2024-04-25T17:39:58.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3eac7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 0x559857686660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3eac7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 0x559857686660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:36702/0 (socket says 172.21.15.98:36702) 2024-04-25T17:39:58.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3eac7700 1 -- 172.21.15.98:0/4127374025 learned_addr learned my addr 172.21.15.98:0/4127374025 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:58.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3e2c6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 0x559857678e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/4127374025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598576793a0 0x559857683d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3eac7700 1 -- 172.21.15.98:0/4127374025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 msgr2=0x559857678e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:58.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3eac7700 1 --2- 172.21.15.98:0/4127374025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 0x559857678e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3eac7700 1 -- 172.21.15.98:0/4127374025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598576793a0 msgr2=0x559857683d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3eac7700 1 --2- 172.21.15.98:0/4127374025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598576793a0 0x559857683d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3eac7700 1 -- 172.21.15.98:0/4127374025 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559857519a60 con 0x559857684270 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/4127374025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598576793a0 0x559857683d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c3eac7700 1 --2- 172.21.15.98:0/4127374025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 0x559857686660 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c30011630 tx=0x7f6c30009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=668f4b31f8e803e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c2cff9700 1 -- 172.21.15.98:0/4127374025 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c30003a00 con 0x559857684270 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c2cff9700 1 -- 172.21.15.98:0/4127374025 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c3001a070 con 0x559857684270 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.792+0000 7f6c2cff9700 1 -- 172.21.15.98:0/4127374025 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c30007760 con 0x559857684270 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.793+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4127374025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 msgr2=0x559857686660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.793+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/4127374025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 0x559857686660 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c30011630 tx=0x7f6c30009920 comp rx=0 tx=0).stop 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.793+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4127374025 shutdown_connections 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.793+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/4127374025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 0x559857678e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.793+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/4127374025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 0x559857686660 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.793+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/4127374025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5598576793a0 0x559857683d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.793+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4127374025 >> 172.21.15.98:0/4127374025 conn(0x5598575a6820 msgr2=0x559857674f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.793+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4127374025 shutdown_connections 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.793+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4127374025 wait complete. 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.794+0000 7f6c4adb3b80 1 -- start start 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.794+0000 7f6c4adb3b80 1 Processor -- start 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.794+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857678a80 0x559857683a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.794+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 0x559857680d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.794+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 0x559857681270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.794+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559857686c00 con 0x559857678a80 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.794+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55985756ad60 con 0x559857684270 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.794+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559857686d80 con 0x5598576793a0 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.794+0000 7f6c4adb3b80 10 client.0 _ll_register_callbacks cb 0x559857564ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:58.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3dac5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 0x559857680d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3eac7700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 0x559857681270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3dac5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 0x559857680d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:43818/0 (socket says 172.21.15.98:43818) 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3dac5700 1 -- 172.21.15.98:0/909422617 learned_addr learned my addr 172.21.15.98:0/909422617 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857678a80 0x559857683a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3dac5700 1 -- 172.21.15.98:0/909422617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 msgr2=0x559857681270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 0x559857681270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3dac5700 1 -- 172.21.15.98:0/909422617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857678a80 msgr2=0x559857683a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857678a80 0x559857683a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3dac5700 1 -- 172.21.15.98:0/909422617 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559857583400 con 0x5598576793a0 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.795+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857678a80 0x559857683a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.796+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 0x559857680d30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c280ce810 tx=0x7f6c2809c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=572ddd0f2b623501 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.796+0000 7f6c21ffb700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:39:58.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.796+0000 7f6c21ffb700 1 -- 172.21.15.98:0/909422617 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c280b5020 con 0x5598576793a0 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.796+0000 7f6c21ffb700 1 -- 172.21.15.98:0/909422617 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f6c2809ccc0 con 0x5598576793a0 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.796+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/909422617 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c3001a070 con 0x5598576793a0 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.796+0000 7f6c21ffb700 1 -- 172.21.15.98:0/909422617 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6c280a0e10 con 0x5598576793a0 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.796+0000 7f6c4adb3b80 10 client.25279 fetch_fsmap learned FSMap version 31 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.796+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/909422617 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5598576a24d0 con 0x5598576793a0 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.797+0000 7f6c21ffb700 1 -- 172.21.15.98:0/909422617 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c3001a070 con 0x5598576793a0 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.797+0000 7f6c21ffb700 1 -- 172.21.15.98:0/909422617 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6c280880b0 con 0x5598576793a0 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.797+0000 7f6c4adb3b80 10 client.25279 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.797+0000 7f6c4adb3b80 10 client.25279 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.797+0000 7f6c4adb3b80 20 client.25279 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.797+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5598576a4ab0 0x5598576a6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.797+0000 7f6c4adb3b80 4 client.25279 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:39:58.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.797+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/909422617 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598575d96b0 con 0x5598576a4ab0 2024-04-25T17:39:58.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.797+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5598576a4ab0 0x5598576a6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.797+0000 7f6c21ffb700 10 client.25279 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:58.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5598576a4ab0 0x5598576a6ea0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c7b1f5a5ca4172e server_cookie=11bacce6b6d981f7 in_seq=0 out_seq=0 2024-04-25T17:39:58.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c21ffb700 1 -- 172.21.15.98:0/909422617 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5598575d96b0 con 0x5598576a4ab0 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c21ffb700 10 client.25279 handle_command_reply: tid=0 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c4adb3b80 1 client.25279 shutdown 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/909422617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 msgr2=0x559857680d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 0x559857680d30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c280ce810 tx=0x7f6c2809c3c0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/909422617 shutdown_connections 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5598576a4ab0 0x5598576a6ea0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857684270 0x559857681270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857678a80 0x559857683a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c21ffb700 0 client.25279 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.798+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/909422617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 0x559857680d30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.799+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/909422617 >> 172.21.15.98:0/909422617 conn(0x5598575a6820 msgr2=0x559857674b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:58.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.799+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/909422617 shutdown_connections 2024-04-25T17:39:58.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.799+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/909422617 wait complete. 2024-04-25T17:39:58.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.799+0000 7f6c4adb3b80 20 client.25279 trim_cache size 0 max 16384 2024-04-25T17:39:58.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c4adb3b80 1 Processor -- start 2024-04-25T17:39:58.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c4adb3b80 1 -- start start 2024-04-25T17:39:58.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857678a80 0x5598576ac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 0x5598576acc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985768bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c381a71f0 con 0x559857684270 2024-04-25T17:39:58.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38073930 con 0x559857678a80 2024-04-25T17:39:58.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38071780 con 0x5598576793a0 2024-04-25T17:39:58.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c3eac7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985768bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c3eac7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985768bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48282/0 (socket says 172.21.15.98:48282) 2024-04-25T17:39:58.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c3eac7700 1 -- 172.21.15.98:0/2139115701 learned_addr learned my addr 172.21.15.98:0/2139115701 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:58.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/2139115701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857678a80 0x5598576ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/2139115701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 0x5598576acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c3eac7700 1 -- 172.21.15.98:0/2139115701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 msgr2=0x5598576acc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:58.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c3eac7700 1 --2- 172.21.15.98:0/2139115701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 0x5598576acc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c3eac7700 1 -- 172.21.15.98:0/2139115701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857678a80 msgr2=0x5598576ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:58.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.826+0000 7f6c3eac7700 1 --2- 172.21.15.98:0/2139115701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857678a80 0x5598576ac750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.827+0000 7f6c3eac7700 1 -- 172.21.15.98:0/2139115701 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598576a7480 con 0x559857684270 2024-04-25T17:39:58.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.827+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/2139115701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857678a80 0x5598576ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:39:58.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.827+0000 7f6c3eac7700 1 --2- 172.21.15.98:0/2139115701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985768bfc0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x55985767a410 tx=0x7f6c30009400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df28ef4a745c6ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:58.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.827+0000 7f6c23fff700 1 -- 172.21.15.98:0/2139115701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c30003c00 con 0x559857684270 2024-04-25T17:39:58.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.827+0000 7f6c23fff700 1 -- 172.21.15.98:0/2139115701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c3001a040 con 0x559857684270 2024-04-25T17:39:58.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.827+0000 7f6c23fff700 1 -- 172.21.15.98:0/2139115701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c30004dd0 con 0x559857684270 2024-04-25T17:39:58.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/2139115701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 msgr2=0x55985768bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:58.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/2139115701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985768bfc0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x55985767a410 tx=0x7f6c30009400 comp rx=0 tx=0).stop 2024-04-25T17:39:58.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/2139115701 shutdown_connections 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/2139115701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985768bfc0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/2139115701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559857678a80 0x5598576ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/2139115701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5598576793a0 0x5598576acc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/2139115701 >> 172.21.15.98:0/2139115701 conn(0x5598575a6820 msgr2=0x559857682df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/2139115701 shutdown_connections 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/2139115701 wait complete. 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 -- start start 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 Processor -- start 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 0x559857688d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5598576793a0 0x5598576892b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985770afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38071df0 con 0x559857684270 2024-04-25T17:39:58.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38073260 con 0x5598576793a0 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c380724c0 con 0x559857678a80 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.828+0000 7f6c4adb3b80 10 client.0 _ll_register_callbacks cb 0x559857564ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3eac7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985770afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3e2c6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 0x559857688d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3e2c6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 0x559857688d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:43828/0 (socket says 172.21.15.98:43828) 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3eac7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985770afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48290/0 (socket says 172.21.15.98:48290) 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3e2c6700 1 -- 172.21.15.98:0/4150930049 learned_addr learned my addr 172.21.15.98:0/4150930049 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5598576793a0 0x5598576892b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3e2c6700 1 -- 172.21.15.98:0/4150930049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5598576793a0 msgr2=0x5598576892b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5598576793a0 0x5598576892b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3e2c6700 1 -- 172.21.15.98:0/4150930049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 msgr2=0x55985770afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:39:58.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985770afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3e2c6700 1 -- 172.21.15.98:0/4150930049 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55985770f920 con 0x559857678a80 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3eac7700 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985770afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.829+0000 7f6c3e2c6700 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 0x559857688d70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6c34005f50 tx=0x7f6c34009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18d470b463154ccc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.830+0000 7f6c21ffb700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.830+0000 7f6c21ffb700 1 -- 172.21.15.98:0/4150930049 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34037070 con 0x559857678a80 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.830+0000 7f6c21ffb700 1 -- 172.21.15.98:0/4150930049 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f6c34033600 con 0x559857678a80 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.830+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4150930049 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c3001a040 con 0x559857678a80 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.830+0000 7f6c21ffb700 1 -- 172.21.15.98:0/4150930049 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6c34043050 con 0x559857678a80 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c4adb3b80 10 client.25285 fetch_fsmap learned FSMap version 31 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4150930049 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559857710aa0 con 0x559857678a80 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c21ffb700 1 -- 172.21.15.98:0/4150930049 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c3001a040 con 0x559857678a80 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c21ffb700 1 -- 172.21.15.98:0/4150930049 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6c3400f460 con 0x559857678a80 2024-04-25T17:39:58.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c4adb3b80 10 client.25285 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:39:58.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c4adb3b80 10 client.25285 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:39:58.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c4adb3b80 20 client.25285 populate_metadata read hostname 'smithi098' 2024-04-25T17:39:58.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5598576a4ab0 0x559857713220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:39:58.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c4adb3b80 4 client.25285 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.25, "await_for": 42.0}] 2024-04-25T17:39:58.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4150930049 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.25, "await_for": 42.0}) -- 0x5598575d3750 con 0x5598576a4ab0 2024-04-25T17:39:58.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.831+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5598576a4ab0 0x559857713220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:39:58.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.832+0000 7f6c21ffb700 10 client.25285 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:58.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:58.832+0000 7f6c3dac5700 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5598576a4ab0 0x559857713220 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d595502a253215b6 server_cookie=e6fe4e0ee777fc6a in_seq=0 out_seq=0 2024-04-25T17:39:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c21ffb700 1 -- 172.21.15.98:0/4150930049 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5598575d3750 con 0x5598576a4ab0 2024-04-25T17:39:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c21ffb700 10 client.25285 handle_command_reply: tid=0 2024-04-25T17:39:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 client.25285 shutdown 2024-04-25T17:39:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4150930049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 msgr2=0x559857688d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 0x559857688d70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6c34005f50 tx=0x7f6c34009ca0 comp rx=0 tx=0).stop 2024-04-25T17:39:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4150930049 shutdown_connections 2024-04-25T17:39:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5598576a4ab0 0x559857713220 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559857678a80 0x559857688d70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559857684270 0x55985770afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 --2- 172.21.15.98:0/4150930049 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5598576793a0 0x5598576892b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4150930049 >> 172.21.15.98:0/4150930049 conn(0x5598575a6820 msgr2=0x5598576804e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c21ffb700 0 client.25285 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4150930049 shutdown_connections 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 1 -- 172.21.15.98:0/4150930049 wait complete. 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4adb3b80 20 client.25285 trim_cache size 0 max 16384 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 31, 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:39:59.051 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 10, 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "572a3f9e": { 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 10, 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 85.2, 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:39:59.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:39:59.053 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:39:59.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:39:59.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:39:59.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:39:59.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:39:59.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:39:59.053 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 -- 172.21.15.98:0/2891960798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6c240a3e50 msgr2=0x7f6c240a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6c240a3e50 0x7f6c240a6310 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6c34006010 tx=0x7f6c3400b280 comp rx=0 tx=0).stop 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 -- 172.21.15.98:0/2891960798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38103480 msgr2=0x7f6c38199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38103480 0x7f6c38199480 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6c2800e9c0 tx=0x7f6c2800ecd0 comp rx=0 tx=0).stop 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 -- 172.21.15.98:0/2891960798 shutdown_connections 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6c240a3e50 0x7f6c240a6310 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c38108a60 0x7f6c3819d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c38102ac0 0x7f6c38198f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 --2- 172.21.15.98:0/2891960798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c38103480 0x7f6c38199480 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 -- 172.21.15.98:0/2891960798 >> 172.21.15.98:0/2891960798 conn(0x7f6c38079b60 msgr2=0x7f6c38100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 -- 172.21.15.98:0/2891960798 shutdown_connections 2024-04-25T17:39:59.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:39:59.050+0000 7f6c4052a700 1 -- 172.21.15.98:0/2891960798 wait complete. 2024-04-25T17:39:59.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:39:59 smithi098 ceph-mon[29192]: pgmap v126: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 614 op/s 2024-04-25T17:39:59.830 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '572a3f9e', quiesce took 7.7 seconds. Will release after: 25.25 2024-04-25T17:39:59.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:39:59 smithi203 ceph-mon[32834]: pgmap v126: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 614 op/s 2024-04-25T17:40:00.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:39:59 smithi173 ceph-mon[32856]: pgmap v126: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 614 op/s 2024-04-25T17:40:00.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:00 smithi098 ceph-mon[29192]: overall HEALTH_OK 2024-04-25T17:40:00.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:00 smithi203 ceph-mon[32834]: overall HEALTH_OK 2024-04-25T17:40:01.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:00 smithi173 ceph-mon[32856]: overall HEALTH_OK 2024-04-25T17:40:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 210.0 s, 1398.4 tps, lat 32.314 ms stddev 34.267 2024-04-25T17:40:01.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:01 smithi098 ceph-mon[29192]: pgmap v127: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 8.2 MiB/s wr, 708 op/s 2024-04-25T17:40:01.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:01 smithi203 ceph-mon[32834]: pgmap v127: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 8.2 MiB/s wr, 708 op/s 2024-04-25T17:40:02.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:01 smithi173 ceph-mon[32856]: pgmap v127: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 8.2 MiB/s wr, 708 op/s 2024-04-25T17:40:02.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:02 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:40:02.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:02 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:40:02.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:02 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:40:02.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:02 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:40:02.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:02 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:40:02.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:02 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:40:02.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:02 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:40:02.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:02 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:40:03.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:02 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:40:03.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:02 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:40:03.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:02 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:40:03.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:02 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:40:03.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:03 smithi098 ceph-mon[29192]: pgmap v128: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.2 MiB/s wr, 734 op/s 2024-04-25T17:40:03.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:03 smithi203 ceph-mon[32834]: pgmap v128: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.2 MiB/s wr, 734 op/s 2024-04-25T17:40:04.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:03 smithi173 ceph-mon[32856]: pgmap v128: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.2 MiB/s wr, 734 op/s 2024-04-25T17:40:05.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:05 smithi098 ceph-mon[29192]: pgmap v129: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.2 MiB/s wr, 526 op/s 2024-04-25T17:40:05.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:05 smithi203 ceph-mon[32834]: pgmap v129: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.2 MiB/s wr, 526 op/s 2024-04-25T17:40:06.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:05 smithi173 ceph-mon[32856]: pgmap v129: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.2 MiB/s wr, 526 op/s 2024-04-25T17:40:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 215.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:40:07.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:07 smithi098 ceph-mon[29192]: pgmap v130: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.3 MiB/s wr, 532 op/s 2024-04-25T17:40:07.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:07 smithi203 ceph-mon[32834]: pgmap v130: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.3 MiB/s wr, 532 op/s 2024-04-25T17:40:08.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:07 smithi173 ceph-mon[32856]: pgmap v130: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.3 MiB/s wr, 532 op/s 2024-04-25T17:40:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.615+0000 7f2dc58e8640 1 -- 172.21.15.98:0/1027982393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc010c5a0 msgr2=0x7f2dc010e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.615+0000 7f2dc58e8640 1 --2- 172.21.15.98:0/1027982393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc010c5a0 0x7f2dc010e9d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009900 tx=0x7f2db802a480 comp rx=0 tx=0).stop 2024-04-25T17:40:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.616+0000 7f2dc58e8640 1 -- 172.21.15.98:0/1027982393 shutdown_connections 2024-04-25T17:40:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.616+0000 7f2dc58e8640 1 --2- 172.21.15.98:0/1027982393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc010c5a0 0x7f2dc010e9d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.616+0000 7f2dc58e8640 1 --2- 172.21.15.98:0/1027982393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dc0103af0 0x7f2dc010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.616+0000 7f2dc58e8640 1 --2- 172.21.15.98:0/1027982393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc01031d0 0x7f2dc01035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.616+0000 7f2dc58e8640 1 -- 172.21.15.98:0/1027982393 >> 172.21.15.98:0/1027982393 conn(0x7f2dc0077fa0 msgr2=0x7f2dc00783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.616+0000 7f2dc58e8640 1 -- 172.21.15.98:0/1027982393 shutdown_connections 2024-04-25T17:40:08.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.616+0000 7f2dc58e8640 1 -- 172.21.15.98:0/1027982393 wait complete. 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.616+0000 7f2dc58e8640 1 Processor -- start 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dc58e8640 1 -- start start 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dc58e8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dc01031d0 0x7f2dc01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dc58e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc0103af0 0x7f2dc0198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dc58e8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc010c5a0 0x7f2dc019ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dc58e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc01121a0 con 0x7f2dc0103af0 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dc58e8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc0112020 con 0x7f2dc010c5a0 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dc58e8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc0112320 con 0x7f2dc01031d0 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbe7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc0103af0 0x7f2dc0198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbe7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc0103af0 0x7f2dc0198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32778/0 (socket says 172.21.15.98:32778) 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/3736523491 learned_addr learned my addr 172.21.15.98:0/3736523491 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:08.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dc01031d0 0x7f2dc01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc010c5a0 0x7f2dc019ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/3736523491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dc01031d0 msgr2=0x7f2dc01985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dc01031d0 0x7f2dc01985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/3736523491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc010c5a0 msgr2=0x7f2dc019ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc010c5a0 0x7f2dc019ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/3736523491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc019d590 con 0x7f2dc0103af0 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.617+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc010c5a0 0x7f2dc019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.618+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc0103af0 0x7f2dc0198ae0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2db400d950 tx=0x7f2db400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.618+0000 7f2dc48e6640 1 -- 172.21.15.98:0/3736523491 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db4017070 con 0x7f2dc0103af0 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.618+0000 7f2dc48e6640 1 -- 172.21.15.98:0/3736523491 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db4009c30 con 0x7f2dc0103af0 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.618+0000 7f2dc48e6640 1 -- 172.21.15.98:0/3736523491 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db400f620 con 0x7f2dc0103af0 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.618+0000 7f2dc58e8640 1 -- 172.21.15.98:0/3736523491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dc019d880 con 0x7f2dc0103af0 2024-04-25T17:40:08.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.618+0000 7f2dc58e8640 1 -- 172.21.15.98:0/3736523491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dc01a5110 con 0x7f2dc0103af0 2024-04-25T17:40:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dc48e6640 1 -- 172.21.15.98:0/3736523491 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2db4010460 con 0x7f2dc0103af0 2024-04-25T17:40:08.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dc8604740 1 Processor -- start 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dc8604740 1 -- start start 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf26b620 0x55f6bf26ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf2762c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf276800 0x55f6bf278bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dc8604740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf1d0970 con 0x55f6bf26bf40 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dc8604740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf16d4a0 con 0x55f6bf26b620 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dc8604740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf1cc930 con 0x55f6bf276800 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dbeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf2762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dbeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf2762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32790/0 (socket says 172.21.15.98:32790) 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dbeffd640 1 -- 172.21.15.98:0/2415162335 learned_addr learned my addr 172.21.15.98:0/2415162335 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dc48e6640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2da80a3e40 0x7f2da80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/2415162335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf276800 0x55f6bf278bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/2415162335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf26b620 0x55f6bf26ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.621+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2da80a3e40 0x7f2da80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2dc48e6640 1 -- 172.21.15.98:0/3736523491 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f2db40d0390 con 0x7f2dc0103af0 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2dbeffd640 1 -- 172.21.15.98:0/2415162335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf276800 msgr2=0x55f6bf278bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/2415162335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf276800 0x55f6bf278bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2dbeffd640 1 -- 172.21.15.98:0/2415162335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf26b620 msgr2=0x55f6bf26ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/2415162335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf26b620 0x55f6bf26ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2dbeffd640 1 -- 172.21.15.98:0/2415162335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6beee3210 con 0x55f6bf26bf40 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/2415162335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf2762c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2db0007ed0 tx=0x7f2db000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326b2c176a8b7c4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2da80a3e40 0x7f2da80a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2db802a9a0 tx=0x7f2db8005fd0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2da5ffb640 1 -- 172.21.15.98:0/2415162335 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0014070 con 0x55f6bf26bf40 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2da5ffb640 1 -- 172.21.15.98:0/2415162335 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db0004d20 con 0x55f6bf26bf40 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.622+0000 7f2da5ffb640 1 -- 172.21.15.98:0/2415162335 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db00053e0 con 0x55f6bf26bf40 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.623+0000 7f2dc8604740 1 -- 172.21.15.98:0/2415162335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 msgr2=0x55f6bf2762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.623+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2415162335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf2762c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2db0007ed0 tx=0x7f2db000e720 comp rx=0 tx=0).stop 2024-04-25T17:40:08.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.623+0000 7f2dc8604740 1 -- 172.21.15.98:0/2415162335 shutdown_connections 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.623+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2415162335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf276800 0x55f6bf278bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.623+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2415162335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf2762c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.623+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2415162335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf26b620 0x55f6bf26ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.623+0000 7f2dc8604740 1 -- 172.21.15.98:0/2415162335 >> 172.21.15.98:0/2415162335 conn(0x55f6bf267b30 msgr2=0x55f6bf267f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.623+0000 7f2dc8604740 1 -- 172.21.15.98:0/2415162335 shutdown_connections 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.623+0000 7f2dc8604740 1 -- 172.21.15.98:0/2415162335 wait complete. 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.623+0000 7f2dc8604740 1 -- start start 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dc8604740 1 Processor -- start 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26b620 0x55f6bf2760f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26bf40 0x55f6bf2733e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf273920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dc8604740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf279430 con 0x55f6bf26b620 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dc8604740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf16b0b0 con 0x55f6bf276800 2024-04-25T17:40:08.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dc8604740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf2795b0 con 0x55f6bf26bf40 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dc8604740 10 client.0 _ll_register_callbacks cb 0x55f6bf158d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26b620 0x55f6bf2760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26b620 0x55f6bf2760f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32798/0 (socket says 172.21.15.98:32798) 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbeffd640 1 -- 172.21.15.98:0/3822033703 learned_addr learned my addr 172.21.15.98:0/3822033703 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26bf40 0x55f6bf2733e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf273920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbeffd640 1 -- 172.21.15.98:0/3822033703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26bf40 msgr2=0x55f6bf2733e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26bf40 0x55f6bf2733e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbeffd640 1 -- 172.21.15.98:0/3822033703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 msgr2=0x55f6bf273920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf273920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbeffd640 1 -- 172.21.15.98:0/3822033703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6befaf620 con 0x55f6bf26b620 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf273920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26b620 0x55f6bf2760f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2db0000f80 tx=0x7f2db0007300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e474fbb9e73071 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:08.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.624+0000 7f2d9effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2d9effd640 1 -- 172.21.15.98:0/3822033703 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0027060 con 0x55f6bf26b620 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2d9effd640 1 -- 172.21.15.98:0/3822033703 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f2db0010640 con 0x55f6bf26b620 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2dc8604740 1 -- 172.21.15.98:0/3822033703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2db0004d20 con 0x55f6bf26b620 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2d9effd640 1 -- 172.21.15.98:0/3822033703 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2db0014070 con 0x55f6bf26b620 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2dc8604740 10 client.15762 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2dc8604740 1 -- 172.21.15.98:0/3822033703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f6bf285300 con 0x55f6bf26b620 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2d9effd640 1 -- 172.21.15.98:0/3822033703 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db0004d20 con 0x55f6bf26b620 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2d9effd640 1 -- 172.21.15.98:0/3822033703 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f2db00055d0 con 0x55f6bf26b620 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2dc8604740 10 client.15762 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2dc8604740 10 client.15762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2dc8604740 20 client.15762 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2dc8604740 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f6bf2866b0 0x55f6bf288aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2dc8604740 4 client.15762 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.625+0000 7f2dc8604740 1 -- 172.21.15.98:0/3822033703 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6bf1de390 con 0x55f6bf2866b0 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.626+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f6bf2866b0 0x55f6bf288aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.626+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f6bf2866b0 0x55f6bf288aa0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd54170cc21e26bf server_cookie=75f19fe53cac6a11 in_seq=0 out_seq=0 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.626+0000 7f2d9effd640 10 client.15762 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2d9effd640 1 -- 172.21.15.98:0/3822033703 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f2db4081ae0 con 0x55f6bf2866b0 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2d9effd640 10 client.15762 handle_command_reply: tid=0 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 client.15762 shutdown 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 -- 172.21.15.98:0/3822033703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26b620 msgr2=0x55f6bf2760f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26b620 0x55f6bf2760f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2db0000f80 tx=0x7f2db0007300 comp rx=0 tx=0).stop 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 -- 172.21.15.98:0/3822033703 shutdown_connections 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f6bf2866b0 0x55f6bf288aa0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf273920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26bf40 0x55f6bf2733e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2d9effd640 0 client.15762 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 --2- 172.21.15.98:0/3822033703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26b620 0x55f6bf2760f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 -- 172.21.15.98:0/3822033703 >> 172.21.15.98:0/3822033703 conn(0x55f6bf267b30 msgr2=0x55f6bf26d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 -- 172.21.15.98:0/3822033703 shutdown_connections 2024-04-25T17:40:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 1 -- 172.21.15.98:0/3822033703 wait complete. 2024-04-25T17:40:08.635 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.627+0000 7f2dc8604740 20 client.15762 trim_cache size 0 max 16384 2024-04-25T17:40:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dc8604740 1 Processor -- start 2024-04-25T17:40:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dc8604740 1 -- start start 2024-04-25T17:40:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26b620 0x55f6bf2822f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf28e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf28e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dc8604740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc01b0480 con 0x55f6bf26bf40 2024-04-25T17:40:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dc8604740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc01adc90 con 0x55f6bf276800 2024-04-25T17:40:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dc8604740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc01abae0 con 0x55f6bf26b620 2024-04-25T17:40:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dbe7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf28e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.648 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dbe7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf28e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32804/0 (socket says 172.21.15.98:32804) 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.642+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/2757223812 learned_addr learned my addr 172.21.15.98:0/2757223812 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/2757223812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26b620 0x55f6bf2822f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/2757223812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf28e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/2757223812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26b620 msgr2=0x55f6bf2822f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/2757223812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26b620 0x55f6bf2822f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/2757223812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 msgr2=0x55f6bf28e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/2757223812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf28e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/2757223812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6bf279550 con 0x55f6bf26bf40 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/2757223812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf28e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/2757223812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf28e0a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2db40cdb70 tx=0x7f2db4098730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3ebd6bef1361a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2da57fa640 1 -- 172.21.15.98:0/2757223812 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db4020040 con 0x55f6bf26bf40 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2da57fa640 1 -- 172.21.15.98:0/2757223812 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db40b5070 con 0x55f6bf26bf40 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.643+0000 7f2da57fa640 1 -- 172.21.15.98:0/2757223812 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db40cdd90 con 0x55f6bf26bf40 2024-04-25T17:40:08.649 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 -- 172.21.15.98:0/2757223812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 msgr2=0x55f6bf28e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2757223812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf28e0a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2db40cdb70 tx=0x7f2db4098730 comp rx=0 tx=0).stop 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 -- 172.21.15.98:0/2757223812 shutdown_connections 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2757223812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf28e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2757223812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf28e0a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2757223812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26b620 0x55f6bf2822f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 -- 172.21.15.98:0/2757223812 >> 172.21.15.98:0/2757223812 conn(0x55f6bf267b30 msgr2=0x55f6bf266cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 -- 172.21.15.98:0/2757223812 shutdown_connections 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 -- 172.21.15.98:0/2757223812 wait complete. 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 -- start start 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 Processor -- start 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26b620 0x55f6bf27c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf27ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf2eb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc01ac150 con 0x55f6bf26bf40 2024-04-25T17:40:08.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc01ad5c0 con 0x55f6bf276800 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.644+0000 7f2dc8604740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc01ac820 con 0x55f6bf26b620 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dc8604740 10 client.0 _ll_register_callbacks cb 0x7f2db0007d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbe7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf27ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbe7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf27ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32818/0 (socket says 172.21.15.98:32818) 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/2535653698 learned_addr learned my addr 172.21.15.98:0/2535653698 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26b620 0x55f6bf27c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/2535653698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26b620 msgr2=0x55f6bf27c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26b620 0x55f6bf27c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/2535653698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 msgr2=0x55f6bf2eb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf2eb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf2eb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbe7fc640 1 -- 172.21.15.98:0/2535653698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6bf2ebe60 con 0x55f6bf26bf40 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbf7fe640 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf2eb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dbe7fc640 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf27ce20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2db409cc10 tx=0x7f2db4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628944ab39d307bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:08.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2d9effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2d9effd640 1 -- 172.21.15.98:0/2535653698 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db4020070 con 0x55f6bf26bf40 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2d9effd640 1 -- 172.21.15.98:0/2535653698 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f2db40989d0 con 0x55f6bf26bf40 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dc8604740 1 -- 172.21.15.98:0/2535653698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2db40b5070 con 0x55f6bf26bf40 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2d9effd640 1 -- 172.21.15.98:0/2535653698 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2db4009de0 con 0x55f6bf26bf40 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dc8604740 10 client.15768 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.645+0000 7f2dc8604740 1 -- 172.21.15.98:0/2535653698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f6bf2ed050 con 0x55f6bf26bf40 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2d9effd640 1 -- 172.21.15.98:0/2535653698 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db40b5070 con 0x55f6bf26bf40 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2d9effd640 1 -- 172.21.15.98:0/2535653698 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f2db40980b0 con 0x55f6bf26bf40 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2dc8604740 10 client.15768 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2dc8604740 10 client.15768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2dc8604740 20 client.15768 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f6bf2ed7b0 0x55f6bf2efbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2dc8604740 4 client.15768 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2dc8604740 1 -- 172.21.15.98:0/2535653698 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f6bf1d8200 con 0x55f6bf2ed7b0 2024-04-25T17:40:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f6bf2ed7b0 0x55f6bf2efbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:08.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2dbeffd640 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f6bf2ed7b0 0x55f6bf2efbd0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2f99470d31d2e61 server_cookie=3936e24c1fab4d9a in_seq=0 out_seq=0 2024-04-25T17:40:08.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.646+0000 7f2d9effd640 10 client.15768 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:08.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2d9effd640 1 -- 172.21.15.98:0/2535653698 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f6bf1d8200 con 0x55f6bf2ed7b0 2024-04-25T17:40:08.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2d9effd640 10 client.15768 handle_command_reply: tid=0 2024-04-25T17:40:08.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 client.15768 shutdown 2024-04-25T17:40:08.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 -- 172.21.15.98:0/2535653698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 msgr2=0x55f6bf27ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:08.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf27ce20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2db409cc10 tx=0x7f2db4000bc0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 -- 172.21.15.98:0/2535653698 shutdown_connections 2024-04-25T17:40:08.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f6bf2ed7b0 0x55f6bf2efbd0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f6bf276800 0x55f6bf2eb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2d9effd640 0 client.15768 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:08.663 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f6bf26bf40 0x55f6bf27ce20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 --2- 172.21.15.98:0/2535653698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f6bf26b620 0x55f6bf27c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 -- 172.21.15.98:0/2535653698 >> 172.21.15.98:0/2535653698 conn(0x55f6bf267b30 msgr2=0x55f6bf2812e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 -- 172.21.15.98:0/2535653698 shutdown_connections 2024-04-25T17:40:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.661+0000 7f2dc8604740 1 -- 172.21.15.98:0/2535653698 wait complete. 2024-04-25T17:40:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc8604740 20 client.15768 trim_cache size 0 max 16384 2024-04-25T17:40:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:40:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:40:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:40:08.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:40:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 -- 172.21.15.98:0/3736523491 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2da80a3e40 msgr2=0x7f2da80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2da80a3e40 0x7f2da80a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2db802a9a0 tx=0x7f2db8005fd0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 -- 172.21.15.98:0/3736523491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc0103af0 msgr2=0x7f2dc0198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:08.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc0103af0 0x7f2dc0198ae0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2db400d950 tx=0x7f2db400de10 comp rx=0 tx=0).stop 2024-04-25T17:40:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 -- 172.21.15.98:0/3736523491 shutdown_connections 2024-04-25T17:40:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f2da80a3e40 0x7f2da80a6300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2dc010c5a0 0x7f2dc019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dc0103af0 0x7f2dc0198ae0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 --2- 172.21.15.98:0/3736523491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dc01031d0 0x7f2dc01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 -- 172.21.15.98:0/3736523491 >> 172.21.15.98:0/3736523491 conn(0x7f2dc0077fa0 msgr2=0x7f2dc00fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 -- 172.21.15.98:0/3736523491 shutdown_connections 2024-04-25T17:40:08.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:08.662+0000 7f2dc58e8640 1 -- 172.21.15.98:0/3736523491 wait complete. 2024-04-25T17:40:09.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:09 smithi098 ceph-mon[29192]: pgmap v131: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.1 MiB/s wr, 229 op/s 2024-04-25T17:40:09.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:09 smithi203 ceph-mon[32834]: pgmap v131: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.1 MiB/s wr, 229 op/s 2024-04-25T17:40:10.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:09 smithi173 ceph-mon[32856]: pgmap v131: 97 pgs: 97 active+clean; 2.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 2.1 MiB/s wr, 229 op/s 2024-04-25T17:40:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 220.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:40:11.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:11 smithi098 ceph-mon[29192]: pgmap v132: 97 pgs: 97 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 4.3 MiB/s wr, 299 op/s 2024-04-25T17:40:11.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:11 smithi203 ceph-mon[32834]: pgmap v132: 97 pgs: 97 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 4.3 MiB/s wr, 299 op/s 2024-04-25T17:40:12.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:11 smithi173 ceph-mon[32856]: pgmap v132: 97 pgs: 97 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 4.3 MiB/s wr, 299 op/s 2024-04-25T17:40:13.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:13 smithi098 ceph-mon[29192]: pgmap v133: 97 pgs: 97 active+clean; 2.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 12 MiB/s wr, 479 op/s 2024-04-25T17:40:13.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:13 smithi203 ceph-mon[32834]: pgmap v133: 97 pgs: 97 active+clean; 2.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 12 MiB/s wr, 479 op/s 2024-04-25T17:40:14.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:13 smithi173 ceph-mon[32856]: pgmap v133: 97 pgs: 97 active+clean; 2.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 12 MiB/s wr, 479 op/s 2024-04-25T17:40:15.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:15 smithi098 ceph-mon[29192]: pgmap v134: 97 pgs: 97 active+clean; 2.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 350 op/s 2024-04-25T17:40:15.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:15 smithi203 ceph-mon[32834]: pgmap v134: 97 pgs: 97 active+clean; 2.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 350 op/s 2024-04-25T17:40:16.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:15 smithi173 ceph-mon[32856]: pgmap v134: 97 pgs: 97 active+clean; 2.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 11 MiB/s wr, 350 op/s 2024-04-25T17:40:16.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 225.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:40:17.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:17 smithi098 ceph-mon[29192]: pgmap v135: 97 pgs: 97 active+clean; 2.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 MiB/s wr, 666 op/s 2024-04-25T17:40:17.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:17 smithi203 ceph-mon[32834]: pgmap v135: 97 pgs: 97 active+clean; 2.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 MiB/s wr, 666 op/s 2024-04-25T17:40:18.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:17 smithi173 ceph-mon[32856]: pgmap v135: 97 pgs: 97 active+clean; 2.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 MiB/s wr, 666 op/s 2024-04-25T17:40:19.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:19 smithi098 ceph-mon[29192]: pgmap v136: 97 pgs: 97 active+clean; 2.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 MiB/s wr, 660 op/s 2024-04-25T17:40:19.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:19 smithi203 ceph-mon[32834]: pgmap v136: 97 pgs: 97 active+clean; 2.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 MiB/s wr, 660 op/s 2024-04-25T17:40:20.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:19 smithi173 ceph-mon[32856]: pgmap v136: 97 pgs: 97 active+clean; 2.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 MiB/s wr, 660 op/s 2024-04-25T17:40:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 230.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:40:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.760+0000 7f5972b3d640 1 -- 172.21.15.98:0/990718975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f596c1072a0 msgr2=0x7f596c10ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.760+0000 7f5972b3d640 1 --2- 172.21.15.98:0/990718975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f596c1072a0 0x7f596c10ee70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5964009900 tx=0x7f596402a410 comp rx=0 tx=0).stop 2024-04-25T17:40:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.760+0000 7f5972b3d640 1 -- 172.21.15.98:0/990718975 shutdown_connections 2024-04-25T17:40:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.760+0000 7f5972b3d640 1 --2- 172.21.15.98:0/990718975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f596c1072a0 0x7f596c10ee70 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.760+0000 7f5972b3d640 1 --2- 172.21.15.98:0/990718975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f596c102b80 0x7f596c106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.760+0000 7f5972b3d640 1 --2- 172.21.15.98:0/990718975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f596c1021d0 0x7f596c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.760+0000 7f5972b3d640 1 -- 172.21.15.98:0/990718975 >> 172.21.15.98:0/990718975 conn(0x7f596c0793e0 msgr2=0x7f596c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.760+0000 7f5972b3d640 1 -- 172.21.15.98:0/990718975 shutdown_connections 2024-04-25T17:40:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.760+0000 7f5972b3d640 1 -- 172.21.15.98:0/990718975 wait complete. 2024-04-25T17:40:21.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f5972b3d640 1 Processor -- start 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f5972b3d640 1 -- start start 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f5972b3d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f596c1021d0 0x7f596c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f5972b3d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f596c1072a0 0x7f596c198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f5972b3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f596c19cef0 0x7f596c19d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f5972b3d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1121f0 con 0x7f596c19cef0 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f5972b3d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f596c112070 con 0x7f596c1021d0 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f5972b3d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f596c112370 con 0x7f596c1072a0 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f59710da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f596c19cef0 0x7f596c19d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f59710da640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f596c19cef0 0x7f596c19d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47678/0 (socket says 172.21.15.98:47678) 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f59710da640 1 -- 172.21.15.98:0/3188665147 learned_addr learned my addr 172.21.15.98:0/3188665147 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f596bfff640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f596c1072a0 0x7f596c198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f59708d9640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f596c1021d0 0x7f596c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f59710da640 1 -- 172.21.15.98:0/3188665147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f596c1072a0 msgr2=0x7f596c198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:21.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f59710da640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f596c1072a0 0x7f596c198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f59710da640 1 -- 172.21.15.98:0/3188665147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f596c1021d0 msgr2=0x7f596c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.761+0000 7f59710da640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f596c1021d0 0x7f596c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.762+0000 7f59710da640 1 -- 172.21.15.98:0/3188665147 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f596c19d9e0 con 0x7f596c19cef0 2024-04-25T17:40:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.762+0000 7f59710da640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f596c19cef0 0x7f596c19d3a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5964004770 tx=0x7f5964004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.762+0000 7f5969ffb640 1 -- 172.21.15.98:0/3188665147 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596403f030 con 0x7f596c19cef0 2024-04-25T17:40:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.762+0000 7f5969ffb640 1 -- 172.21.15.98:0/3188665147 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5964003aa0 con 0x7f596c19cef0 2024-04-25T17:40:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.762+0000 7f5972b3d640 1 -- 172.21.15.98:0/3188665147 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f596c19dc70 con 0x7f596c19cef0 2024-04-25T17:40:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.762+0000 7f5969ffb640 1 -- 172.21.15.98:0/3188665147 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964035070 con 0x7f596c19cef0 2024-04-25T17:40:21.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.762+0000 7f5972b3d640 1 -- 172.21.15.98:0/3188665147 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f596c1a5430 con 0x7f596c19cef0 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.765+0000 7f5969ffb640 1 -- 172.21.15.98:0/3188665147 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5964042030 con 0x7f596c19cef0 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.765+0000 7f597584b740 1 Processor -- start 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.765+0000 7f597584b740 1 -- start start 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.765+0000 7f597584b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e055620 0x560c5e055a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.765+0000 7f597584b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e0602c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.765+0000 7f597584b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e060800 0x560c5e062bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.765+0000 7f597584b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560c5dfba970 con 0x560c5e060800 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.765+0000 7f597584b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560c5df574a0 con 0x560c5e055620 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.765+0000 7f597584b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560c5dfb6930 con 0x560c5e055f40 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f596bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e060800 0x560c5e062bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f596bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e060800 0x560c5e062bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47692/0 (socket says 172.21.15.98:47692) 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f596bfff640 1 -- 172.21.15.98:0/421903240 learned_addr learned my addr 172.21.15.98:0/421903240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:21.772 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f5969ffb640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59580a3dc0 0x7f59580a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f59708d9640 1 --2- 172.21.15.98:0/421903240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e0602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f596bfff640 1 --2- 172.21.15.98:0/421903240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e055620 0x560c5e055a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f59710da640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59580a3dc0 0x7f59580a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f596bfff640 1 -- 172.21.15.98:0/421903240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 msgr2=0x560c5e0602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f596bfff640 1 --2- 172.21.15.98:0/421903240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e0602c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f596bfff640 1 -- 172.21.15.98:0/421903240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e055620 msgr2=0x560c5e055a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f596bfff640 1 --2- 172.21.15.98:0/421903240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e055620 0x560c5e055a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f596bfff640 1 -- 172.21.15.98:0/421903240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c5dccd210 con 0x560c5e060800 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f5969ffb640 1 -- 172.21.15.98:0/3188665147 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f59640f0e20 con 0x7f596c19cef0 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f596bfff640 1 --2- 172.21.15.98:0/421903240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e060800 0x560c5e062bf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f595c012840 tx=0x7f595c009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e2fb727848f3d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f5962ffd640 1 -- 172.21.15.98:0/421903240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c010a70 con 0x560c5e060800 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f59710da640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59580a3dc0 0x7f59580a6280 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f596408f860 tx=0x7f596408f890 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f5962ffd640 1 -- 172.21.15.98:0/421903240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f595c010c10 con 0x560c5e060800 2024-04-25T17:40:21.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.766+0000 7f5962ffd640 1 -- 172.21.15.98:0/421903240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c00e380 con 0x560c5e060800 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.767+0000 7f597584b740 1 -- 172.21.15.98:0/421903240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e060800 msgr2=0x560c5e062bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.767+0000 7f597584b740 1 --2- 172.21.15.98:0/421903240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e060800 0x560c5e062bf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f595c012840 tx=0x7f595c009ac0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.767+0000 7f597584b740 1 -- 172.21.15.98:0/421903240 shutdown_connections 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.767+0000 7f597584b740 1 --2- 172.21.15.98:0/421903240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e060800 0x560c5e062bf0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.767+0000 7f597584b740 1 --2- 172.21.15.98:0/421903240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e0602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.767+0000 7f597584b740 1 --2- 172.21.15.98:0/421903240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e055620 0x560c5e055a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.767+0000 7f597584b740 1 -- 172.21.15.98:0/421903240 >> 172.21.15.98:0/421903240 conn(0x560c5e051b30 msgr2=0x560c5e051f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.767+0000 7f597584b740 1 -- 172.21.15.98:0/421903240 shutdown_connections 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.767+0000 7f597584b740 1 -- 172.21.15.98:0/421903240 wait complete. 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.767+0000 7f597584b740 1 -- start start 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f597584b740 1 Processor -- start 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f597584b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055620 0x560c5e060130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f597584b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055f40 0x560c5e05d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f597584b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e060800 0x560c5e05d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f597584b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560c5e063430 con 0x560c5e055f40 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f597584b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560c5df550b0 con 0x560c5e060800 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f597584b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560c5e0635b0 con 0x560c5e055620 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f597584b740 10 client.0 _ll_register_callbacks cb 0x560c5df42d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f59708d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055f40 0x560c5e05d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f59708d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055f40 0x560c5e05d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47696/0 (socket says 172.21.15.98:47696) 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f59708d9640 1 -- 172.21.15.98:0/2183677174 learned_addr learned my addr 172.21.15.98:0/2183677174 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f596bfff640 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055620 0x560c5e060130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f596bfff640 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e060800 0x560c5e05d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f59708d9640 1 -- 172.21.15.98:0/2183677174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055620 msgr2=0x560c5e060130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f59708d9640 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055620 0x560c5e060130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f59708d9640 1 -- 172.21.15.98:0/2183677174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e060800 msgr2=0x560c5e05d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:21.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f59708d9640 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e060800 0x560c5e05d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.768+0000 7f59708d9640 1 -- 172.21.15.98:0/2183677174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c5dd99620 con 0x560c5e055f40 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f59708d9640 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055f40 0x560c5e05d420 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f595400bfc0 tx=0x7f59540118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e475c1cc4c5e548 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f594bfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f594bfff640 1 -- 172.21.15.98:0/2183677174 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954004020 con 0x560c5e055f40 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f594bfff640 1 -- 172.21.15.98:0/2183677174 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f59540041c0 con 0x560c5e055f40 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f597584b740 1 -- 172.21.15.98:0/2183677174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f595c010c10 con 0x560c5e055f40 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f594bfff640 1 -- 172.21.15.98:0/2183677174 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f59540043c0 con 0x560c5e055f40 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f597584b740 10 client.15780 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f597584b740 1 -- 172.21.15.98:0/2183677174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560c5e06f400 con 0x560c5e055f40 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f594bfff640 1 -- 172.21.15.98:0/2183677174 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f595c010c10 con 0x560c5e055f40 2024-04-25T17:40:21.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f594bfff640 1 -- 172.21.15.98:0/2183677174 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5954003a30 con 0x560c5e055f40 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f597584b740 10 client.15780 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f597584b740 10 client.15780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f597584b740 20 client.15780 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f597584b740 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560c5e070910 0x560c5e072d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.769+0000 7f597584b740 4 client.15780 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.770+0000 7f597584b740 1 -- 172.21.15.98:0/2183677174 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c5dfc8390 con 0x560c5e070910 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.770+0000 7f596bfff640 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560c5e070910 0x560c5e072d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.770+0000 7f596bfff640 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560c5e070910 0x560c5e072d20 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6fab2a8ade6e8bc server_cookie=6033a9233e7d780a in_seq=0 out_seq=0 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.770+0000 7f594bfff640 10 client.15780 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f594bfff640 1 -- 172.21.15.98:0/2183677174 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560c5dfc8390 con 0x560c5e070910 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f594bfff640 10 client.15780 handle_command_reply: tid=0 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 client.15780 shutdown 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 -- 172.21.15.98:0/2183677174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055f40 msgr2=0x560c5e05d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:21.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055f40 0x560c5e05d420 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f595400bfc0 tx=0x7f59540118d0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 -- 172.21.15.98:0/2183677174 shutdown_connections 2024-04-25T17:40:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560c5e070910 0x560c5e072d20 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e060800 0x560c5e05d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f594bfff640 0 client.15780 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055f40 0x560c5e05d420 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 --2- 172.21.15.98:0/2183677174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055620 0x560c5e060130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 -- 172.21.15.98:0/2183677174 >> 172.21.15.98:0/2183677174 conn(0x560c5e051b30 msgr2=0x560c5e057860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 -- 172.21.15.98:0/2183677174 shutdown_connections 2024-04-25T17:40:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 1 -- 172.21.15.98:0/2183677174 wait complete. 2024-04-25T17:40:21.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.771+0000 7f597584b740 20 client.15780 trim_cache size 0 max 16384 2024-04-25T17:40:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f597584b740 1 Processor -- start 2024-04-25T17:40:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f597584b740 1 -- start start 2024-04-25T17:40:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f597584b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 0x560c5e06c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f597584b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e050b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f597584b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e05fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f597584b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1b0320 con 0x560c5e055620 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f597584b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1adb30 con 0x560c5e051b30 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f597584b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1ab9a0 con 0x560c5e055f40 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f59708d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e050b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f59708d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e050b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47706/0 (socket says 172.21.15.98:47706) 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.786+0000 7f59708d9640 1 -- 172.21.15.98:0/2306733342 learned_addr learned my addr 172.21.15.98:0/2306733342 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f59708d9640 1 -- 172.21.15.98:0/2306733342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 msgr2=0x560c5e05fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f596bfff640 1 --2- 172.21.15.98:0/2306733342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e05fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f59708d9640 1 --2- 172.21.15.98:0/2306733342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e05fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f596bfff640 1 --2- 172.21.15.98:0/2306733342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 0x560c5e06c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f59708d9640 1 -- 172.21.15.98:0/2306733342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 msgr2=0x560c5e06c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f59708d9640 1 --2- 172.21.15.98:0/2306733342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 0x560c5e06c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f59708d9640 1 -- 172.21.15.98:0/2306733342 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c5e05c5a0 con 0x560c5e055620 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f596bfff640 1 --2- 172.21.15.98:0/2306733342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 0x560c5e06c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f59708d9640 1 --2- 172.21.15.98:0/2306733342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e050b30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f595400bf90 tx=0x7f5954004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2a602a3d77025c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f59627fc640 1 -- 172.21.15.98:0/2306733342 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595401a3d0 con 0x560c5e055620 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f59627fc640 1 -- 172.21.15.98:0/2306733342 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f595401a570 con 0x560c5e055620 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f59627fc640 1 -- 172.21.15.98:0/2306733342 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954003a50 con 0x560c5e055620 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f597584b740 1 -- 172.21.15.98:0/2306733342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 msgr2=0x560c5e050b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.787+0000 7f597584b740 1 --2- 172.21.15.98:0/2306733342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e050b30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f595400bf90 tx=0x7f5954004280 comp rx=0 tx=0).stop 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 -- 172.21.15.98:0/2306733342 shutdown_connections 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 --2- 172.21.15.98:0/2306733342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e05fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 --2- 172.21.15.98:0/2306733342 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e050b30 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 --2- 172.21.15.98:0/2306733342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 0x560c5e06c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 -- 172.21.15.98:0/2306733342 >> 172.21.15.98:0/2306733342 conn(0x560c5e05dac0 msgr2=0x560c5e050420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 -- 172.21.15.98:0/2306733342 shutdown_connections 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 -- 172.21.15.98:0/2306733342 wait complete. 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 -- start start 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 Processor -- start 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 0x560c5e0d4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e0d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e0d2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1abff0 con 0x560c5e055620 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1ad460 con 0x560c5e051b30 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f596c1ac6c0 con 0x560c5e055f40 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f597584b740 10 client.0 _ll_register_callbacks cb 0x7f595400ba90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f59708d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e0d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f59708d9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e0d4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47720/0 (socket says 172.21.15.98:47720) 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.788+0000 7f59708d9640 1 -- 172.21.15.98:0/70162393 learned_addr learned my addr 172.21.15.98:0/70162393 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f596bfff640 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 0x560c5e0d4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f596bfff640 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e0d2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f59708d9640 1 -- 172.21.15.98:0/70162393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 msgr2=0x560c5e0d2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f59708d9640 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e0d2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f59708d9640 1 -- 172.21.15.98:0/70162393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 msgr2=0x560c5e0d4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f59708d9640 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 0x560c5e0d4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f59708d9640 1 -- 172.21.15.98:0/70162393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c5e0d3510 con 0x560c5e055620 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f596bfff640 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e0d2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f59708d9640 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e0d4650 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5954003630 tx=0x7f59540038c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f3255a4469879c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f594bfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f594bfff640 1 -- 172.21.15.98:0/70162393 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595401a3d0 con 0x560c5e055620 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f594bfff640 1 -- 172.21.15.98:0/70162393 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f5954005050 con 0x560c5e055620 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f597584b740 1 -- 172.21.15.98:0/70162393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f595401a570 con 0x560c5e055620 2024-04-25T17:40:21.795 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f594bfff640 1 -- 172.21.15.98:0/70162393 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f595400b410 con 0x560c5e055620 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f597584b740 10 client.15786 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f597584b740 1 -- 172.21.15.98:0/70162393 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560c5e0db980 con 0x560c5e055620 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f594bfff640 1 -- 172.21.15.98:0/70162393 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f595401a570 con 0x560c5e055620 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.789+0000 7f594bfff640 1 -- 172.21.15.98:0/70162393 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f59540107c0 con 0x560c5e055620 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.790+0000 7f597584b740 10 client.15786 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.790+0000 7f597584b740 10 client.15786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.790+0000 7f597584b740 20 client.15786 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.790+0000 7f597584b740 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560c5e060800 0x560c5e0de000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.790+0000 7f597584b740 4 client.15786 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.790+0000 7f597584b740 1 -- 172.21.15.98:0/70162393 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560c5dfc2200 con 0x560c5e060800 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.790+0000 7f596bfff640 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560c5e060800 0x560c5e0de000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.790+0000 7f596bfff640 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560c5e060800 0x560c5e0de000 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e7b4827059089d8 server_cookie=43eedd685e598457 in_seq=0 out_seq=0 2024-04-25T17:40:21.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.790+0000 7f594bfff640 10 client.15786 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f594bfff640 1 -- 172.21.15.98:0/70162393 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560c5dfc2200 con 0x560c5e060800 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f594bfff640 10 client.15786 handle_command_reply: tid=0 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 client.15786 shutdown 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 -- 172.21.15.98:0/70162393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 msgr2=0x560c5e0d4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e0d4650 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5954003630 tx=0x7f59540038c0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 -- 172.21.15.98:0/70162393 shutdown_connections 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560c5e060800 0x560c5e0de000 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560c5e055f40 0x560c5e0d2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560c5e055620 0x560c5e0d4650 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f594bfff640 0 client.15786 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 --2- 172.21.15.98:0/70162393 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560c5e051b30 0x560c5e0d4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 -- 172.21.15.98:0/70162393 >> 172.21.15.98:0/70162393 conn(0x560c5e05dac0 msgr2=0x560c5e050cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:21.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 -- 172.21.15.98:0/70162393 shutdown_connections 2024-04-25T17:40:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 1 -- 172.21.15.98:0/70162393 wait complete. 2024-04-25T17:40:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.802+0000 7f597584b740 20 client.15786 trim_cache size 0 max 16384 2024-04-25T17:40:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:40:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:40:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:40:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:40:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 -- 172.21.15.98:0/3188665147 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59580a3dc0 msgr2=0x7f59580a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59580a3dc0 0x7f59580a6280 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f596408f860 tx=0x7f596408f890 comp rx=0 tx=0).stop 2024-04-25T17:40:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 -- 172.21.15.98:0/3188665147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f596c19cef0 msgr2=0x7f596c19d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f596c19cef0 0x7f596c19d3a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5964004770 tx=0x7f5964004270 comp rx=0 tx=0).stop 2024-04-25T17:40:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 -- 172.21.15.98:0/3188665147 shutdown_connections 2024-04-25T17:40:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59580a3dc0 0x7f59580a6280 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f596c19cef0 0x7f596c19d3a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f596c1072a0 0x7f596c198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 --2- 172.21.15.98:0/3188665147 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f596c1021d0 0x7f596c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 -- 172.21.15.98:0/3188665147 >> 172.21.15.98:0/3188665147 conn(0x7f596c0793e0 msgr2=0x7f596c0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 -- 172.21.15.98:0/3188665147 shutdown_connections 2024-04-25T17:40:21.806 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:21.803+0000 7f5972b3d640 1 -- 172.21.15.98:0/3188665147 wait complete. 2024-04-25T17:40:22.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:22 smithi098 ceph-mon[29192]: pgmap v137: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 29 MiB/s wr, 701 op/s 2024-04-25T17:40:22.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:22 smithi203 ceph-mon[32834]: pgmap v137: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 29 MiB/s wr, 701 op/s 2024-04-25T17:40:22.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:22 smithi173 ceph-mon[32856]: pgmap v137: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 29 MiB/s wr, 701 op/s 2024-04-25T17:40:24.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:24 smithi098 ceph-mon[29192]: pgmap v138: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 727 op/s 2024-04-25T17:40:24.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:24 smithi203 ceph-mon[32834]: pgmap v138: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 727 op/s 2024-04-25T17:40:24.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:24 smithi173 ceph-mon[32856]: pgmap v138: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 727 op/s 2024-04-25T17:40:25.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:40:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:40:26.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:26 smithi098 ceph-mon[29192]: pgmap v139: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 22 MiB/s wr, 453 op/s 2024-04-25T17:40:26.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:26 smithi203 ceph-mon[32834]: pgmap v139: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 22 MiB/s wr, 453 op/s 2024-04-25T17:40:26.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:26 smithi173 ceph-mon[32856]: pgmap v139: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 22 MiB/s wr, 453 op/s 2024-04-25T17:40:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 235.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:40:27.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.238+0000 7fad48a10700 1 -- 172.21.15.98:0/3836291873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad44108a50 msgr2=0x7fad44108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:27.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.238+0000 7fad48a10700 1 --2- 172.21.15.98:0/3836291873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad44108a50 0x7fad44108e30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fad34009990 tx=0x7fad34009ca0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.238+0000 7fad48a10700 1 -- 172.21.15.98:0/3836291873 shutdown_connections 2024-04-25T17:40:27.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.238+0000 7fad48a10700 1 --2- 172.21.15.98:0/3836291873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad44103450 0x7fad4410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.239+0000 7fad48a10700 1 --2- 172.21.15.98:0/3836291873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad44108a50 0x7fad44108e30 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.239+0000 7fad48a10700 1 --2- 172.21.15.98:0/3836291873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad44102a90 0x7fad44102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.239+0000 7fad48a10700 1 -- 172.21.15.98:0/3836291873 >> 172.21.15.98:0/3836291873 conn(0x7fad44079ba0 msgr2=0x7fad44079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:27.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.239+0000 7fad48a10700 1 -- 172.21.15.98:0/3836291873 shutdown_connections 2024-04-25T17:40:27.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.239+0000 7fad48a10700 1 -- 172.21.15.98:0/3836291873 wait complete. 2024-04-25T17:40:27.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.239+0000 7fad48a10700 1 Processor -- start 2024-04-25T17:40:27.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.239+0000 7fad48a10700 1 -- start start 2024-04-25T17:40:27.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad48a10700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad44102a90 0x7fad44074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:27.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad48a10700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad44103450 0x7fad44075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:27.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad48a10700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad44108a50 0x7fad440757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:27.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad48a10700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad44112c00 con 0x7fad44108a50 2024-04-25T17:40:27.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad48a10700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fad44112a80 con 0x7fad44102a90 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad48a10700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fad44112d80 con 0x7fad44103450 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad42d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad44108a50 0x7fad440757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad42d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad44108a50 0x7fad440757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48210/0 (socket says 172.21.15.98:48210) 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad42d9d700 1 -- 172.21.15.98:0/482167466 learned_addr learned my addr 172.21.15.98:0/482167466 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad41d9b700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad44103450 0x7fad44075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad4259c700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad44102a90 0x7fad44074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad42d9d700 1 -- 172.21.15.98:0/482167466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad44103450 msgr2=0x7fad44075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad42d9d700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad44103450 0x7fad44075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad42d9d700 1 -- 172.21.15.98:0/482167466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad44102a90 msgr2=0x7fad44074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad42d9d700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad44102a90 0x7fad44074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad42d9d700 1 -- 172.21.15.98:0/482167466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad441a7640 con 0x7fad44108a50 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad4259c700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad44102a90 0x7fad44074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad42d9d700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad44108a50 0x7fad440757c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fad3c007ee0 tx=0x7fad3c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:27.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad337fe700 1 -- 172.21.15.98:0/482167466 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c00dbd0 con 0x7fad44108a50 2024-04-25T17:40:27.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad48a10700 1 -- 172.21.15.98:0/482167466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad441a7930 con 0x7fad44108a50 2024-04-25T17:40:27.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad337fe700 1 -- 172.21.15.98:0/482167466 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad3c00dd70 con 0x7fad44108a50 2024-04-25T17:40:27.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.240+0000 7fad337fe700 1 -- 172.21.15.98:0/482167466 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c005990 con 0x7fad44108a50 2024-04-25T17:40:27.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.241+0000 7fad48a10700 1 -- 172.21.15.98:0/482167466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad441a7ec0 con 0x7fad44108a50 2024-04-25T17:40:27.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.242+0000 7fad48a10700 1 -- 172.21.15.98:0/482167466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad44069730 con 0x7fad44108a50 2024-04-25T17:40:27.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.243+0000 7fad337fe700 1 -- 172.21.15.98:0/482167466 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fad3c01a020 con 0x7fad44108a50 2024-04-25T17:40:27.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.244+0000 7fad337fe700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fad2c0a3db0 0x7fad2c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:27.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.244+0000 7fad4259c700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fad2c0a3db0 0x7fad2c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:27.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.244+0000 7fad337fe700 1 -- 172.21.15.98:0/482167466 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fad3c0d0530 con 0x7fad44108a50 2024-04-25T17:40:27.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.244+0000 7fad4259c700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fad2c0a3db0 0x7fad2c0a6270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fad34006010 tx=0x7fad3400b540 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:27.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.248+0000 7fad337fe700 1 -- 172.21.15.98:0/482167466 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad3c010070 con 0x7fad44108a50 2024-04-25T17:40:27.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.490+0000 7fad48a10700 1 -- 172.21.15.98:0/482167466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad44076270 con 0x7fad44108a50 2024-04-25T17:40:27.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.490+0000 7fad337fe700 1 -- 172.21.15.98:0/482167466 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11883 (secure 0 0 0) 0x7fad3c09cc00 con 0x7fad44108a50 2024-04-25T17:40:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:40:27.491 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:39:15.018336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:40:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.492+0000 7fad48a10700 1 -- 172.21.15.98:0/482167466 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fad2c0a3db0 msgr2=0x7fad2c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.492+0000 7fad48a10700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fad2c0a3db0 0x7fad2c0a6270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fad34006010 tx=0x7fad3400b540 comp rx=0 tx=0).stop 2024-04-25T17:40:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.492+0000 7fad48a10700 1 -- 172.21.15.98:0/482167466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad44108a50 msgr2=0x7fad440757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.492+0000 7fad48a10700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad44108a50 0x7fad440757c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fad3c007ee0 tx=0x7fad3c00d3b0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.493+0000 7fad48a10700 1 -- 172.21.15.98:0/482167466 shutdown_connections 2024-04-25T17:40:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.493+0000 7fad48a10700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fad2c0a3db0 0x7fad2c0a6270 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.493+0000 7fad48a10700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad44108a50 0x7fad440757c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.493+0000 7fad48a10700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad44102a90 0x7fad44074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.493+0000 7fad48a10700 1 --2- 172.21.15.98:0/482167466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fad44103450 0x7fad44075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.493+0000 7fad48a10700 1 -- 172.21.15.98:0/482167466 >> 172.21.15.98:0/482167466 conn(0x7fad44079ba0 msgr2=0x7fad44101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.493+0000 7fad48a10700 1 -- 172.21.15.98:0/482167466 shutdown_connections 2024-04-25T17:40:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:27.493+0000 7fad48a10700 1 -- 172.21.15.98:0/482167466 wait complete. 2024-04-25T17:40:27.494 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 31 2024-04-25T17:40:28.193 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:28 smithi098 ceph-mon[29192]: pgmap v140: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 MiB/s wr, 458 op/s 2024-04-25T17:40:28.193 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:28 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/482167466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:40:28.222 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id 572a3f9e --release --await' 2024-04-25T17:40:28.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id 572a3f9e --release --await 2024-04-25T17:40:28.462 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:40:28.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:28 smithi203 ceph-mon[32834]: pgmap v140: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 MiB/s wr, 458 op/s 2024-04-25T17:40:28.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:28 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/482167466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:40:28.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:28 smithi173 ceph-mon[32856]: pgmap v140: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 MiB/s wr, 458 op/s 2024-04-25T17:40:28.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:28 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/482167466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:40:30.133 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:30 smithi098 ceph-mon[29192]: pgmap v141: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 142 op/s 2024-04-25T17:40:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.239+0000 7f9a26d13700 1 -- 172.21.15.98:0/1210604765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a20101400 msgr2=0x7f9a20101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.239+0000 7f9a26d13700 1 --2- 172.21.15.98:0/1210604765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a20101400 0x7f9a20101880 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009990 tx=0x7f9a0c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 -- 172.21.15.98:0/1210604765 shutdown_connections 2024-04-25T17:40:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 --2- 172.21.15.98:0/1210604765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a20101dc0 0x7f9a2010f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 --2- 172.21.15.98:0/1210604765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a20107b20 0x7f9a20107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 --2- 172.21.15.98:0/1210604765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a20101400 0x7f9a20101880 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 -- 172.21.15.98:0/1210604765 >> 172.21.15.98:0/1210604765 conn(0x7f9a20076c60 msgr2=0x7f9a20077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 -- 172.21.15.98:0/1210604765 shutdown_connections 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 -- 172.21.15.98:0/1210604765 wait complete. 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 Processor -- start 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 -- start start 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a20101400 0x7f9a20198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a20101dc0 0x7f9a20199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a20107b20 0x7f9a2019d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a20112c50 con 0x7f9a20107b20 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a20112ad0 con 0x7f9a20101400 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a26d13700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a20112dd0 con 0x7f9a20101dc0 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a252b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a20107b20 0x7f9a2019d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a252b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a20107b20 0x7f9a2019d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48234/0 (socket says 172.21.15.98:48234) 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.240+0000 7f9a252b0700 1 -- 172.21.15.98:0/2093872904 learned_addr learned my addr 172.21.15.98:0/2093872904 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a1ffff700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a20101dc0 0x7f9a20199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a20101400 0x7f9a20198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a252b0700 1 -- 172.21.15.98:0/2093872904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a20101dc0 msgr2=0x7f9a20199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a252b0700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a20101dc0 0x7f9a20199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a252b0700 1 -- 172.21.15.98:0/2093872904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a20101400 msgr2=0x7f9a20198fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a252b0700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a20101400 0x7f9a20198fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a252b0700 1 -- 172.21.15.98:0/2093872904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2019dfd0 con 0x7f9a20107b20 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a20101400 0x7f9a20198fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a252b0700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a20107b20 0x7f9a2019d890 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9a18007d70 tx=0x7f9a1800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a1dffb700 1 -- 172.21.15.98:0/2093872904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a18016020 con 0x7f9a20107b20 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a1dffb700 1 -- 172.21.15.98:0/2093872904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a1800ad40 con 0x7f9a20107b20 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a26d13700 1 -- 172.21.15.98:0/2093872904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2019e2c0 con 0x7f9a20107b20 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a1dffb700 1 -- 172.21.15.98:0/2093872904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a180115e0 con 0x7f9a20107b20 2024-04-25T17:40:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.241+0000 7f9a26d13700 1 -- 172.21.15.98:0/2093872904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a201a5b40 con 0x7f9a20107b20 2024-04-25T17:40:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.244+0000 7f9a1dffb700 1 -- 172.21.15.98:0/2093872904 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9a18019020 con 0x7f9a20107b20 2024-04-25T17:40:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.245+0000 7f9a1dffb700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a100a3e30 0x7f9a100a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.245+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a100a3e30 0x7f9a100a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.245+0000 7f9a1dffb700 1 -- 172.21.15.98:0/2093872904 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f9a180d0630 con 0x7f9a20107b20 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.245+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a100a3e30 0x7f9a100a62f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9a14005fd0 tx=0x7f9a14005e40 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.245+0000 7f9a3159cb80 1 Processor -- start 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a3159cb80 1 -- start start 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1a650 0x5557d9c1aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c1af70 0x5557d9c25930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c25e70 0x5557d9c28260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5557d9b6dd90 con 0x5557d9c1af70 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5557d9b14a30 con 0x5557d9c1a650 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5557d9b69d50 con 0x5557d9c25e70 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a1ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c1af70 0x5557d9c25930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a1ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c1af70 0x5557d9c25930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48250/0 (socket says 172.21.15.98:48250) 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a1ffff700 1 -- 172.21.15.98:0/742381093 learned_addr learned my addr 172.21.15.98:0/742381093 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:30.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/742381093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1a650 0x5557d9c1aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a252b0700 1 --2- 172.21.15.98:0/742381093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c25e70 0x5557d9c28260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a1ffff700 1 -- 172.21.15.98:0/742381093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c25e70 msgr2=0x5557d9c28260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a1ffff700 1 --2- 172.21.15.98:0/742381093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c25e70 0x5557d9c28260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a1ffff700 1 -- 172.21.15.98:0/742381093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1a650 msgr2=0x5557d9c1aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a1ffff700 1 --2- 172.21.15.98:0/742381093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1a650 0x5557d9c1aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a1ffff700 1 -- 172.21.15.98:0/742381093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557d9ad8840 con 0x5557d9c1af70 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.246+0000 7f9a1ffff700 1 --2- 172.21.15.98:0/742381093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c1af70 0x5557d9c25930 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009990 tx=0x7f9a0c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea83259ce64f9e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a0affd700 1 -- 172.21.15.98:0/742381093 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c00bb90 con 0x5557d9c1af70 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a0affd700 1 -- 172.21.15.98:0/742381093 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0c026070 con 0x5557d9c1af70 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a0affd700 1 -- 172.21.15.98:0/742381093 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c005510 con 0x5557d9c1af70 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a3159cb80 1 -- 172.21.15.98:0/742381093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c1af70 msgr2=0x5557d9c25930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/742381093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c1af70 0x5557d9c25930 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009990 tx=0x7f9a0c003680 comp rx=0 tx=0).stop 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a3159cb80 1 -- 172.21.15.98:0/742381093 shutdown_connections 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/742381093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c25e70 0x5557d9c28260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/742381093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1a650 0x5557d9c1aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/742381093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c1af70 0x5557d9c25930 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a3159cb80 1 -- 172.21.15.98:0/742381093 >> 172.21.15.98:0/742381093 conn(0x5557d9b47990 msgr2=0x5557d9c16b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a3159cb80 1 -- 172.21.15.98:0/742381093 shutdown_connections 2024-04-25T17:40:30.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.247+0000 7f9a3159cb80 1 -- 172.21.15.98:0/742381093 wait complete. 2024-04-25T17:40:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a3159cb80 1 -- start start 2024-04-25T17:40:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a3159cb80 1 Processor -- start 2024-04-25T17:40:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 0x5557d9c255d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 0x5557d9c228d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c22e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5557d9c288a0 con 0x5557d9c25e70 2024-04-25T17:40:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5557d9b276b0 con 0x5557d9c1af70 2024-04-25T17:40:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5557d9c28a20 con 0x5557d9c1a650 2024-04-25T17:40:30.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a3159cb80 10 client.0 _ll_register_callbacks cb 0x5557d9b06a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a252b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c22e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a252b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c22e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48254/0 (socket says 172.21.15.98:48254) 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.248+0000 7f9a252b0700 1 -- 172.21.15.98:0/211876306 learned_addr learned my addr 172.21.15.98:0/211876306 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 0x5557d9c255d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a1ffff700 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 0x5557d9c228d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a252b0700 1 -- 172.21.15.98:0/211876306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 msgr2=0x5557d9c255d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a252b0700 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 0x5557d9c255d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a252b0700 1 -- 172.21.15.98:0/211876306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 msgr2=0x5557d9c228d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a252b0700 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 0x5557d9c228d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a252b0700 1 -- 172.21.15.98:0/211876306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557d9abb9d0 con 0x5557d9c25e70 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a1ffff700 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 0x5557d9c228d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a252b0700 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c22e10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9a1809c6a0 tx=0x7f9a1809ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4008c2d4fe6937b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f99fffff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:40:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f99fffff700 1 -- 172.21.15.98:0/211876306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a180a5380 con 0x5557d9c25e70 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f99fffff700 1 -- 172.21.15.98:0/211876306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f9a18018d80 con 0x5557d9c25e70 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a3159cb80 1 -- 172.21.15.98:0/211876306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a0c026070 con 0x5557d9c25e70 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f99fffff700 1 -- 172.21.15.98:0/211876306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9a1809c970 con 0x5557d9c25e70 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a3159cb80 10 client.15804 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f9a3159cb80 1 -- 172.21.15.98:0/211876306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5557d9c33190 con 0x5557d9c25e70 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.249+0000 7f99fffff700 1 -- 172.21.15.98:0/211876306 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0c026070 con 0x5557d9c25e70 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.250+0000 7f99fffff700 1 -- 172.21.15.98:0/211876306 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f9a18004b20 con 0x5557d9c25e70 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.250+0000 7f9a3159cb80 10 client.15804 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.250+0000 7f9a3159cb80 10 client.15804 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.250+0000 7f9a3159cb80 20 client.15804 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.250+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557d9c35720 0x5557d9c37b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.250+0000 7f9a3159cb80 4 client.15804 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.250+0000 7f9a3159cb80 1 -- 172.21.15.98:0/211876306 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557d9b7b580 con 0x5557d9c35720 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.250+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557d9c35720 0x5557d9c37b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.250+0000 7f99fffff700 10 client.15804 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.250+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557d9c35720 0x5557d9c37b10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37d7030eaab84bea server_cookie=4c578823dbf91b50 in_seq=0 out_seq=0 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f99fffff700 1 -- 172.21.15.98:0/211876306 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557d9b7b580 con 0x5557d9c35720 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f99fffff700 10 client.15804 handle_command_reply: tid=0 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 client.15804 shutdown 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 -- 172.21.15.98:0/211876306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 msgr2=0x5557d9c22e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c22e10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9a1809c6a0 tx=0x7f9a1809ce90 comp rx=0 tx=0).stop 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 -- 172.21.15.98:0/211876306 shutdown_connections 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557d9c35720 0x5557d9c37b10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 0x5557d9c255d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c22e10 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f99fffff700 0 client.15804 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/211876306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 0x5557d9c228d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 -- 172.21.15.98:0/211876306 >> 172.21.15.98:0/211876306 conn(0x5557d9b47990 msgr2=0x5557d9c1cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:30.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 -- 172.21.15.98:0/211876306 shutdown_connections 2024-04-25T17:40:30.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 1 -- 172.21.15.98:0/211876306 wait complete. 2024-04-25T17:40:30.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.251+0000 7f9a3159cb80 20 client.15804 trim_cache size 0 max 16384 2024-04-25T17:40:30.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a3159cb80 1 Processor -- start 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a3159cb80 1 -- start start 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1a650 0x5557d9c24810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1af70 0x5557d9c3d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c3d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a201af960 con 0x5557d9c25e70 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a201ad130 con 0x5557d9c1a650 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a201aaf80 con 0x5557d9c1af70 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a252b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c3d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a252b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c3d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48264/0 (socket says 172.21.15.98:48264) 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a252b0700 1 -- 172.21.15.98:0/1465865567 learned_addr learned my addr 172.21.15.98:0/1465865567 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a1ffff700 1 --2- 172.21.15.98:0/1465865567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1af70 0x5557d9c3d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/1465865567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1a650 0x5557d9c24810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a252b0700 1 -- 172.21.15.98:0/1465865567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1af70 msgr2=0x5557d9c3d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:30.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a252b0700 1 --2- 172.21.15.98:0/1465865567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1af70 0x5557d9c3d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a252b0700 1 -- 172.21.15.98:0/1465865567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1a650 msgr2=0x5557d9c24810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a252b0700 1 --2- 172.21.15.98:0/1465865567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1a650 0x5557d9c24810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a252b0700 1 -- 172.21.15.98:0/1465865567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557d9c1c0b0 con 0x5557d9c25e70 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a1ffff700 1 --2- 172.21.15.98:0/1465865567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1af70 0x5557d9c3d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.271+0000 7f9a252b0700 1 --2- 172.21.15.98:0/1465865567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c3d8e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9a180990e0 tx=0x7f9a180a0c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d019f5f972d39b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a0a7fc700 1 -- 172.21.15.98:0/1465865567 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1809cb10 con 0x5557d9c25e70 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a0a7fc700 1 -- 172.21.15.98:0/1465865567 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a18018d80 con 0x5557d9c25e70 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a0a7fc700 1 -- 172.21.15.98:0/1465865567 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a180a5150 con 0x5557d9c25e70 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a3159cb80 1 -- 172.21.15.98:0/1465865567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 msgr2=0x5557d9c3d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/1465865567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c3d8e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9a180990e0 tx=0x7f9a180a0c40 comp rx=0 tx=0).stop 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a3159cb80 1 -- 172.21.15.98:0/1465865567 shutdown_connections 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/1465865567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c3d8e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/1465865567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1a650 0x5557d9c24810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/1465865567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1af70 0x5557d9c3d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a3159cb80 1 -- 172.21.15.98:0/1465865567 >> 172.21.15.98:0/1465865567 conn(0x5557d9b47990 msgr2=0x5557d9c24100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:30.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a3159cb80 1 -- 172.21.15.98:0/1465865567 shutdown_connections 2024-04-25T17:40:30.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a3159cb80 1 -- 172.21.15.98:0/1465865567 wait complete. 2024-04-25T17:40:30.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.272+0000 7f9a3159cb80 1 -- start start 2024-04-25T17:40:30.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a3159cb80 1 Processor -- start 2024-04-25T17:40:30.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 0x5557d9c2de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 0x5557d9c2a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a3159cb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c2a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a201ab5f0 con 0x5557d9c25e70 2024-04-25T17:40:30.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a201aca60 con 0x5557d9c1af70 2024-04-25T17:40:30.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a3159cb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a201abcc0 con 0x5557d9c1a650 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a3159cb80 10 client.0 _ll_register_callbacks cb 0x5557d9b06a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a252b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c2a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a252b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c2a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48266/0 (socket says 172.21.15.98:48266) 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a252b0700 1 -- 172.21.15.98:0/3155009008 learned_addr learned my addr 172.21.15.98:0/3155009008 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a1ffff700 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 0x5557d9c2a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 0x5557d9c2de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a252b0700 1 -- 172.21.15.98:0/3155009008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 msgr2=0x5557d9c2de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a252b0700 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 0x5557d9c2de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a252b0700 1 -- 172.21.15.98:0/3155009008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 msgr2=0x5557d9c2a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a252b0700 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 0x5557d9c2a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a252b0700 1 -- 172.21.15.98:0/3155009008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557d9c2ad00 con 0x5557d9c25e70 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.273+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 0x5557d9c2de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f9a252b0700 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c2a710 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9a1809a9c0 tx=0x7f9a180982f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c1e43879d587c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f99fffff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:40:30.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f99fffff700 1 -- 172.21.15.98:0/3155009008 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a18004cd0 con 0x5557d9c25e70 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f99fffff700 1 -- 172.21.15.98:0/3155009008 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f9a18098580 con 0x5557d9c25e70 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f9a3159cb80 1 -- 172.21.15.98:0/3155009008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a18018d80 con 0x5557d9c25e70 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f99fffff700 1 -- 172.21.15.98:0/3155009008 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9a18098780 con 0x5557d9c25e70 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f9a3159cb80 10 client.15810 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f9a3159cb80 1 -- 172.21.15.98:0/3155009008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5557d9c9ca60 con 0x5557d9c25e70 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f99fffff700 1 -- 172.21.15.98:0/3155009008 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a18018d80 con 0x5557d9c25e70 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f99fffff700 1 -- 172.21.15.98:0/3155009008 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f9a180d00d0 con 0x5557d9c25e70 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f9a3159cb80 10 client.15810 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f9a3159cb80 10 client.15810 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f9a3159cb80 20 client.15810 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557d9c9cec0 0x5557d9c9f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f9a3159cb80 4 client.15810 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "572a3f9e", "release": true, "await": true}] 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.274+0000 7f9a3159cb80 1 -- 172.21.15.98:0/3155009008 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "572a3f9e", "release": true, "await": true}) -- 0x5557d9b75620 con 0x5557d9c9cec0 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.275+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557d9c9cec0 0x5557d9c9f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:30.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.275+0000 7f99fffff700 10 client.15810 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:30.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.275+0000 7f9a24aaf700 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557d9c9cec0 0x5557d9c9f2b0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5f1bba52ae8641a server_cookie=d2852287fb2afbbe in_seq=0 out_seq=0 2024-04-25T17:40:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.285+0000 7f99fffff700 1 -- 172.21.15.98:0/3155009008 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5557d9b75620 con 0x5557d9c9cec0 2024-04-25T17:40:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.285+0000 7f99fffff700 10 client.15810 handle_command_reply: tid=0 2024-04-25T17:40:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.285+0000 7f9a3159cb80 1 client.15810 shutdown 2024-04-25T17:40:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.285+0000 7f9a3159cb80 1 -- 172.21.15.98:0/3155009008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 msgr2=0x5557d9c2a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.285+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c2a710 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9a1809a9c0 tx=0x7f9a180982f0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.286+0000 7f9a3159cb80 1 -- 172.21.15.98:0/3155009008 shutdown_connections 2024-04-25T17:40:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.286+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557d9c9cec0 0x5557d9c9f2b0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.286+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557d9c1a650 0x5557d9c2de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.286+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557d9c25e70 0x5557d9c2a710 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.286+0000 7f9a3159cb80 1 --2- 172.21.15.98:0/3155009008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557d9c1af70 0x5557d9c2a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.286+0000 7f9a3159cb80 1 -- 172.21.15.98:0/3155009008 >> 172.21.15.98:0/3155009008 conn(0x5557d9b47990 msgr2=0x5557d9c24990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.286+0000 7f99fffff700 0 client.15810 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.286+0000 7f9a3159cb80 1 -- 172.21.15.98:0/3155009008 shutdown_connections 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.286+0000 7f9a3159cb80 1 -- 172.21.15.98:0/3155009008 wait complete. 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.286+0000 7f9a3159cb80 20 client.15810 trim_cache size 0 max 16384 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 31, 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:40:30.287 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 12, 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "572a3f9e": { 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 12, 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 85.2, 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:40:30.288 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:40:30.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:40:30.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:40:30.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:40:30.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:40:30.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:40:30.289 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:40:30.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 -- 172.21.15.98:0/2093872904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a100a3e30 msgr2=0x7f9a100a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:30.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a100a3e30 0x7f9a100a62f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9a14005fd0 tx=0x7f9a14005e40 comp rx=0 tx=0).stop 2024-04-25T17:40:30.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 -- 172.21.15.98:0/2093872904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a20107b20 msgr2=0x7f9a2019d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:30.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a20107b20 0x7f9a2019d890 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9a18007d70 tx=0x7f9a1800a340 comp rx=0 tx=0).stop 2024-04-25T17:40:30.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 -- 172.21.15.98:0/2093872904 shutdown_connections 2024-04-25T17:40:30.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9a100a3e30 0x7f9a100a62f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a20107b20 0x7f9a2019d890 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a20101400 0x7f9a20198fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 --2- 172.21.15.98:0/2093872904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a20101dc0 0x7f9a20199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:30.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 -- 172.21.15.98:0/2093872904 >> 172.21.15.98:0/2093872904 conn(0x7f9a20076c60 msgr2=0x7f9a200ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:30.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 -- 172.21.15.98:0/2093872904 shutdown_connections 2024-04-25T17:40:30.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:30.287+0000 7f9a26d13700 1 -- 172.21.15.98:0/2093872904 wait complete. 2024-04-25T17:40:30.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:30 smithi203 ceph-mon[32834]: pgmap v141: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 142 op/s 2024-04-25T17:40:30.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:30 smithi173 ceph-mon[32856]: pgmap v141: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 142 op/s 2024-04-25T17:40:31.016 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '572a3f9e', seconds elapsed: 38.9 2024-04-25T17:40:31.017 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.25 2024-04-25T17:40:31.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:40:31.364 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:40:31.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 240.0 s, 673.1 tps, lat 973.511 ms stddev 5349.337 2024-04-25T17:40:32.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:32 smithi098 ceph-mon[29192]: pgmap v142: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 145 op/s 2024-04-25T17:40:32.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:32 smithi203 ceph-mon[32834]: pgmap v142: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 145 op/s 2024-04-25T17:40:32.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:32 smithi173 ceph-mon[32856]: pgmap v142: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 145 op/s 2024-04-25T17:40:33.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.287+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2866335399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28101940 msgr2=0x7fbf28101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:33.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.287+0000 7fbf2ec0a700 1 --2- 172.21.15.98:0/2866335399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28101940 0x7fbf28101dc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009990 tx=0x7fbf14009ca0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.287+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2866335399 shutdown_connections 2024-04-25T17:40:33.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.287+0000 7fbf2ec0a700 1 --2- 172.21.15.98:0/2866335399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf28102300 0x7fbf2810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.287+0000 7fbf2ec0a700 1 --2- 172.21.15.98:0/2866335399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28107990 0x7fbf28107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.287+0000 7fbf2ec0a700 1 --2- 172.21.15.98:0/2866335399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28101940 0x7fbf28101dc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.287+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2866335399 >> 172.21.15.98:0/2866335399 conn(0x7fbf280788f0 msgr2=0x7fbf28078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:33.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.289+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2866335399 shutdown_connections 2024-04-25T17:40:33.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.290+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2866335399 wait complete. 2024-04-25T17:40:33.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.290+0000 7fbf2ec0a700 1 Processor -- start 2024-04-25T17:40:33.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.291+0000 7fbf2ec0a700 1 -- start start 2024-04-25T17:40:33.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.291+0000 7fbf2ec0a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf28101940 0x7fbf28199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:33.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.291+0000 7fbf2ec0a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28102300 0x7fbf28199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:33.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.291+0000 7fbf2ec0a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28107990 0x7fbf2819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:33.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.291+0000 7fbf2ec0a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbf28111b70 con 0x7fbf28102300 2024-04-25T17:40:33.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.291+0000 7fbf2ec0a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbf281119f0 con 0x7fbf28107990 2024-04-25T17:40:33.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.291+0000 7fbf2ec0a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbf28111cf0 con 0x7fbf28101940 2024-04-25T17:40:33.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.291+0000 7fbf27fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28102300 0x7fbf28199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:33.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.291+0000 7fbf27fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28102300 0x7fbf28199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48284/0 (socket says 172.21.15.98:48284) 2024-04-25T17:40:33.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.291+0000 7fbf27fff700 1 -- 172.21.15.98:0/2510866831 learned_addr learned my addr 172.21.15.98:0/2510866831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:33.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.292+0000 7fbf2d1a7700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28107990 0x7fbf2819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:33.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.293+0000 7fbf2c9a6700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf28101940 0x7fbf28199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:33.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.293+0000 7fbf2d1a7700 1 -- 172.21.15.98:0/2510866831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf28101940 msgr2=0x7fbf28199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:33.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.293+0000 7fbf2d1a7700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf28101940 0x7fbf28199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.293+0000 7fbf2d1a7700 1 -- 172.21.15.98:0/2510866831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28102300 msgr2=0x7fbf28199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:33.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.293+0000 7fbf2d1a7700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28102300 0x7fbf28199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.293+0000 7fbf2d1a7700 1 -- 172.21.15.98:0/2510866831 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf2819df10 con 0x7fbf28107990 2024-04-25T17:40:33.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.293+0000 7fbf2d1a7700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28107990 0x7fbf2819d9d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000efc0 tx=0x7fbf2000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.295+0000 7fbf25ffb700 1 -- 172.21.15.98:0/2510866831 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf200092e0 con 0x7fbf28107990 2024-04-25T17:40:33.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.295+0000 7fbf25ffb700 1 -- 172.21.15.98:0/2510866831 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf20004bb0 con 0x7fbf28107990 2024-04-25T17:40:33.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.295+0000 7fbf25ffb700 1 -- 172.21.15.98:0/2510866831 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf20004280 con 0x7fbf28107990 2024-04-25T17:40:33.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.295+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2510866831 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf2819e1a0 con 0x7fbf28107990 2024-04-25T17:40:33.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.296+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2510866831 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf2819e690 con 0x7fbf28107990 2024-04-25T17:40:33.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.297+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2510866831 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf28069730 con 0x7fbf28107990 2024-04-25T17:40:33.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.298+0000 7fbf25ffb700 1 -- 172.21.15.98:0/2510866831 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbf2001b030 con 0x7fbf28107990 2024-04-25T17:40:33.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.298+0000 7fbf25ffb700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbf180a3f40 0x7fbf180a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:33.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.299+0000 7fbf2c9a6700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbf180a3f40 0x7fbf180a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:33.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.299+0000 7fbf25ffb700 1 -- 172.21.15.98:0/2510866831 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fbf200d01c0 con 0x7fbf28107990 2024-04-25T17:40:33.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.299+0000 7fbf2c9a6700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbf180a3f40 0x7fbf180a6400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009b10 tx=0x7fbf1c009260 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:33.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.303+0000 7fbf25ffb700 1 -- 172.21.15.98:0/2510866831 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf20022020 con 0x7fbf28107990 2024-04-25T17:40:33.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.666+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2510866831 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf281a3c70 con 0x7fbf28107990 2024-04-25T17:40:33.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.668+0000 7fbf25ffb700 1 -- 172.21.15.98:0/2510866831 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11883 (secure 0 0 0) 0x7fbf20010040 con 0x7fbf28107990 2024-04-25T17:40:33.668 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:40:33.668 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:39:15.018336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:40:33.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2510866831 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbf180a3f40 msgr2=0x7fbf180a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:33.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbf180a3f40 0x7fbf180a6400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009b10 tx=0x7fbf1c009260 comp rx=0 tx=0).stop 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2510866831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28107990 msgr2=0x7fbf2819d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28107990 0x7fbf2819d9d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000efc0 tx=0x7fbf2000c3a0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2510866831 shutdown_connections 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbf180a3f40 0x7fbf180a6400 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf28101940 0x7fbf28199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf28107990 0x7fbf2819d9d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 --2- 172.21.15.98:0/2510866831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf28102300 0x7fbf28199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2510866831 >> 172.21.15.98:0/2510866831 conn(0x7fbf280788f0 msgr2=0x7fbf280ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2510866831 shutdown_connections 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:33.671+0000 7fbf2ec0a700 1 -- 172.21.15.98:0/2510866831 wait complete. 2024-04-25T17:40:33.672 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 31 2024-04-25T17:40:34.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:34 smithi098 ceph-mon[29192]: pgmap v143: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 111 op/s 2024-04-25T17:40:34.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:34 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2510866831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:40:34.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:34 smithi203 ceph-mon[32834]: pgmap v143: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 111 op/s 2024-04-25T17:40:34.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:34 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2510866831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:40:34.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:34 smithi173 ceph-mon[32856]: pgmap v143: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 4.1 MiB/s wr, 111 op/s 2024-04-25T17:40:34.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:34 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2510866831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:40:34.872 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 85.25 --await-for 42.0' 2024-04-25T17:40:34.872 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 85.25 --await-for 42.0 2024-04-25T17:40:34.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.929+0000 7f518a060640 1 -- 172.21.15.98:0/662626911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51841043e0 msgr2=0x7f51841047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:34.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.929+0000 7f518a060640 1 --2- 172.21.15.98:0/662626911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51841043e0 0x7f51841047c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5174009900 tx=0x7f517402a440 comp rx=0 tx=0).stop 2024-04-25T17:40:34.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.929+0000 7f518a060640 1 -- 172.21.15.98:0/662626911 shutdown_connections 2024-04-25T17:40:34.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.929+0000 7f518a060640 1 --2- 172.21.15.98:0/662626911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51841094b0 0x7f5184111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.929+0000 7f518a060640 1 --2- 172.21.15.98:0/662626911 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5184104d90 0x7f5184108dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.929+0000 7f518a060640 1 --2- 172.21.15.98:0/662626911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51841043e0 0x7f51841047c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.929+0000 7f518a060640 1 -- 172.21.15.98:0/662626911 >> 172.21.15.98:0/662626911 conn(0x7f51840ffc60 msgr2=0x7f5184102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.929+0000 7f518a060640 1 -- 172.21.15.98:0/662626911 shutdown_connections 2024-04-25T17:40:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.929+0000 7f518a060640 1 -- 172.21.15.98:0/662626911 wait complete. 2024-04-25T17:40:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.930+0000 7f518a060640 1 Processor -- start 2024-04-25T17:40:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f518a060640 1 -- start start 2024-04-25T17:40:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f518a060640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51841043e0 0x7f51841a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f518a060640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51841094b0 0x7f51841a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f518a060640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f518419a8e0 0x7f518419ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f518a060640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5184114560 con 0x7f51841043e0 2024-04-25T17:40:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f518a060640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f51841143e0 con 0x7f518419a8e0 2024-04-25T17:40:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f518a060640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51841146e0 con 0x7f51841094b0 2024-04-25T17:40:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f51837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51841043e0 0x7f51841a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f5183fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f518419a8e0 0x7f518419ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f51837fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51841043e0 0x7f51841a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55042/0 (socket says 172.21.15.98:55042) 2024-04-25T17:40:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f51837fe640 1 -- 172.21.15.98:0/1773068339 learned_addr learned my addr 172.21.15.98:0/1773068339 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.931+0000 7f5182ffd640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51841094b0 0x7f51841a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f51837fe640 1 -- 172.21.15.98:0/1773068339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51841094b0 msgr2=0x7f51841a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f51837fe640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51841094b0 0x7f51841a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f51837fe640 1 -- 172.21.15.98:0/1773068339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f518419a8e0 msgr2=0x7f518419ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f51837fe640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f518419a8e0 0x7f518419ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f51837fe640 1 -- 172.21.15.98:0/1773068339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518419b400 con 0x7f51841043e0 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f5182ffd640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51841094b0 0x7f51841a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f51837fe640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51841043e0 0x7f51841a06e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f517402aa30 tx=0x7f5174003de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f5180ff9640 1 -- 172.21.15.98:0/1773068339 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f517403f070 con 0x7f51841043e0 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f518a060640 1 -- 172.21.15.98:0/1773068339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f518419b690 con 0x7f51841043e0 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f5180ff9640 1 -- 172.21.15.98:0/1773068339 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5174031d60 con 0x7f51841043e0 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f5180ff9640 1 -- 172.21.15.98:0/1773068339 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174035030 con 0x7f51841043e0 2024-04-25T17:40:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.932+0000 7f518a060640 1 -- 172.21.15.98:0/1773068339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51841a7890 con 0x7f51841043e0 2024-04-25T17:40:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.935+0000 7f5180ff9640 1 -- 172.21.15.98:0/1773068339 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5174042070 con 0x7f51841043e0 2024-04-25T17:40:34.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.936+0000 7f518cd72740 1 Processor -- start 2024-04-25T17:40:34.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.936+0000 7f5180ff9640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f516c0a3e40 0x7f516c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.936+0000 7f518cd72740 1 -- start start 2024-04-25T17:40:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.937+0000 7f5180ff9640 1 -- 172.21.15.98:0/1773068339 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f51740f1210 con 0x7f51841043e0 2024-04-25T17:40:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.937+0000 7f518cd72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76620 0x563737d76a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.937+0000 7f518cd72740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 0x563737d812c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.937+0000 7f518cd72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d81800 0x563737d83bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.937+0000 7f518cd72740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563737cdb970 con 0x563737d76f40 2024-04-25T17:40:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.937+0000 7f518cd72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563737c784a0 con 0x563737d81800 2024-04-25T17:40:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.937+0000 7f518cd72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563737cd7930 con 0x563737d76620 2024-04-25T17:40:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f5183fff640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f516c0a3e40 0x7f516c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f51837fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76620 0x563737d76a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f51837fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76620 0x563737d76a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:46214/0 (socket says 172.21.15.98:46214) 2024-04-25T17:40:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f51837fe640 1 -- 172.21.15.98:0/1039756404 learned_addr learned my addr 172.21.15.98:0/1039756404 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f5182ffd640 1 --2- 172.21.15.98:0/1039756404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 0x563737d812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f5183fff640 1 --2- 172.21.15.98:0/1039756404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d81800 0x563737d83bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f51837fe640 1 -- 172.21.15.98:0/1039756404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d81800 msgr2=0x563737d83bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f51837fe640 1 --2- 172.21.15.98:0/1039756404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d81800 0x563737d83bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f51837fe640 1 -- 172.21.15.98:0/1039756404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 msgr2=0x563737d812c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f51837fe640 1 --2- 172.21.15.98:0/1039756404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 0x563737d812c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f51837fe640 1 -- 172.21.15.98:0/1039756404 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637379ee210 con 0x563737d76620 2024-04-25T17:40:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.938+0000 7f5183fff640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f516c0a3e40 0x7f516c0a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f517c00a530 tx=0x7f517c00a930 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.939+0000 7f51837fe640 1 --2- 172.21.15.98:0/1039756404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76620 0x563737d76a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f51740b9b50 tx=0x7f5174009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb22fb71fecad219 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.939+0000 7f5169ffb640 1 -- 172.21.15.98:0/1039756404 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174049040 con 0x563737d76620 2024-04-25T17:40:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.939+0000 7f5169ffb640 1 -- 172.21.15.98:0/1039756404 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51740bd180 con 0x563737d76620 2024-04-25T17:40:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.939+0000 7f5169ffb640 1 -- 172.21.15.98:0/1039756404 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f517403b6a0 con 0x563737d76620 2024-04-25T17:40:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.940+0000 7f518cd72740 1 -- 172.21.15.98:0/1039756404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76620 msgr2=0x563737d76a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.940+0000 7f518cd72740 1 --2- 172.21.15.98:0/1039756404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76620 0x563737d76a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f51740b9b50 tx=0x7f5174009260 comp rx=0 tx=0).stop 2024-04-25T17:40:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.940+0000 7f518cd72740 1 -- 172.21.15.98:0/1039756404 shutdown_connections 2024-04-25T17:40:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.940+0000 7f518cd72740 1 --2- 172.21.15.98:0/1039756404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d81800 0x563737d83bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.940+0000 7f518cd72740 1 --2- 172.21.15.98:0/1039756404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 0x563737d812c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.940+0000 7f518cd72740 1 --2- 172.21.15.98:0/1039756404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76620 0x563737d76a00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.940+0000 7f518cd72740 1 -- 172.21.15.98:0/1039756404 >> 172.21.15.98:0/1039756404 conn(0x563737d72b30 msgr2=0x563737d72f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.941+0000 7f518cd72740 1 -- 172.21.15.98:0/1039756404 shutdown_connections 2024-04-25T17:40:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.941+0000 7f518cd72740 1 -- 172.21.15.98:0/1039756404 wait complete. 2024-04-25T17:40:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.941+0000 7f518cd72740 1 -- start start 2024-04-25T17:40:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.941+0000 7f518cd72740 1 Processor -- start 2024-04-25T17:40:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.942+0000 7f518cd72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d76620 0x563737d812a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.942+0000 7f518cd72740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 0x563737d7e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.942+0000 7f518cd72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737d7ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.942+0000 7f51837fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d76620 0x563737d812a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.942+0000 7f51837fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d76620 0x563737d812a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40746/0 (socket says 172.21.15.98:40746) 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.942+0000 7f51837fe640 1 -- 172.21.15.98:0/150956569 learned_addr learned my addr 172.21.15.98:0/150956569 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.942+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563737d8ca00 con 0x563737d76f40 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.942+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563737c760b0 con 0x563737d76620 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.942+0000 7f5183fff640 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737d7ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.943+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563737d8cb80 con 0x563737d81800 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.943+0000 7f5183fff640 1 -- 172.21.15.98:0/150956569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d76620 msgr2=0x563737d812a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.943+0000 7f5183fff640 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d76620 0x563737d812a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.943+0000 7f5183fff640 1 -- 172.21.15.98:0/150956569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 msgr2=0x563737d7e590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.943+0000 7f5183fff640 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 0x563737d7e590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.943+0000 7f5183fff640 1 -- 172.21.15.98:0/150956569 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563737aba620 con 0x563737d81800 2024-04-25T17:40:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.943+0000 7f518cd72740 10 client.0 _ll_register_callbacks cb 0x563737c63d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.943+0000 7f5183fff640 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737d7ec80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x563737d73cc0 tx=0x7f517c004a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cb42576bb40818f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.944+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51740bd180 con 0x563737d81800 2024-04-25T17:40:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.944+0000 7f5162ffd640 10 client.25345 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:40:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.944+0000 7f5162ffd640 1 -- 172.21.15.98:0/150956569 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f517c028060 con 0x563737d81800 2024-04-25T17:40:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.944+0000 7f5162ffd640 1 -- 172.21.15.98:0/150956569 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f517c014040 con 0x563737d81800 2024-04-25T17:40:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.944+0000 7f5162ffd640 1 -- 172.21.15.98:0/150956569 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f517c019bf0 con 0x563737d81800 2024-04-25T17:40:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.944+0000 7f518cd72740 10 client.25345 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.944+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=31}) -- 0x563737d98710 con 0x563737d81800 2024-04-25T17:40:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.945+0000 7f5162ffd640 1 -- 172.21.15.98:0/150956569 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f517c018050 con 0x563737d81800 2024-04-25T17:40:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.945+0000 7f518cd72740 10 client.25345 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.945+0000 7f518cd72740 10 client.25345 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:40:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.945+0000 7f518cd72740 20 client.25345 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.945+0000 7f518cd72740 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563737d99ac0 0x563737d9bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.945+0000 7f518cd72740 4 client.25345 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:40:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.945+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f51740bd180 con 0x563737d99ac0 2024-04-25T17:40:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.945+0000 7f51837fe640 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563737d99ac0 0x563737d9bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.946+0000 7f51837fe640 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563737d99ac0 0x563737d9bed0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3349caa39645464 server_cookie=d7edf6082fb2e5f0 in_seq=0 out_seq=0 2024-04-25T17:40:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.946+0000 7f5162ffd640 10 client.25345 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f5162ffd640 1 -- 172.21.15.98:0/150956569 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f51740bd180 con 0x563737d99ac0 2024-04-25T17:40:34.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f5162ffd640 10 client.25345 handle_command_reply: tid=0 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f518cd72740 1 client.25345 shutdown 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 msgr2=0x563737d7ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f518cd72740 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737d7ec80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x563737d73cc0 tx=0x7f517c004a60 comp rx=0 tx=0).stop 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 shutdown_connections 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f518cd72740 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563737d99ac0 0x563737d9bed0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f518cd72740 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737d7ec80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f518cd72740 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 0x563737d7e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f518cd72740 1 --2- 172.21.15.98:0/150956569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d76620 0x563737d812a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.947+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 >> 172.21.15.98:0/150956569 conn(0x563737d72b30 msgr2=0x563737d788b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.948+0000 7f5162ffd640 0 client.25345 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.948+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 shutdown_connections 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.948+0000 7f518cd72740 1 -- 172.21.15.98:0/150956569 wait complete. 2024-04-25T17:40:34.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.948+0000 7f518cd72740 20 client.25345 trim_cache size 0 max 16384 2024-04-25T17:40:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.981+0000 7f518cd72740 1 Processor -- start 2024-04-25T17:40:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.982+0000 7f518cd72740 1 -- start start 2024-04-25T17:40:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.982+0000 7f518cd72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d76620 0x563737d72430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.982+0000 7f518cd72740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 0x563737d80dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.982+0000 7f518cd72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737da17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:34.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.982+0000 7f518cd72740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51841ae360 con 0x563737d76f40 2024-04-25T17:40:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.982+0000 7f518cd72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f51841abb70 con 0x563737d76620 2024-04-25T17:40:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.982+0000 7f518cd72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51841a99e0 con 0x563737d81800 2024-04-25T17:40:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.983+0000 7f5183fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737da17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.983+0000 7f5183fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737da17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:46226/0 (socket says 172.21.15.98:46226) 2024-04-25T17:40:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.983+0000 7f5183fff640 1 -- 172.21.15.98:0/287403282 learned_addr learned my addr 172.21.15.98:0/287403282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.983+0000 7f5183fff640 1 -- 172.21.15.98:0/287403282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d76620 msgr2=0x563737d72430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:34.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.983+0000 7f5183fff640 1 --2- 172.21.15.98:0/287403282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d76620 0x563737d72430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.983+0000 7f5183fff640 1 -- 172.21.15.98:0/287403282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 msgr2=0x563737d80dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.983+0000 7f5183fff640 1 --2- 172.21.15.98:0/287403282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 0x563737d80dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.983+0000 7f5183fff640 1 -- 172.21.15.98:0/287403282 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563737d6e2e0 con 0x563737d81800 2024-04-25T17:40:34.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.984+0000 7f5183fff640 1 --2- 172.21.15.98:0/287403282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737da17e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f517c009700 tx=0x7f517c01b050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5acdd3b392d57020 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.988+0000 7f51697fa640 1 -- 172.21.15.98:0/287403282 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f517c003ad0 con 0x563737d81800 2024-04-25T17:40:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.988+0000 7f51697fa640 1 -- 172.21.15.98:0/287403282 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f517c032070 con 0x563737d81800 2024-04-25T17:40:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.989+0000 7f518cd72740 1 -- 172.21.15.98:0/287403282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 msgr2=0x563737da17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.989+0000 7f518cd72740 1 --2- 172.21.15.98:0/287403282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737da17e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f517c009700 tx=0x7f517c01b050 comp rx=0 tx=0).stop 2024-04-25T17:40:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.993+0000 7f518cd72740 1 -- 172.21.15.98:0/287403282 shutdown_connections 2024-04-25T17:40:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.993+0000 7f518cd72740 1 --2- 172.21.15.98:0/287403282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d81800 0x563737da17e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.993+0000 7f518cd72740 1 --2- 172.21.15.98:0/287403282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76f40 0x563737d80dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.993+0000 7f518cd72740 1 --2- 172.21.15.98:0/287403282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d76620 0x563737d72430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.993+0000 7f518cd72740 1 -- 172.21.15.98:0/287403282 >> 172.21.15.98:0/287403282 conn(0x563737d72b30 msgr2=0x563737d78860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.996+0000 7f518cd72740 1 -- 172.21.15.98:0/287403282 shutdown_connections 2024-04-25T17:40:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.996+0000 7f518cd72740 1 -- 172.21.15.98:0/287403282 wait complete. 2024-04-25T17:40:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.996+0000 7f518cd72740 1 -- start start 2024-04-25T17:40:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.997+0000 7f518cd72740 1 Processor -- start 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.999+0000 7f518cd72740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76620 0x563737d8d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.999+0000 7f518cd72740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76f40 0x563737d8dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.999+0000 7f518cd72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d81800 0x563737d8e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.999+0000 7f518cd72740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f51841aa030 con 0x563737d76620 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.999+0000 7f518cd72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f51841ab4a0 con 0x563737d81800 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.999+0000 7f518cd72740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51841aa700 con 0x563737d76f40 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:34.999+0000 7f518cd72740 10 client.0 _ll_register_callbacks cb 0x7f517c017510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.000+0000 7f5182ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76f40 0x563737d8dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.000+0000 7f5182ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76f40 0x563737d8dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:46230/0 (socket says 172.21.15.98:46230) 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.000+0000 7f5182ffd640 1 -- 172.21.15.98:0/3591990267 learned_addr learned my addr 172.21.15.98:0/3591990267 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.000+0000 7f51837fe640 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76620 0x563737d8d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.000+0000 7f5182ffd640 1 -- 172.21.15.98:0/3591990267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d81800 msgr2=0x563737d8e0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.000+0000 7f5182ffd640 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d81800 0x563737d8e0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.000+0000 7f5182ffd640 1 -- 172.21.15.98:0/3591990267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76620 msgr2=0x563737d8d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.000+0000 7f5182ffd640 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76620 0x563737d8d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.000+0000 7f5182ffd640 1 -- 172.21.15.98:0/3591990267 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563737dff050 con 0x563737d76f40 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.001+0000 7f5182ffd640 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76f40 0x563737d8dbb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5184100080 tx=0x7f517800f5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfc3f386f445f00a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.001+0000 7f5162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.001+0000 7f5162ffd640 1 -- 172.21.15.98:0/3591990267 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5178015410 con 0x563737d76f40 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.001+0000 7f518cd72740 1 -- 172.21.15.98:0/3591990267 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f517c032070 con 0x563737d76f40 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.001+0000 7f5162ffd640 1 -- 172.21.15.98:0/3591990267 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f51780155b0 con 0x563737d76f40 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.001+0000 7f5162ffd640 1 -- 172.21.15.98:0/3591990267 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5178023050 con 0x563737d76f40 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.001+0000 7f518cd72740 10 client.25351 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.001+0000 7f518cd72740 1 -- 172.21.15.98:0/3591990267 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563737e00380 con 0x563737d76f40 2024-04-25T17:40:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.001+0000 7f5162ffd640 1 -- 172.21.15.98:0/3591990267 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f517c032070 con 0x563737d76f40 2024-04-25T17:40:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.002+0000 7f5162ffd640 1 -- 172.21.15.98:0/3591990267 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5178013920 con 0x563737d76f40 2024-04-25T17:40:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.002+0000 7f518cd72740 10 client.25351 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.002+0000 7f518cd72740 10 client.25351 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:40:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.002+0000 7f518cd72740 20 client.25351 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.002+0000 7f518cd72740 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563737e00b50 0x563737e02f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.002+0000 7f518cd72740 4 client.25351 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:40:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.002+0000 7f518cd72740 1 -- 172.21.15.98:0/3591990267 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563737ce9390 con 0x563737e00b50 2024-04-25T17:40:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.002+0000 7f51837fe640 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563737e00b50 0x563737e02f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.003+0000 7f51837fe640 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563737e00b50 0x563737e02f70 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d586e2f9bf37e2a server_cookie=40d575c678f9a655 in_seq=0 out_seq=0 2024-04-25T17:40:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.003+0000 7f5162ffd640 10 client.25351 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f5162ffd640 1 -- 172.21.15.98:0/3591990267 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563737ce9390 con 0x563737e00b50 2024-04-25T17:40:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f5162ffd640 10 client.25351 handle_command_reply: tid=0 2024-04-25T17:40:35.014 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 client.25351 shutdown 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 -- 172.21.15.98:0/3591990267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76f40 msgr2=0x563737d8dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76f40 0x563737d8dbb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5184100080 tx=0x7f517800f5b0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 -- 172.21.15.98:0/3591990267 shutdown_connections 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563737e00b50 0x563737e02f70 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563737d81800 0x563737d8e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563737d76f40 0x563737d8dbb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 --2- 172.21.15.98:0/3591990267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563737d76620 0x563737d8d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f5162ffd640 0 client.25351 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 -- 172.21.15.98:0/3591990267 >> 172.21.15.98:0/3591990267 conn(0x563737d72b30 msgr2=0x563737d79140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 -- 172.21.15.98:0/3591990267 shutdown_connections 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.014+0000 7f518cd72740 1 -- 172.21.15.98:0/3591990267 wait complete. 2024-04-25T17:40:35.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.015+0000 7f518cd72740 20 client.25351 trim_cache size 0 max 16384 2024-04-25T17:40:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:40:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:40:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:40:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:40:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.015+0000 7f518a060640 1 -- 172.21.15.98:0/1773068339 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f516c0a3e40 msgr2=0x7f516c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.015+0000 7f518a060640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f516c0a3e40 0x7f516c0a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f517c00a530 tx=0x7f517c00a930 comp rx=0 tx=0).stop 2024-04-25T17:40:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.015+0000 7f518a060640 1 -- 172.21.15.98:0/1773068339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51841043e0 msgr2=0x7f51841a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.015+0000 7f518a060640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51841043e0 0x7f51841a06e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f517402aa30 tx=0x7f5174003de0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.015+0000 7f518a060640 1 -- 172.21.15.98:0/1773068339 shutdown_connections 2024-04-25T17:40:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.015+0000 7f518a060640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f516c0a3e40 0x7f516c0a6300 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.016+0000 7f518a060640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f518419a8e0 0x7f518419ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.016+0000 7f518a060640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51841094b0 0x7f51841a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.016+0000 7f518a060640 1 --2- 172.21.15.98:0/1773068339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f51841043e0 0x7f51841a06e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.016+0000 7f518a060640 1 -- 172.21.15.98:0/1773068339 >> 172.21.15.98:0/1773068339 conn(0x7f51840ffc60 msgr2=0x7f5184101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.016+0000 7f518a060640 1 -- 172.21.15.98:0/1773068339 shutdown_connections 2024-04-25T17:40:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:35.016+0000 7f518a060640 1 -- 172.21.15.98:0/1773068339 wait complete. 2024-04-25T17:40:35.261 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:40:36.426 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:36 smithi098 ceph-mon[29192]: pgmap v144: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 57 KiB/s wr, 15 op/s 2024-04-25T17:40:36.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:36 smithi203 ceph-mon[32834]: pgmap v144: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 57 KiB/s wr, 15 op/s 2024-04-25T17:40:36.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:36 smithi173 ceph-mon[32856]: pgmap v144: 97 pgs: 97 active+clean; 2.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 57 KiB/s wr, 15 op/s 2024-04-25T17:40:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 245.0 s, 2857.4 tps, lat 35.079 ms stddev 36.486 2024-04-25T17:40:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.422+0000 7fe5f7e11700 1 -- 172.21.15.98:0/769950750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5f0102330 msgr2=0x7fe5f010e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.422+0000 7fe5f7e11700 1 --2- 172.21.15.98:0/769950750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5f0102330 0x7fe5f010e7d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800b0a0 tx=0x7fe5e800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.422+0000 7fe5f7e11700 1 -- 172.21.15.98:0/769950750 shutdown_connections 2024-04-25T17:40:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.422+0000 7fe5f7e11700 1 --2- 172.21.15.98:0/769950750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5f0102330 0x7fe5f010e7d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.422+0000 7fe5f7e11700 1 --2- 172.21.15.98:0/769950750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f0106840 0x7fe5f0106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.422+0000 7fe5f7e11700 1 --2- 172.21.15.98:0/769950750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f0101970 0x7fe5f0101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.422+0000 7fe5f7e11700 1 -- 172.21.15.98:0/769950750 >> 172.21.15.98:0/769950750 conn(0x7fe5f0078980 msgr2=0x7fe5f0078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.422+0000 7fe5f7e11700 1 -- 172.21.15.98:0/769950750 shutdown_connections 2024-04-25T17:40:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.422+0000 7fe5f7e11700 1 -- 172.21.15.98:0/769950750 wait complete. 2024-04-25T17:40:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.423+0000 7fe5f7e11700 1 Processor -- start 2024-04-25T17:40:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.423+0000 7fe5f7e11700 1 -- start start 2024-04-25T17:40:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.423+0000 7fe5f7e11700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5f0101970 0x7fe5f0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.423+0000 7fe5f7e11700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f0102330 0x7fe5f01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.423+0000 7fe5f7e11700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f0106840 0x7fe5f019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.423+0000 7fe5f7e11700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f0111bf0 con 0x7fe5f0102330 2024-04-25T17:40:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.423+0000 7fe5f7e11700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f0111a70 con 0x7fe5f0101970 2024-04-25T17:40:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.423+0000 7fe5f7e11700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f0111d70 con 0x7fe5f0106840 2024-04-25T17:40:37.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.423+0000 7fe5f53ac700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f0102330 0x7fe5f01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.424+0000 7fe5f63ae700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f0106840 0x7fe5f019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.424+0000 7fe5f53ac700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f0102330 0x7fe5f01985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55086/0 (socket says 172.21.15.98:55086) 2024-04-25T17:40:37.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.424+0000 7fe5f53ac700 1 -- 172.21.15.98:0/2183661961 learned_addr learned my addr 172.21.15.98:0/2183661961 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:37.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.424+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5f0101970 0x7fe5f0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.424+0000 7fe5f53ac700 1 -- 172.21.15.98:0/2183661961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f0106840 msgr2=0x7fe5f019c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.424+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f0106840 0x7fe5f019c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.424+0000 7fe5f53ac700 1 -- 172.21.15.98:0/2183661961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5f0101970 msgr2=0x7fe5f0198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.424+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5f0101970 0x7fe5f0198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.424+0000 7fe5f53ac700 1 -- 172.21.15.98:0/2183661961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f019d070 con 0x7fe5f0102330 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.425+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f0102330 0x7fe5f01985a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000d950 tx=0x7fe5e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.425+0000 7fe5e6ffd700 1 -- 172.21.15.98:0/2183661961 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0009940 con 0x7fe5f0102330 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.425+0000 7fe5e6ffd700 1 -- 172.21.15.98:0/2183661961 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5e0027070 con 0x7fe5f0102330 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.426+0000 7fe5e6ffd700 1 -- 172.21.15.98:0/2183661961 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e000f3c0 con 0x7fe5f0102330 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.426+0000 7fe5f7e11700 1 -- 172.21.15.98:0/2183661961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f019d360 con 0x7fe5f0102330 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.426+0000 7fe5f7e11700 1 -- 172.21.15.98:0/2183661961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f006c460 con 0x7fe5f0102330 2024-04-25T17:40:37.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.425+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5f0101970 0x7fe5f0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:40:37.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.431+0000 7fe5e6ffd700 1 -- 172.21.15.98:0/2183661961 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe5e0009ae0 con 0x7fe5f0102330 2024-04-25T17:40:37.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.431+0000 7fe5e6ffd700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe5dc0a4080 0x7fe5dc0a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.432+0000 7fe5e6ffd700 1 -- 172.21.15.98:0/2183661961 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fe5e00e1500 con 0x7fe5f0102330 2024-04-25T17:40:37.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.432+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe5dc0a4080 0x7fe5dc0a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.432+0000 7fe60269ab80 1 Processor -- start 2024-04-25T17:40:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.432+0000 7fe60269ab80 1 -- start start 2024-04-25T17:40:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.432+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468191e20 0x562468192200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.432+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468192740 0x562468184760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468184cf0 0x562468185170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe60269ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5624680d6e20 con 0x562468192740 2024-04-25T17:40:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe60269ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56246807dac0 con 0x562468191e20 2024-04-25T17:40:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe60269ab80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5624680d2de0 con 0x562468184cf0 2024-04-25T17:40:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe5f53ac700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468192740 0x562468184760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe5f53ac700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468192740 0x562468184760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55088/0 (socket says 172.21.15.98:55088) 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe5f53ac700 1 -- 172.21.15.98:0/2079434952 learned_addr learned my addr 172.21.15.98:0/2079434952 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe5f63ae700 1 --2- 172.21.15.98:0/2079434952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468184cf0 0x562468185170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe5f53ac700 1 -- 172.21.15.98:0/2079434952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468184cf0 msgr2=0x562468185170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/2079434952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468184cf0 0x562468185170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe5f53ac700 1 -- 172.21.15.98:0/2079434952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468191e20 msgr2=0x562468192200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/2079434952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468191e20 0x562468192200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.433+0000 7fe5f53ac700 1 -- 172.21.15.98:0/2079434952 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624680418d0 con 0x562468192740 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.434+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/2079434952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468192740 0x562468184760 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00a9e40 tx=0x7fe5e00adbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18e14d9e7c10cde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.434+0000 7fe5dbfff700 1 -- 172.21.15.98:0/2079434952 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00c5020 con 0x562468192740 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.434+0000 7fe5dbfff700 1 -- 172.21.15.98:0/2079434952 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5e00b60e0 con 0x562468192740 2024-04-25T17:40:37.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.434+0000 7fe5dbfff700 1 -- 172.21.15.98:0/2079434952 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e00ad6d0 con 0x562468192740 2024-04-25T17:40:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.435+0000 7fe60269ab80 1 -- 172.21.15.98:0/2079434952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468192740 msgr2=0x562468184760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.435+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe5dc0a4080 0x7fe5dc0a6540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec0099b0 tx=0x7fe5ec008040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.435+0000 7fe60269ab80 1 --2- 172.21.15.98:0/2079434952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468192740 0x562468184760 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00a9e40 tx=0x7fe5e00adbf0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.436+0000 7fe60269ab80 1 -- 172.21.15.98:0/2079434952 shutdown_connections 2024-04-25T17:40:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.436+0000 7fe60269ab80 1 --2- 172.21.15.98:0/2079434952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468184cf0 0x562468185170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.436+0000 7fe60269ab80 1 --2- 172.21.15.98:0/2079434952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468191e20 0x562468192200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.436+0000 7fe60269ab80 1 --2- 172.21.15.98:0/2079434952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468192740 0x562468184760 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.436+0000 7fe60269ab80 1 -- 172.21.15.98:0/2079434952 >> 172.21.15.98:0/2079434952 conn(0x5624680b0a20 msgr2=0x56246817fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.436+0000 7fe60269ab80 1 -- 172.21.15.98:0/2079434952 shutdown_connections 2024-04-25T17:40:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.436+0000 7fe60269ab80 1 -- 172.21.15.98:0/2079434952 wait complete. 2024-04-25T17:40:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.437+0000 7fe60269ab80 1 -- start start 2024-04-25T17:40:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.437+0000 7fe60269ab80 1 Processor -- start 2024-04-25T17:40:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.437+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x562468199760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.437+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 0x562468196a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.437+0000 7fe5f5bad700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x562468199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.437+0000 7fe5f5bad700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x562468199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55098/0 (socket says 172.21.15.98:55098) 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.437+0000 7fe5f5bad700 1 -- 172.21.15.98:0/1241062827 learned_addr learned my addr 172.21.15.98:0/1241062827 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.437+0000 7fe60269ab80 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x562468197000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.437+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562468185d30 con 0x562468184cf0 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.437+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562468090740 con 0x562468192740 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562468185eb0 con 0x562468191e20 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe60269ab80 10 client.0 _ll_register_callbacks cb 0x56246806fab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5f63ae700 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x562468197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 0x562468196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5f5bad700 1 -- 172.21.15.98:0/1241062827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 msgr2=0x562468196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 0x562468196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5f5bad700 1 -- 172.21.15.98:0/1241062827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 msgr2=0x562468197000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x562468197000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5f5bad700 1 -- 172.21.15.98:0/1241062827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562468024a60 con 0x562468184cf0 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5f63ae700 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x562468197000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x562468199760 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00dca0 tx=0x7fe5ec004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac7c165f5b783cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5d97fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5d97fa700 1 -- 172.21.15.98:0/1241062827 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec024060 con 0x562468184cf0 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5e00b60e0 con 0x562468184cf0 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.438+0000 7fe5d97fa700 1 -- 172.21.15.98:0/1241062827 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fe5ec014ad0 con 0x562468184cf0 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.439+0000 7fe5d97fa700 1 -- 172.21.15.98:0/1241062827 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe5ec014cd0 con 0x562468184cf0 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.439+0000 7fe60269ab80 10 client.15837 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.439+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56246819c420 con 0x562468184cf0 2024-04-25T17:40:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.439+0000 7fe5d97fa700 1 -- 172.21.15.98:0/1241062827 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5e00b60e0 con 0x562468184cf0 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.440+0000 7fe5d97fa700 1 -- 172.21.15.98:0/1241062827 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fe5ec013070 con 0x562468184cf0 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.440+0000 7fe60269ab80 10 client.15837 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.440+0000 7fe60269ab80 10 client.15837 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.440+0000 7fe60269ab80 20 client.15837 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.440+0000 7fe60269ab80 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56246819ee00 0x5624681a1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.440+0000 7fe60269ab80 4 client.15837 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.440+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624680e4610 con 0x56246819ee00 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.440+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56246819ee00 0x5624681a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.441+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56246819ee00 0x5624681a1220 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47635cf5fb4c10ef server_cookie=83085795253ce205 in_seq=0 out_seq=0 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.441+0000 7fe5d97fa700 10 client.15837 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe5d97fa700 1 -- 172.21.15.98:0/1241062827 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5624680e4610 con 0x56246819ee00 2024-04-25T17:40:37.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe5d97fa700 10 client.15837 handle_command_reply: tid=0 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 client.15837 shutdown 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 msgr2=0x562468199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x562468199760 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec00dca0 tx=0x7fe5ec004800 comp rx=0 tx=0).stop 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 shutdown_connections 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56246819ee00 0x5624681a1220 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x562468197000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x562468199760 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 --2- 172.21.15.98:0/1241062827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 0x562468196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe5d97fa700 0 client.15837 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 >> 172.21.15.98:0/1241062827 conn(0x5624680b0a20 msgr2=0x5624681934f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 shutdown_connections 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 1 -- 172.21.15.98:0/1241062827 wait complete. 2024-04-25T17:40:37.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.442+0000 7fe60269ab80 20 client.15837 trim_cache size 0 max 16384 2024-04-25T17:40:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.472+0000 7fe60269ab80 1 Processor -- start 2024-04-25T17:40:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.472+0000 7fe60269ab80 1 -- start start 2024-04-25T17:40:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468184cf0 0x562468190fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468191e20 0x562468184380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468192740 0x5624681a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe60269ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f01a5260 con 0x562468191e20 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe60269ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f0072ac0 con 0x562468184cf0 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe60269ab80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f0070910 con 0x562468192740 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f5bad700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468184cf0 0x562468190fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f5bad700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468184cf0 0x562468190fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40778/0 (socket says 172.21.15.98:40778) 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f5bad700 1 -- 172.21.15.98:0/3658368455 learned_addr learned my addr 172.21.15.98:0/3658368455 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f63ae700 1 --2- 172.21.15.98:0/3658368455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468192740 0x5624681a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/3658368455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468191e20 0x562468184380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f5bad700 1 -- 172.21.15.98:0/3658368455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468192740 msgr2=0x5624681a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/3658368455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468192740 0x5624681a6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f5bad700 1 -- 172.21.15.98:0/3658368455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468191e20 msgr2=0x562468184380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/3658368455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468191e20 0x562468184380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f5bad700 1 -- 172.21.15.98:0/3658368455 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624681a18a0 con 0x562468184cf0 2024-04-25T17:40:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.473+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/3658368455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468184cf0 0x562468190fb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec028f60 tx=0x7fe5ec01e700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b1feda4edd6e0a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.474+0000 7fe5db7fe700 1 -- 172.21.15.98:0/3658368455 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec014710 con 0x562468184cf0 2024-04-25T17:40:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.474+0000 7fe5db7fe700 1 -- 172.21.15.98:0/3658368455 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5ec014ad0 con 0x562468184cf0 2024-04-25T17:40:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.474+0000 7fe5db7fe700 1 -- 172.21.15.98:0/3658368455 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec004280 con 0x562468184cf0 2024-04-25T17:40:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.475+0000 7fe60269ab80 1 -- 172.21.15.98:0/3658368455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468184cf0 msgr2=0x562468190fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.475+0000 7fe60269ab80 1 --2- 172.21.15.98:0/3658368455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468184cf0 0x562468190fb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec028f60 tx=0x7fe5ec01e700 comp rx=0 tx=0).stop 2024-04-25T17:40:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.475+0000 7fe60269ab80 1 -- 172.21.15.98:0/3658368455 shutdown_connections 2024-04-25T17:40:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.475+0000 7fe60269ab80 1 --2- 172.21.15.98:0/3658368455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468192740 0x5624681a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.475+0000 7fe60269ab80 1 --2- 172.21.15.98:0/3658368455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468184cf0 0x562468190fb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.475+0000 7fe60269ab80 1 --2- 172.21.15.98:0/3658368455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468191e20 0x562468184380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.475+0000 7fe60269ab80 1 -- 172.21.15.98:0/3658368455 >> 172.21.15.98:0/3658368455 conn(0x5624680b0a20 msgr2=0x562468183c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.475+0000 7fe60269ab80 1 -- 172.21.15.98:0/3658368455 shutdown_connections 2024-04-25T17:40:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.475+0000 7fe60269ab80 1 -- 172.21.15.98:0/3658368455 wait complete. 2024-04-25T17:40:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.476+0000 7fe60269ab80 1 -- start start 2024-04-25T17:40:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.476+0000 7fe60269ab80 1 Processor -- start 2024-04-25T17:40:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x56246818d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 0x56246818a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe60269ab80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x56246818aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe60269ab80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f0070f80 con 0x562468184cf0 2024-04-25T17:40:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe60269ab80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f00723f0 con 0x562468192740 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe60269ab80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f0071650 con 0x562468191e20 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe60269ab80 10 client.0 _ll_register_callbacks cb 0x56246806fab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f63ae700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x56246818aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f63ae700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x56246818aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40780/0 (socket says 172.21.15.98:40780) 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f63ae700 1 -- 172.21.15.98:0/3419583591 learned_addr learned my addr 172.21.15.98:0/3419583591 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 0x56246818a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x56246818d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f63ae700 1 -- 172.21.15.98:0/3419583591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 msgr2=0x56246818a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f63ae700 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 0x56246818a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f63ae700 1 -- 172.21.15.98:0/3419583591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 msgr2=0x56246818d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f63ae700 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x56246818d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f63ae700 1 -- 172.21.15.98:0/3419583591 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56246818b260 con 0x562468192740 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f53ac700 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 0x56246818a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:40:37.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x56246818d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:40:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5f63ae700 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x56246818aa20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800b070 tx=0x7fe5e8007900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc6a778ff1f914ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.477+0000 7fe5d97fa700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:40:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.478+0000 7fe5d97fa700 1 -- 172.21.15.98:0/3419583591 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e802b060 con 0x562468192740 2024-04-25T17:40:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.478+0000 7fe5d97fa700 1 -- 172.21.15.98:0/3419583591 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fe5e8007d70 con 0x562468192740 2024-04-25T17:40:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.478+0000 7fe60269ab80 1 -- 172.21.15.98:0/3419583591 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5ec014ad0 con 0x562468192740 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.478+0000 7fe5d97fa700 1 -- 172.21.15.98:0/3419583591 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe5e8030480 con 0x562468192740 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.479+0000 7fe60269ab80 10 client.25391 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.479+0000 7fe60269ab80 1 -- 172.21.15.98:0/3419583591 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562468205300 con 0x562468192740 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.479+0000 7fe5d97fa700 1 -- 172.21.15.98:0/3419583591 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5e8030ad0 con 0x562468192740 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.479+0000 7fe5d97fa700 1 -- 172.21.15.98:0/3419583591 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fe5e8025070 con 0x562468192740 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.479+0000 7fe60269ab80 10 client.25391 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.479+0000 7fe60269ab80 10 client.25391 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.479+0000 7fe60269ab80 20 client.25391 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.479+0000 7fe60269ab80 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562468205b30 0x562468207f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.479+0000 7fe60269ab80 4 client.25391 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.25, "await_for": 42.0}] 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.479+0000 7fe60269ab80 1 -- 172.21.15.98:0/3419583591 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.25, "await_for": 42.0}) -- 0x7fe5ec014ad0 con 0x562468205b30 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.480+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562468205b30 0x562468207f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.480+0000 7fe5d97fa700 10 client.25391 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.480+0000 7fe5f5bad700 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562468205b30 0x562468207f50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd5e45829ad5425e server_cookie=5df27ed33f004a9a in_seq=0 out_seq=0 2024-04-25T17:40:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.699+0000 7fe5d97fa700 1 -- 172.21.15.98:0/3419583591 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x7fe5ec014ad0 con 0x562468205b30 2024-04-25T17:40:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.699+0000 7fe5d97fa700 10 client.25391 handle_command_reply: tid=0 2024-04-25T17:40:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.699+0000 7fe60269ab80 1 client.25391 shutdown 2024-04-25T17:40:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.699+0000 7fe60269ab80 1 -- 172.21.15.98:0/3419583591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 msgr2=0x56246818aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.699+0000 7fe60269ab80 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x56246818aa20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800b070 tx=0x7fe5e8007900 comp rx=0 tx=0).stop 2024-04-25T17:40:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.699+0000 7fe60269ab80 1 -- 172.21.15.98:0/3419583591 shutdown_connections 2024-04-25T17:40:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.699+0000 7fe60269ab80 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562468205b30 0x562468207f50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.699+0000 7fe60269ab80 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562468192740 0x56246818aa20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5d97fa700 0 client.25391 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe60269ab80 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562468184cf0 0x56246818d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe60269ab80 1 --2- 172.21.15.98:0/3419583591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562468191e20 0x56246818a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe60269ab80 1 -- 172.21.15.98:0/3419583591 >> 172.21.15.98:0/3419583591 conn(0x5624680b0a20 msgr2=0x562468190e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe60269ab80 1 -- 172.21.15.98:0/3419583591 shutdown_connections 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe60269ab80 1 -- 172.21.15.98:0/3419583591 wait complete. 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe60269ab80 20 client.25391 trim_cache size 0 max 16384 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 31, 2024-04-25T17:40:37.701 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 14, 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "839b44ee": { 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 14, 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 85.2, 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:40:37.702 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:40:37.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:40:37.703 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:40:37.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:40:37.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:40:37.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:40:37.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:40:37.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:40:37.703 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5f7e11700 1 -- 172.21.15.98:0/2183661961 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe5dc0a4080 msgr2=0x7fe5dc0a6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5f7e11700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe5dc0a4080 0x7fe5dc0a6540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec0099b0 tx=0x7fe5ec008040 comp rx=0 tx=0).stop 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5f7e11700 1 -- 172.21.15.98:0/2183661961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f0102330 msgr2=0x7fe5f01985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5f7e11700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f0102330 0x7fe5f01985a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000d950 tx=0x7fe5e000dc60 comp rx=0 tx=0).stop 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5f7e11700 1 -- 172.21.15.98:0/2183661961 shutdown_connections 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5f7e11700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe5dc0a4080 0x7fe5dc0a6540 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5f7e11700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f0106840 0x7fe5f019c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5f7e11700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5f0101970 0x7fe5f0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5f7e11700 1 --2- 172.21.15.98:0/2183661961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5f0102330 0x7fe5f01985a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.700+0000 7fe5f7e11700 1 -- 172.21.15.98:0/2183661961 >> 172.21.15.98:0/2183661961 conn(0x7fe5f0078980 msgr2=0x7fe5f00ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.701+0000 7fe5f7e11700 1 -- 172.21.15.98:0/2183661961 shutdown_connections 2024-04-25T17:40:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:40:37.701+0000 7fe5f7e11700 1 -- 172.21.15.98:0/2183661961 wait complete. 2024-04-25T17:40:38.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:38 smithi098 ceph-mon[29192]: pgmap v145: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 2.9 MiB/s wr, 316 op/s 2024-04-25T17:40:38.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:38 smithi203 ceph-mon[32834]: pgmap v145: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 2.9 MiB/s wr, 316 op/s 2024-04-25T17:40:38.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:38 smithi173 ceph-mon[32856]: pgmap v145: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 2.9 MiB/s wr, 316 op/s 2024-04-25T17:40:38.590 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '839b44ee', quiesce took 7.6 seconds. Will release after: 25.25 2024-04-25T17:40:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:40 smithi098 ceph-mon[29192]: pgmap v146: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 2.9 MiB/s wr, 310 op/s 2024-04-25T17:40:40.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:40 smithi203 ceph-mon[32834]: pgmap v146: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 2.9 MiB/s wr, 310 op/s 2024-04-25T17:40:40.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:40 smithi173 ceph-mon[32856]: pgmap v146: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 2.9 MiB/s wr, 310 op/s 2024-04-25T17:40:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 250.0 s, 574.1 tps, lat 32.055 ms stddev 30.840 2024-04-25T17:40:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:42 smithi098 ceph-mon[29192]: pgmap v147: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 3.0 MiB/s wr, 334 op/s 2024-04-25T17:40:42.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:42 smithi203 ceph-mon[32834]: pgmap v147: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 3.0 MiB/s wr, 334 op/s 2024-04-25T17:40:42.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:42 smithi173 ceph-mon[32856]: pgmap v147: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 3.0 MiB/s wr, 334 op/s 2024-04-25T17:40:44.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:44 smithi098 ceph-mon[29192]: pgmap v148: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 4.1 MiB/s wr, 435 op/s 2024-04-25T17:40:44.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:44 smithi203 ceph-mon[32834]: pgmap v148: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 4.1 MiB/s wr, 435 op/s 2024-04-25T17:40:44.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:44 smithi173 ceph-mon[32856]: pgmap v148: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 4.1 MiB/s wr, 435 op/s 2024-04-25T17:40:46.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:46 smithi098 ceph-mon[29192]: pgmap v149: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 4.0 MiB/s wr, 428 op/s 2024-04-25T17:40:46.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:46 smithi203 ceph-mon[32834]: pgmap v149: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 4.0 MiB/s wr, 428 op/s 2024-04-25T17:40:46.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:46 smithi173 ceph-mon[32856]: pgmap v149: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 4.0 MiB/s wr, 428 op/s 2024-04-25T17:40:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 255.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:40:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.121+0000 7f592e79b640 1 -- 172.21.15.98:0/1256898774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928100d80 msgr2=0x7f5928101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.121+0000 7f592e79b640 1 --2- 172.21.15.98:0/1256898774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928100d80 0x7f5928101160 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5918009900 tx=0x7f591802a410 comp rx=0 tx=0).stop 2024-04-25T17:40:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.121+0000 7f592e79b640 1 -- 172.21.15.98:0/1256898774 shutdown_connections 2024-04-25T17:40:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.121+0000 7f592e79b640 1 --2- 172.21.15.98:0/1256898774 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f592810e930 0x7f5928110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.121+0000 7f592e79b640 1 --2- 172.21.15.98:0/1256898774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59281016a0 0x7f592810e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.121+0000 7f592e79b640 1 --2- 172.21.15.98:0/1256898774 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928100d80 0x7f5928101160 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.121+0000 7f592e79b640 1 -- 172.21.15.98:0/1256898774 >> 172.21.15.98:0/1256898774 conn(0x7f5928077fd0 msgr2=0x7f59280783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.121+0000 7f592e79b640 1 -- 172.21.15.98:0/1256898774 shutdown_connections 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.121+0000 7f592e79b640 1 -- 172.21.15.98:0/1256898774 wait complete. 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f592e79b640 1 Processor -- start 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f592e79b640 1 -- start start 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f592e79b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928100d80 0x7f59281a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f592e79b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59281016a0 0x7f59281a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f592e79b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f592810e930 0x7f592819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f592e79b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59281143c0 con 0x7f5928100d80 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f592e79b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5928114240 con 0x7f59281016a0 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f592e79b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5928114540 con 0x7f592810e930 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f5927fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928100d80 0x7f59281a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f5927fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928100d80 0x7f59281a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58978/0 (socket says 172.21.15.98:58978) 2024-04-25T17:40:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f5927fff640 1 -- 172.21.15.98:0/1157952970 learned_addr learned my addr 172.21.15.98:0/1157952970 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f592cd38640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f592810e930 0x7f592819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.122+0000 7f59277fe640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59281016a0 0x7f59281a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f592cd38640 1 -- 172.21.15.98:0/1157952970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59281016a0 msgr2=0x7f59281a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f592cd38640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59281016a0 0x7f59281a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f592cd38640 1 -- 172.21.15.98:0/1157952970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928100d80 msgr2=0x7f59281a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f592cd38640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928100d80 0x7f59281a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f592cd38640 1 -- 172.21.15.98:0/1157952970 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592819b020 con 0x7f592810e930 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f592cd38640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f592810e930 0x7f592819a760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5920005fd0 tx=0x7f592000edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f59257fa640 1 -- 172.21.15.98:0/1157952970 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5920004d20 con 0x7f592810e930 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f59257fa640 1 -- 172.21.15.98:0/1157952970 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5920005020 con 0x7f592810e930 2024-04-25T17:40:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f592e79b640 1 -- 172.21.15.98:0/1157952970 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f592819b310 con 0x7f592810e930 2024-04-25T17:40:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f59257fa640 1 -- 172.21.15.98:0/1157952970 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59200116b0 con 0x7f592810e930 2024-04-25T17:40:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.123+0000 7f592e79b640 1 -- 172.21.15.98:0/1157952970 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59281a75d0 con 0x7f592810e930 2024-04-25T17:40:48.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.125+0000 7f59257fa640 1 -- 172.21.15.98:0/1157952970 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f59200054d0 con 0x7f592810e930 2024-04-25T17:40:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.125+0000 7f5931593740 1 Processor -- start 2024-04-25T17:40:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f5931593740 1 -- start start 2024-04-25T17:40:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f5931593740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fba620 0x55edc1fbaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f5931593740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fbaf40 0x55edc1fc52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f5931593740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f5931593740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55edc1f1f970 con 0x55edc1fc5800 2024-04-25T17:40:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f5931593740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55edc1ebc4a0 con 0x55edc1fba620 2024-04-25T17:40:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f5931593740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55edc1f1b930 con 0x55edc1fbaf40 2024-04-25T17:40:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59257fa640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59100a3dc0 0x7f59100a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58990/0 (socket says 172.21.15.98:58990) 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59277fe640 1 -- 172.21.15.98:0/1147117368 learned_addr learned my addr 172.21.15.98:0/1147117368 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59277fe640 1 --2- 172.21.15.98:0/1147117368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fba620 0x55edc1fbaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f592cd38640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59100a3dc0 0x7f59100a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f5927fff640 1 --2- 172.21.15.98:0/1147117368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fbaf40 0x55edc1fc52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59277fe640 1 -- 172.21.15.98:0/1147117368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fbaf40 msgr2=0x55edc1fc52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59277fe640 1 --2- 172.21.15.98:0/1147117368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fbaf40 0x55edc1fc52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59277fe640 1 -- 172.21.15.98:0/1147117368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fba620 msgr2=0x55edc1fbaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59277fe640 1 --2- 172.21.15.98:0/1147117368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fba620 0x55edc1fbaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59277fe640 1 -- 172.21.15.98:0/1147117368 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edc1c32210 con 0x55edc1fc5800 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59257fa640 1 -- 172.21.15.98:0/1157952970 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f59200d07e0 con 0x7f592810e930 2024-04-25T17:40:48.135 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.126+0000 7f59277fe640 1 --2- 172.21.15.98:0/1147117368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc7bf0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f591c0125b0 tx=0x7f591c00b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b3f2c85b411f5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f590e7fc640 1 -- 172.21.15.98:0/1147117368 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c004220 con 0x55edc1fc5800 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f592cd38640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59100a3dc0 0x7f59100a6280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f592005d010 tx=0x7f5920087040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f590e7fc640 1 -- 172.21.15.98:0/1147117368 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f591c0043c0 con 0x55edc1fc5800 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f590e7fc640 1 -- 172.21.15.98:0/1147117368 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c0109b0 con 0x55edc1fc5800 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f5931593740 1 -- 172.21.15.98:0/1147117368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 msgr2=0x55edc1fc7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f5931593740 1 --2- 172.21.15.98:0/1147117368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc7bf0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f591c0125b0 tx=0x7f591c00b650 comp rx=0 tx=0).stop 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f5931593740 1 -- 172.21.15.98:0/1147117368 shutdown_connections 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f5931593740 1 --2- 172.21.15.98:0/1147117368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc7bf0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f5931593740 1 --2- 172.21.15.98:0/1147117368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fbaf40 0x55edc1fc52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f5931593740 1 --2- 172.21.15.98:0/1147117368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fba620 0x55edc1fbaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f5931593740 1 -- 172.21.15.98:0/1147117368 >> 172.21.15.98:0/1147117368 conn(0x55edc1fb6b30 msgr2=0x55edc1fb6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:48.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f5931593740 1 -- 172.21.15.98:0/1147117368 shutdown_connections 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.127+0000 7f5931593740 1 -- 172.21.15.98:0/1147117368 wait complete. 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f5931593740 1 -- start start 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f5931593740 1 Processor -- start 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f5931593740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fba620 0x55edc1fc51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f5931593740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fc2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f5931593740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f5931593740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55edc1fc8430 con 0x55edc1fc5800 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f5931593740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55edc1eba0b0 con 0x55edc1fbaf40 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f5931593740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55edc1fc85b0 con 0x55edc1fba620 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f5931593740 10 client.0 _ll_register_callbacks cb 0x55edc1ea7d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f59277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f59277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59002/0 (socket says 172.21.15.98:59002) 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f59277fe640 1 -- 172.21.15.98:0/1953193439 learned_addr learned my addr 172.21.15.98:0/1953193439 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.128+0000 7f59277fe640 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fba620 0x55edc1fc51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f5927fff640 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fc2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59277fe640 1 -- 172.21.15.98:0/1953193439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fba620 msgr2=0x55edc1fc51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59277fe640 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fba620 0x55edc1fc51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59277fe640 1 -- 172.21.15.98:0/1953193439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 msgr2=0x55edc1fc2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59277fe640 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fc2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59277fe640 1 -- 172.21.15.98:0/1953193439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edc1cfe620 con 0x55edc1fc5800 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f5927fff640 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fc2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59277fe640 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc2a30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f591c012f00 tx=0x7f591c0104b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4488b4dbc411e6d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59077fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:40:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59077fe640 1 -- 172.21.15.98:0/1953193439 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c010d70 con 0x55edc1fc5800 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59077fe640 1 -- 172.21.15.98:0/1953193439 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f591c0035f0 con 0x55edc1fc5800 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f5931593740 1 -- 172.21.15.98:0/1953193439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f591c0043c0 con 0x55edc1fc5800 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59077fe640 1 -- 172.21.15.98:0/1953193439 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f591c0037f0 con 0x55edc1fc5800 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f5931593740 10 client.15852 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f5931593740 1 -- 172.21.15.98:0/1953193439 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55edc1fd4380 con 0x55edc1fc5800 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59077fe640 1 -- 172.21.15.98:0/1953193439 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f591c0043c0 con 0x55edc1fc5800 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.129+0000 7f59077fe640 1 -- 172.21.15.98:0/1953193439 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f591c01c070 con 0x55edc1fc5800 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.130+0000 7f5931593740 10 client.15852 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.130+0000 7f5931593740 10 client.15852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.130+0000 7f5931593740 20 client.15852 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.130+0000 7f5931593740 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55edc1fd5790 0x55edc1fd7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.130+0000 7f5931593740 4 client.15852 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.130+0000 7f5931593740 1 -- 172.21.15.98:0/1953193439 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edc1f2d390 con 0x55edc1fd5790 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.130+0000 7f5927fff640 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55edc1fd5790 0x55edc1fd7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.130+0000 7f59077fe640 10 client.15852 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.130+0000 7f5927fff640 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55edc1fd5790 0x55edc1fd7b80 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec8329da7c58ad66 server_cookie=516023a638099a0d in_seq=0 out_seq=0 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f59077fe640 1 -- 172.21.15.98:0/1953193439 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55edc1f2d390 con 0x55edc1fd5790 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f59077fe640 10 client.15852 handle_command_reply: tid=0 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f5931593740 1 client.15852 shutdown 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f5931593740 1 -- 172.21.15.98:0/1953193439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 msgr2=0x55edc1fc2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f5931593740 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc2a30 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f591c012f00 tx=0x7f591c0104b0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f5931593740 1 -- 172.21.15.98:0/1953193439 shutdown_connections 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f5931593740 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55edc1fd5790 0x55edc1fd7b80 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f5931593740 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fc2a30 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.140 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f5931593740 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fc2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f5931593740 1 --2- 172.21.15.98:0/1953193439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fba620 0x55edc1fc51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f59077fe640 0 client.15852 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f5931593740 1 -- 172.21.15.98:0/1953193439 >> 172.21.15.98:0/1953193439 conn(0x55edc1fb6b30 msgr2=0x55edc1fbc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.131+0000 7f5931593740 1 -- 172.21.15.98:0/1953193439 shutdown_connections 2024-04-25T17:40:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.132+0000 7f5931593740 1 -- 172.21.15.98:0/1953193439 wait complete. 2024-04-25T17:40:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.132+0000 7f5931593740 20 client.15852 trim_cache size 0 max 16384 2024-04-25T17:40:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.146+0000 7f5931593740 1 Processor -- start 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.146+0000 7f5931593740 1 -- start start 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.146+0000 7f5931593740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fba620 0x55edc1fc4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.146+0000 7f5931593740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fdd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.146+0000 7f5931593740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fdd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.146+0000 7f5931593740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59281b2660 con 0x55edc1fc5800 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.146+0000 7f5931593740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59281afe70 con 0x55edc1fbaf40 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.146+0000 7f5931593740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f59281adcc0 con 0x55edc1fba620 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f59277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fdd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f59277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fdd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59018/0 (socket says 172.21.15.98:59018) 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f59277fe640 1 -- 172.21.15.98:0/2590607178 learned_addr learned my addr 172.21.15.98:0/2590607178 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f59277fe640 1 --2- 172.21.15.98:0/2590607178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fba620 0x55edc1fc4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f59277fe640 1 -- 172.21.15.98:0/2590607178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fba620 msgr2=0x55edc1fc4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f59277fe640 1 --2- 172.21.15.98:0/2590607178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fba620 0x55edc1fc4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f59277fe640 1 -- 172.21.15.98:0/2590607178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 msgr2=0x55edc1fdd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f5927fff640 1 --2- 172.21.15.98:0/2590607178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fdd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f59277fe640 1 --2- 172.21.15.98:0/2590607178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fdd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f59277fe640 1 -- 172.21.15.98:0/2590607178 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edc1fd8120 con 0x55edc1fc5800 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f59277fe640 1 --2- 172.21.15.98:0/2590607178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fdd890 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f591c01dce0 tx=0x7f591c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceca8ecc6da35138 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f590dffb640 1 -- 172.21.15.98:0/2590607178 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c039070 con 0x55edc1fc5800 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f590dffb640 1 -- 172.21.15.98:0/2590607178 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f591c021030 con 0x55edc1fc5800 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.147+0000 7f590dffb640 1 -- 172.21.15.98:0/2590607178 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c0120e0 con 0x55edc1fc5800 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 -- 172.21.15.98:0/2590607178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 msgr2=0x55edc1fdd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 --2- 172.21.15.98:0/2590607178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fdd890 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f591c01dce0 tx=0x7f591c000bc0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 -- 172.21.15.98:0/2590607178 shutdown_connections 2024-04-25T17:40:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 --2- 172.21.15.98:0/2590607178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fc5800 0x55edc1fdd890 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 --2- 172.21.15.98:0/2590607178 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fdd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 --2- 172.21.15.98:0/2590607178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fba620 0x55edc1fc4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 -- 172.21.15.98:0/2590607178 >> 172.21.15.98:0/2590607178 conn(0x55edc1fb6b30 msgr2=0x55edc1fc2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 -- 172.21.15.98:0/2590607178 shutdown_connections 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 -- 172.21.15.98:0/2590607178 wait complete. 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 -- start start 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.148+0000 7f5931593740 1 Processor -- start 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f5931593740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fba620 0x55edc1fca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f5931593740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fcaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f5931593740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fc5800 0x55edc2039bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f5931593740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59281ae330 con 0x55edc1fba620 2024-04-25T17:40:48.154 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f5931593740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59281af7a0 con 0x55edc1fbaf40 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f5931593740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f59281aea00 con 0x55edc1fc5800 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f5931593740 10 client.0 _ll_register_callbacks cb 0x7f591c004260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fba620 0x55edc1fca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fba620 0x55edc1fca9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59028/0 (socket says 172.21.15.98:59028) 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59277fe640 1 -- 172.21.15.98:0/2317942236 learned_addr learned my addr 172.21.15.98:0/2317942236 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f5927fff640 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fcaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59277fe640 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fc5800 0x55edc2039bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59277fe640 1 -- 172.21.15.98:0/2317942236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fc5800 msgr2=0x55edc2039bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59277fe640 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fc5800 0x55edc2039bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59277fe640 1 -- 172.21.15.98:0/2317942236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 msgr2=0x55edc1fcaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59277fe640 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fcaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59277fe640 1 -- 172.21.15.98:0/2317942236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edc203a150 con 0x55edc1fba620 2024-04-25T17:40:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59277fe640 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fba620 0x55edc1fca9c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f591c013f10 tx=0x7f591c004420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac8afd72fffe0fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59077fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.149+0000 7f59077fe640 1 -- 172.21.15.98:0/2317942236 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c049060 con 0x55edc1fba620 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f59077fe640 1 -- 172.21.15.98:0/2317942236 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f591c00b410 con 0x55edc1fba620 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f5931593740 1 -- 172.21.15.98:0/2317942236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f591c021030 con 0x55edc1fba620 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f59077fe640 1 -- 172.21.15.98:0/2317942236 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f591c039040 con 0x55edc1fba620 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f5931593740 10 client.15858 fetch_fsmap learned FSMap version 31 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f5931593740 1 -- 172.21.15.98:0/2317942236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55edc203b460 con 0x55edc1fba620 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f59077fe640 1 -- 172.21.15.98:0/2317942236 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f591c021030 con 0x55edc1fba620 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f59077fe640 1 -- 172.21.15.98:0/2317942236 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f591c01c040 con 0x55edc1fba620 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f5931593740 10 client.15858 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f5931593740 10 client.15858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f5931593740 20 client.15858 populate_metadata read hostname 'smithi098' 2024-04-25T17:40:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f5931593740 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55edc203bb00 0x55edc203df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:40:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f5931593740 4 client.15858 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:40:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f5931593740 1 -- 172.21.15.98:0/2317942236 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55edc1f27200 con 0x55edc203bb00 2024-04-25T17:40:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.150+0000 7f5927fff640 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55edc203bb00 0x55edc203df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:40:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.151+0000 7f5927fff640 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55edc203bb00 0x55edc203df20 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80ac194bc2b341da server_cookie=2ad1e1b8472d1668 in_seq=0 out_seq=0 2024-04-25T17:40:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.151+0000 7f59077fe640 10 client.15858 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.163+0000 7f59077fe640 1 -- 172.21.15.98:0/2317942236 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55edc1f27200 con 0x55edc203bb00 2024-04-25T17:40:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.163+0000 7f59077fe640 10 client.15858 handle_command_reply: tid=0 2024-04-25T17:40:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.163+0000 7f5931593740 1 client.15858 shutdown 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 1 -- 172.21.15.98:0/2317942236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fba620 msgr2=0x55edc1fca9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fba620 0x55edc1fca9c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f591c013f10 tx=0x7f591c004420 comp rx=0 tx=0).stop 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 1 -- 172.21.15.98:0/2317942236 shutdown_connections 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55edc203bb00 0x55edc203df20 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55edc1fc5800 0x55edc2039bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55edc1fbaf40 0x55edc1fcaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f59077fe640 0 client.15858 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 1 --2- 172.21.15.98:0/2317942236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55edc1fba620 0x55edc1fca9c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 1 -- 172.21.15.98:0/2317942236 >> 172.21.15.98:0/2317942236 conn(0x55edc1fb6b30 msgr2=0x55edc1fb5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 1 -- 172.21.15.98:0/2317942236 shutdown_connections 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 1 -- 172.21.15.98:0/2317942236 wait complete. 2024-04-25T17:40:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f5931593740 20 client.15858 trim_cache size 0 max 16384 2024-04-25T17:40:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:40:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:40:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:40:48.166 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:40:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f592e79b640 1 -- 172.21.15.98:0/1157952970 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59100a3dc0 msgr2=0x7f59100a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f592e79b640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59100a3dc0 0x7f59100a6280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f592005d010 tx=0x7f5920087040 comp rx=0 tx=0).stop 2024-04-25T17:40:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f592e79b640 1 -- 172.21.15.98:0/1157952970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f592810e930 msgr2=0x7f592819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:40:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.164+0000 7f592e79b640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f592810e930 0x7f592819a760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5920005fd0 tx=0x7f592000edd0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.165+0000 7f592e79b640 1 -- 172.21.15.98:0/1157952970 shutdown_connections 2024-04-25T17:40:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.165+0000 7f592e79b640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f59100a3dc0 0x7f59100a6280 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.165+0000 7f592e79b640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f592810e930 0x7f592819a760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.165+0000 7f592e79b640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59281016a0 0x7f59281a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.165+0000 7f592e79b640 1 --2- 172.21.15.98:0/1157952970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5928100d80 0x7f59281a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:40:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.165+0000 7f592e79b640 1 -- 172.21.15.98:0/1157952970 >> 172.21.15.98:0/1157952970 conn(0x7f5928077fd0 msgr2=0x7f592810fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:40:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.165+0000 7f592e79b640 1 -- 172.21.15.98:0/1157952970 shutdown_connections 2024-04-25T17:40:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:40:48.165+0000 7f592e79b640 1 -- 172.21.15.98:0/1157952970 wait complete. 2024-04-25T17:40:48.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:48 smithi098 ceph-mon[29192]: pgmap v150: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 4.0 MiB/s wr, 432 op/s 2024-04-25T17:40:48.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:48 smithi203 ceph-mon[32834]: pgmap v150: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 4.0 MiB/s wr, 432 op/s 2024-04-25T17:40:48.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:48 smithi173 ceph-mon[32856]: pgmap v150: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 4.0 MiB/s wr, 432 op/s 2024-04-25T17:40:50.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:50 smithi098 ceph-mon[29192]: pgmap v151: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-04-25T17:40:50.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:50 smithi203 ceph-mon[32834]: pgmap v151: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-04-25T17:40:50.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:50 smithi173 ceph-mon[32856]: pgmap v151: 97 pgs: 97 active+clean; 2.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 1.2 MiB/s wr, 131 op/s 2024-04-25T17:40:51.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:51 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:40:51.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:51 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:40:51.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:51 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:40:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 260.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:40:52.364 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:52 smithi203 ceph-mon[32834]: pgmap v152: 97 pgs: 97 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 3.2 MiB/s wr, 325 op/s 2024-04-25T17:40:52.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:52 smithi173 ceph-mon[32856]: pgmap v152: 97 pgs: 97 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 3.2 MiB/s wr, 325 op/s 2024-04-25T17:40:52.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:52 smithi098 ceph-mon[29192]: pgmap v152: 97 pgs: 97 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 3.2 MiB/s wr, 325 op/s 2024-04-25T17:40:54.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:54 smithi098 ceph-mon[29192]: pgmap v153: 97 pgs: 97 active+clean; 2.4 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 998 op/s 2024-04-25T17:40:54.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:54 smithi203 ceph-mon[32834]: pgmap v153: 97 pgs: 97 active+clean; 2.4 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 998 op/s 2024-04-25T17:40:54.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:54 smithi173 ceph-mon[32856]: pgmap v153: 97 pgs: 97 active+clean; 2.4 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 998 op/s 2024-04-25T17:40:56.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:56 smithi098 ceph-mon[29192]: pgmap v154: 97 pgs: 97 active+clean; 2.4 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-04-25T17:40:56.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:56 smithi203 ceph-mon[32834]: pgmap v154: 97 pgs: 97 active+clean; 2.4 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-04-25T17:40:56.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:56 smithi173 ceph-mon[32856]: pgmap v154: 97 pgs: 97 active+clean; 2.4 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-04-25T17:40:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 265.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:40:58.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:58 smithi098 ceph-mon[29192]: pgmap v155: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.4 MiB/s wr, 898 op/s 2024-04-25T17:40:58.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:58 smithi203 ceph-mon[32834]: pgmap v155: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.4 MiB/s wr, 898 op/s 2024-04-25T17:40:58.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:58 smithi173 ceph-mon[32856]: pgmap v155: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.4 MiB/s wr, 898 op/s 2024-04-25T17:40:59.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:40:59 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:40:59.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:40:59 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:40:59.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:40:59 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:41:00.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:00 smithi098 ceph-mon[29192]: pgmap v156: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-04-25T17:41:00.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:00 smithi203 ceph-mon[32834]: pgmap v156: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-04-25T17:41:00.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:00 smithi173 ceph-mon[32856]: pgmap v156: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 895 op/s 2024-04-25T17:41:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.261+0000 7f097a2cf640 1 -- 172.21.15.98:0/343979375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0974106fb0 msgr2=0x7f0974107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.261+0000 7f097a2cf640 1 --2- 172.21.15.98:0/343979375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0974106fb0 0x7f0974107390 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0964009900 tx=0x7f096402a400 comp rx=0 tx=0).stop 2024-04-25T17:41:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.261+0000 7f097a2cf640 1 -- 172.21.15.98:0/343979375 shutdown_connections 2024-04-25T17:41:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.261+0000 7f097a2cf640 1 --2- 172.21.15.98:0/343979375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0974101b60 0x7f097410dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.261+0000 7f097a2cf640 1 --2- 172.21.15.98:0/343979375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09741011a0 0x7f0974101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.261+0000 7f097a2cf640 1 --2- 172.21.15.98:0/343979375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0974106fb0 0x7f0974107390 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.261+0000 7f097a2cf640 1 -- 172.21.15.98:0/343979375 >> 172.21.15.98:0/343979375 conn(0x7f0974076c50 msgr2=0x7f0974077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.261+0000 7f097a2cf640 1 -- 172.21.15.98:0/343979375 shutdown_connections 2024-04-25T17:41:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.261+0000 7f097a2cf640 1 -- 172.21.15.98:0/343979375 wait complete. 2024-04-25T17:41:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.262+0000 7f097a2cf640 1 Processor -- start 2024-04-25T17:41:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.262+0000 7f097a2cf640 1 -- start start 2024-04-25T17:41:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.262+0000 7f097a2cf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09741011a0 0x7f0974198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.262+0000 7f097a2cf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0974101b60 0x7f0974198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.262+0000 7f097a2cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0974106fb0 0x7f097419cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.262+0000 7f097a2cf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0974111090 con 0x7f0974106fb0 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.262+0000 7f097a2cf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0974110f10 con 0x7f0974101b60 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.262+0000 7f097a2cf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0974111210 con 0x7f09741011a0 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097886c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0974106fb0 0x7f097419cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097886c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0974106fb0 0x7f097419cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57634/0 (socket says 172.21.15.98:57634) 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097886c640 1 -- 172.21.15.98:0/3693061636 learned_addr learned my addr 172.21.15.98:0/3693061636 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f0973fff640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09741011a0 0x7f0974198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f09737fe640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0974101b60 0x7f0974198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097886c640 1 -- 172.21.15.98:0/3693061636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09741011a0 msgr2=0x7f0974198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097886c640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09741011a0 0x7f0974198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097886c640 1 -- 172.21.15.98:0/3693061636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0974101b60 msgr2=0x7f0974198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097886c640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0974101b60 0x7f0974198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097886c640 1 -- 172.21.15.98:0/3693061636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097419d620 con 0x7f0974106fb0 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097886c640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0974106fb0 0x7f097419cef0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f096c00eaa0 tx=0x7f096c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f09717fa640 1 -- 172.21.15.98:0/3693061636 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c00cdb0 con 0x7f0974106fb0 2024-04-25T17:41:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f09717fa640 1 -- 172.21.15.98:0/3693061636 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f096c005020 con 0x7f0974106fb0 2024-04-25T17:41:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f09717fa640 1 -- 172.21.15.98:0/3693061636 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c0116d0 con 0x7f0974106fb0 2024-04-25T17:41:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097a2cf640 1 -- 172.21.15.98:0/3693061636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097419d910 con 0x7f0974106fb0 2024-04-25T17:41:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.263+0000 7f097a2cf640 1 -- 172.21.15.98:0/3693061636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09741a51a0 con 0x7f0974106fb0 2024-04-25T17:41:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.265+0000 7f09717fa640 1 -- 172.21.15.98:0/3693061636 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f096c0051d0 con 0x7f0974106fb0 2024-04-25T17:41:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f097cfef740 1 Processor -- start 2024-04-25T17:41:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f097cfef740 1 -- start start 2024-04-25T17:41:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f09717fa640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f095c0a3dc0 0x7f095c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f097cfef740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x55638571da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f097cfef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x5563857282c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f097cfef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x55638572abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f09737fe640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f095c0a3dc0 0x7f095c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f097cfef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x556385682970 con 0x55638571df40 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f097cfef740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55638561f4a0 con 0x55638571d620 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f097cfef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55638567e930 con 0x556385728800 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f09717fa640 1 -- 172.21.15.98:0/3693061636 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f096c0d0930 con 0x7f0974106fb0 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f09737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x5563857282c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f09737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x5563857282c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57642/0 (socket says 172.21.15.98:57642) 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.266+0000 7f09737fe640 1 -- 172.21.15.98:0/531162307 learned_addr learned my addr 172.21.15.98:0/531162307 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f097886c640 1 --2- 172.21.15.98:0/531162307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x55638572abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f0973fff640 1 --2- 172.21.15.98:0/531162307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x55638571da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f09737fe640 1 -- 172.21.15.98:0/531162307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 msgr2=0x55638572abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f09737fe640 1 --2- 172.21.15.98:0/531162307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x55638572abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f09737fe640 1 -- 172.21.15.98:0/531162307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 msgr2=0x55638571da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f09737fe640 1 --2- 172.21.15.98:0/531162307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x55638571da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f09737fe640 1 -- 172.21.15.98:0/531162307 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556385395210 con 0x55638571df40 2024-04-25T17:41:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f09737fe640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f095c0a3dc0 0x7f095c0a6280 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f096800a5c0 tx=0x7f096800ab40 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f09737fe640 1 --2- 172.21.15.98:0/531162307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x5563857282c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f09680117b0 tx=0x7f096801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d791e04575f1d233 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f095a7fc640 1 -- 172.21.15.98:0/531162307 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096800fc60 con 0x55638571df40 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f095a7fc640 1 -- 172.21.15.98:0/531162307 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f096800fe00 con 0x55638571df40 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.267+0000 7f095a7fc640 1 -- 172.21.15.98:0/531162307 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09680037f0 con 0x55638571df40 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.268+0000 7f097cfef740 1 -- 172.21.15.98:0/531162307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 msgr2=0x5563857282c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.268+0000 7f097cfef740 1 --2- 172.21.15.98:0/531162307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x5563857282c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f09680117b0 tx=0x7f096801b000 comp rx=0 tx=0).stop 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.268+0000 7f097cfef740 1 -- 172.21.15.98:0/531162307 shutdown_connections 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.268+0000 7f097cfef740 1 --2- 172.21.15.98:0/531162307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x55638572abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.268+0000 7f097cfef740 1 --2- 172.21.15.98:0/531162307 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x5563857282c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.268+0000 7f097cfef740 1 --2- 172.21.15.98:0/531162307 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x55638571da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.268+0000 7f097cfef740 1 -- 172.21.15.98:0/531162307 >> 172.21.15.98:0/531162307 conn(0x556385719b30 msgr2=0x556385719f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.268+0000 7f097cfef740 1 -- 172.21.15.98:0/531162307 shutdown_connections 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.268+0000 7f097cfef740 1 -- 172.21.15.98:0/531162307 wait complete. 2024-04-25T17:41:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.268+0000 7f097cfef740 1 -- start start 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f097cfef740 1 Processor -- start 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f097cfef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571d620 0x556385728160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f097cfef740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571df40 0x556385725450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f097cfef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x5563857259b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f097cfef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55638572b3d0 con 0x55638571d620 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f097cfef740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55638561d0b0 con 0x55638571df40 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f097cfef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55638572b550 con 0x556385728800 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f097cfef740 10 client.0 _ll_register_callbacks cb 0x55638560ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f0973fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571d620 0x556385728160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f0973fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571d620 0x556385728160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57646/0 (socket says 172.21.15.98:57646) 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f0973fff640 1 -- 172.21.15.98:0/2926991937 learned_addr learned my addr 172.21.15.98:0/2926991937 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f09737fe640 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571df40 0x556385725450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f097886c640 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x5563857259b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f0973fff640 1 -- 172.21.15.98:0/2926991937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 msgr2=0x5563857259b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f0973fff640 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x5563857259b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f0973fff640 1 -- 172.21.15.98:0/2926991937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571df40 msgr2=0x556385725450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f0973fff640 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571df40 0x556385725450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f0973fff640 1 -- 172.21.15.98:0/2926991937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556385461620 con 0x55638571d620 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f0973fff640 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571d620 0x556385728160 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0964009900 tx=0x7f09640316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f68091c30bd4e24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f09537fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f09537fe640 1 -- 172.21.15.98:0/2926991937 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0964035040 con 0x55638571d620 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.269+0000 7f09537fe640 1 -- 172.21.15.98:0/2926991937 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f0964031d10 con 0x55638571d620 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f097cfef740 1 -- 172.21.15.98:0/2926991937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f096800fe00 con 0x55638571d620 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f09537fe640 1 -- 172.21.15.98:0/2926991937 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0964004040 con 0x55638571d620 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f097cfef740 10 client.15870 fetch_fsmap learned FSMap version 33 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f097cfef740 1 -- 172.21.15.98:0/2926991937 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556385737340 con 0x55638571d620 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f09537fe640 1 -- 172.21.15.98:0/2926991937 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f096800fe00 con 0x55638571d620 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f09537fe640 1 -- 172.21.15.98:0/2926991937 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f09640047e0 con 0x55638571d620 2024-04-25T17:41:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f097cfef740 10 client.15870 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f097cfef740 10 client.15870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f097cfef740 20 client.15870 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f097cfef740 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563857387b0 0x55638573aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f097cfef740 4 client.15870 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f097cfef740 1 -- 172.21.15.98:0/2926991937 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556385690390 con 0x5563857387b0 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.270+0000 7f09737fe640 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563857387b0 0x55638573aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.271+0000 7f09737fe640 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563857387b0 0x55638573aba0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea89a1a5b9842d0d server_cookie=ab2a7cfa383ecb07 in_seq=0 out_seq=0 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.271+0000 7f09537fe640 10 client.15870 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.271+0000 7f09537fe640 1 -- 172.21.15.98:0/2926991937 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556385690390 con 0x5563857387b0 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.271+0000 7f09537fe640 10 client.15870 handle_command_reply: tid=0 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.271+0000 7f097cfef740 1 client.15870 shutdown 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 1 -- 172.21.15.98:0/2926991937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571d620 msgr2=0x556385728160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571d620 0x556385728160 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0964009900 tx=0x7f09640316d0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 1 -- 172.21.15.98:0/2926991937 shutdown_connections 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563857387b0 0x55638573aba0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x5563857259b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f09537fe640 0 client.15870 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:01.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571df40 0x556385725450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 1 --2- 172.21.15.98:0/2926991937 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571d620 0x556385728160 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 1 -- 172.21.15.98:0/2926991937 >> 172.21.15.98:0/2926991937 conn(0x556385719b30 msgr2=0x556385718cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:01.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 1 -- 172.21.15.98:0/2926991937 shutdown_connections 2024-04-25T17:41:01.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 1 -- 172.21.15.98:0/2926991937 wait complete. 2024-04-25T17:41:01.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.272+0000 7f097cfef740 20 client.15870 trim_cache size 0 max 16384 2024-04-25T17:41:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.286+0000 7f097cfef740 1 Processor -- start 2024-04-25T17:41:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.286+0000 7f097cfef740 1 -- start start 2024-04-25T17:41:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f097cfef740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x556385734290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f097cfef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x556385718b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.291 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f097cfef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x556385727c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f097cfef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09741b0290 con 0x55638571df40 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f097cfef740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f09741adb30 con 0x55638571d620 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f097cfef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09741ab980 con 0x556385728800 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f09737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x556385718b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f09737fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x556385718b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57656/0 (socket says 172.21.15.98:57656) 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f09737fe640 1 -- 172.21.15.98:0/2855964588 learned_addr learned my addr 172.21.15.98:0/2855964588 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f0973fff640 1 --2- 172.21.15.98:0/2855964588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x556385734290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f09737fe640 1 -- 172.21.15.98:0/2855964588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 msgr2=0x556385727c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f09737fe640 1 --2- 172.21.15.98:0/2855964588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x556385727c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f09737fe640 1 -- 172.21.15.98:0/2855964588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 msgr2=0x556385734290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f09737fe640 1 --2- 172.21.15.98:0/2855964588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x556385734290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f09737fe640 1 -- 172.21.15.98:0/2855964588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556385724420 con 0x55638571df40 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f0973fff640 1 --2- 172.21.15.98:0/2855964588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x556385734290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:41:01.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f09737fe640 1 --2- 172.21.15.98:0/2855964588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x556385718b30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0968011800 tx=0x7f0968014310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608e9647601f70cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f0959ffb640 1 -- 172.21.15.98:0/2855964588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09680140d0 con 0x55638571df40 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f0959ffb640 1 -- 172.21.15.98:0/2855964588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09680338b0 con 0x55638571df40 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.287+0000 7f0959ffb640 1 -- 172.21.15.98:0/2855964588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0968005270 con 0x55638571df40 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.288+0000 7f097cfef740 1 -- 172.21.15.98:0/2855964588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 msgr2=0x556385718b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.288+0000 7f097cfef740 1 --2- 172.21.15.98:0/2855964588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x556385718b30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0968011800 tx=0x7f0968014310 comp rx=0 tx=0).stop 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.288+0000 7f097cfef740 1 -- 172.21.15.98:0/2855964588 shutdown_connections 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.288+0000 7f097cfef740 1 --2- 172.21.15.98:0/2855964588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556385728800 0x556385727c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.288+0000 7f097cfef740 1 --2- 172.21.15.98:0/2855964588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55638571df40 0x556385718b30 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.288+0000 7f097cfef740 1 --2- 172.21.15.98:0/2855964588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x556385734290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.288+0000 7f097cfef740 1 -- 172.21.15.98:0/2855964588 >> 172.21.15.98:0/2855964588 conn(0x556385719b30 msgr2=0x556385718420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.288+0000 7f097cfef740 1 -- 172.21.15.98:0/2855964588 shutdown_connections 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.288+0000 7f097cfef740 1 -- 172.21.15.98:0/2855964588 wait complete. 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.288+0000 7f097cfef740 1 -- start start 2024-04-25T17:41:01.293 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097cfef740 1 Processor -- start 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097cfef740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x5563857307a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097cfef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55638571df40 0x556385730ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097cfef740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556385728800 0x55638572d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097cfef740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09741abff0 con 0x556385728800 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097cfef740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f09741ad460 con 0x55638571d620 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097cfef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09741ac6c0 con 0x55638571df40 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097cfef740 10 client.0 _ll_register_callbacks cb 0x7f0964032dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097886c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556385728800 0x55638572d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097886c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556385728800 0x55638572d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57666/0 (socket says 172.21.15.98:57666) 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097886c640 1 -- 172.21.15.98:0/2967149784 learned_addr learned my addr 172.21.15.98:0/2967149784 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f0973fff640 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x5563857307a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097886c640 1 -- 172.21.15.98:0/2967149784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55638571df40 msgr2=0x556385730ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097886c640 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55638571df40 0x556385730ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097886c640 1 -- 172.21.15.98:0/2967149784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 msgr2=0x5563857307a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:01.294 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097886c640 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x5563857307a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097886c640 1 -- 172.21.15.98:0/2967149784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556385725d20 con 0x556385728800 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f0973fff640 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x5563857307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f097886c640 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556385728800 0x55638572d770 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f096c09c940 tx=0x7f096c09cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c132046e70bce874 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f09537fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f09537fe640 1 -- 172.21.15.98:0/2967149784 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c01f040 con 0x556385728800 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.289+0000 7f09537fe640 1 -- 172.21.15.98:0/2967149784 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f096c0870b0 con 0x556385728800 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f097cfef740 1 -- 172.21.15.98:0/2967149784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09680338b0 con 0x556385728800 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f09537fe640 1 -- 172.21.15.98:0/2967149784 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f096c0872b0 con 0x556385728800 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f097cfef740 10 client.15876 fetch_fsmap learned FSMap version 33 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f097cfef740 1 -- 172.21.15.98:0/2967149784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55638573af10 con 0x556385728800 2024-04-25T17:41:01.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f09537fe640 1 -- 172.21.15.98:0/2967149784 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09680338b0 con 0x556385728800 2024-04-25T17:41:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f09537fe640 1 -- 172.21.15.98:0/2967149784 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f096c00f040 con 0x556385728800 2024-04-25T17:41:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f097cfef740 10 client.15876 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T17:41:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f097cfef740 10 client.15876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:41:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f097cfef740 20 client.15876 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f097cfef740 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55638579fa60 0x5563857a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f097cfef740 4 client.15876 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:41:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f097cfef740 1 -- 172.21.15.98:0/2967149784 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55638568a200 con 0x55638579fa60 2024-04-25T17:41:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.290+0000 7f0973fff640 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55638579fa60 0x5563857a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.291+0000 7f0973fff640 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55638579fa60 0x5563857a1e80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5c1db36bd394a87 server_cookie=4abc19095b306c81 in_seq=0 out_seq=0 2024-04-25T17:41:01.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.291+0000 7f09537fe640 10 client.15876 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.300+0000 7f09537fe640 1 -- 172.21.15.98:0/2967149784 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55638568a200 con 0x55638579fa60 2024-04-25T17:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.300+0000 7f09537fe640 10 client.15876 handle_command_reply: tid=0 2024-04-25T17:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.300+0000 7f097cfef740 1 client.15876 shutdown 2024-04-25T17:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 1 -- 172.21.15.98:0/2967149784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556385728800 msgr2=0x55638572d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556385728800 0x55638572d770 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f096c09c940 tx=0x7f096c09cae0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 1 -- 172.21.15.98:0/2967149784 shutdown_connections 2024-04-25T17:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55638579fa60 0x5563857a1e80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x556385728800 0x55638572d770 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55638571df40 0x556385730ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 1 --2- 172.21.15.98:0/2967149784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55638571d620 0x5563857307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f09537fe640 0 client.15876 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 1 -- 172.21.15.98:0/2967149784 >> 172.21.15.98:0/2967149784 conn(0x556385719b30 msgr2=0x556385718cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 1 -- 172.21.15.98:0/2967149784 shutdown_connections 2024-04-25T17:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 1 -- 172.21.15.98:0/2967149784 wait complete. 2024-04-25T17:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097cfef740 20 client.15876 trim_cache size 0 max 16384 2024-04-25T17:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:41:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097a2cf640 1 -- 172.21.15.98:0/3693061636 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f095c0a3dc0 msgr2=0x7f095c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097a2cf640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f095c0a3dc0 0x7f095c0a6280 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f096800a5c0 tx=0x7f096800ab40 comp rx=0 tx=0).stop 2024-04-25T17:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097a2cf640 1 -- 172.21.15.98:0/3693061636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0974106fb0 msgr2=0x7f097419cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097a2cf640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0974106fb0 0x7f097419cef0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f096c00eaa0 tx=0x7f096c00ef60 comp rx=0 tx=0).stop 2024-04-25T17:41:01.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097a2cf640 1 -- 172.21.15.98:0/3693061636 shutdown_connections 2024-04-25T17:41:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097a2cf640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f095c0a3dc0 0x7f095c0a6280 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097a2cf640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0974106fb0 0x7f097419cef0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097a2cf640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0974101b60 0x7f0974198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.301+0000 7f097a2cf640 1 --2- 172.21.15.98:0/3693061636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09741011a0 0x7f0974198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.302+0000 7f097a2cf640 1 -- 172.21.15.98:0/3693061636 >> 172.21.15.98:0/3693061636 conn(0x7f0974076c50 msgr2=0x7f097406a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.302+0000 7f097a2cf640 1 -- 172.21.15.98:0/3693061636 shutdown_connections 2024-04-25T17:41:01.304 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:01.302+0000 7f097a2cf640 1 -- 172.21.15.98:0/3693061636 wait complete. 2024-04-25T17:41:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 270.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:41:02.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:02 smithi098 ceph-mon[29192]: pgmap v157: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.5 MiB/s wr, 911 op/s 2024-04-25T17:41:02.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:02 smithi203 ceph-mon[32834]: pgmap v157: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.5 MiB/s wr, 911 op/s 2024-04-25T17:41:02.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:02 smithi173 ceph-mon[32856]: pgmap v157: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.5 MiB/s wr, 911 op/s 2024-04-25T17:41:03.458 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:03 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:41:03.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:03 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:41:03.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:03 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:41:03.840 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:41:04.085 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:41:04.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:04 smithi098 ceph-mon[29192]: pgmap v158: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.5 MiB/s wr, 718 op/s 2024-04-25T17:41:04.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:04 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:41:04.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:04 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:41:04.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:04 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:41:04.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:04 smithi203 ceph-mon[32834]: pgmap v158: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.5 MiB/s wr, 718 op/s 2024-04-25T17:41:04.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:04 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:41:04.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:04 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:41:04.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:04 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:41:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:04 smithi173 ceph-mon[32856]: pgmap v158: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.5 MiB/s wr, 718 op/s 2024-04-25T17:41:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:04 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:41:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:04 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:41:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:04 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:41:06.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.050+0000 7f970da52700 1 -- 172.21.15.98:0/1648107116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f970806b980 msgr2=0x7f970810f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:06.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.050+0000 7f970da52700 1 --2- 172.21.15.98:0/1648107116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f970806b980 0x7f970810f8f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f970000b0a0 tx=0x7f970000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:41:06.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.050+0000 7f970da52700 1 -- 172.21.15.98:0/1648107116 shutdown_connections 2024-04-25T17:41:06.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.050+0000 7f970da52700 1 --2- 172.21.15.98:0/1648107116 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f970806b980 0x7f970810f8f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:06.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.050+0000 7f970da52700 1 --2- 172.21.15.98:0/1648107116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9708108a20 0x7f9708108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:06.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.050+0000 7f970da52700 1 --2- 172.21.15.98:0/1648107116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f970806afe0 0x7f970806b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:06.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.050+0000 7f970da52700 1 -- 172.21.15.98:0/1648107116 >> 172.21.15.98:0/1648107116 conn(0x7f97080788b0 msgr2=0x7f9708078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.050+0000 7f970da52700 1 -- 172.21.15.98:0/1648107116 shutdown_connections 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.050+0000 7f970da52700 1 -- 172.21.15.98:0/1648107116 wait complete. 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f970da52700 1 Processor -- start 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f970da52700 1 -- start start 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f970da52700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f970806afe0 0x7f9708198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f970da52700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f970806b980 0x7f97081994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f970da52700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9708108a20 0x7f970819d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f970da52700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9708112ba0 con 0x7f9708108a20 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f970da52700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9708112a20 con 0x7f970806b980 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f970da52700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9708112d20 con 0x7f970806afe0 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f97077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9708108a20 0x7f970819d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f97077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9708108a20 0x7f970819d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50088/0 (socket says 172.21.15.98:50088) 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f97077fe700 1 -- 172.21.15.98:0/135349828 learned_addr learned my addr 172.21.15.98:0/135349828 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:06.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f9706ffd700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f970806afe0 0x7f9708198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f96fe7fc700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f970806b980 0x7f97081994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f97077fe700 1 -- 172.21.15.98:0/135349828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f970806afe0 msgr2=0x7f9708198fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f97077fe700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f970806afe0 0x7f9708198fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f97077fe700 1 -- 172.21.15.98:0/135349828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f970806b980 msgr2=0x7f97081994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f97077fe700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f970806b980 0x7f97081994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f97077fe700 1 -- 172.21.15.98:0/135349828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f970819df70 con 0x7f9708108a20 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.051+0000 7f96fe7fc700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f970806b980 0x7f97081994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.052+0000 7f97077fe700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9708108a20 0x7f970819d830 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f97000062a0 tx=0x7f9700004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.052+0000 7f970ca50700 1 -- 172.21.15.98:0/135349828 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700021070 con 0x7f9708108a20 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.052+0000 7f970ca50700 1 -- 172.21.15.98:0/135349828 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9700007c80 con 0x7f9708108a20 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.052+0000 7f970da52700 1 -- 172.21.15.98:0/135349828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f970819e200 con 0x7f9708108a20 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.052+0000 7f970ca50700 1 -- 172.21.15.98:0/135349828 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970001d450 con 0x7f9708108a20 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.052+0000 7f970da52700 1 -- 172.21.15.98:0/135349828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97081a5ae0 con 0x7f9708108a20 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.053+0000 7f970da52700 1 -- 172.21.15.98:0/135349828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9708069730 con 0x7f9708108a20 2024-04-25T17:41:06.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.054+0000 7f970ca50700 1 -- 172.21.15.98:0/135349828 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9700004000 con 0x7f9708108a20 2024-04-25T17:41:06.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.054+0000 7f970ca50700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f96f40a3db0 0x7f96f40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:06.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.054+0000 7f9706ffd700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f96f40a3db0 0x7f96f40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:06.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.055+0000 7f970ca50700 1 -- 172.21.15.98:0/135349828 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f970000f020 con 0x7f9708108a20 2024-04-25T17:41:06.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.055+0000 7f9706ffd700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f96f40a3db0 0x7f96f40a6270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f96f800abd0 tx=0x7f96f800a380 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:06.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.058+0000 7f970ca50700 1 -- 172.21.15.98:0/135349828 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f970002a020 con 0x7f9708108a20 2024-04-25T17:41:06.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:06 smithi098 ceph-mon[29192]: pgmap v159: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 21 op/s 2024-04-25T17:41:06.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.307+0000 7f970da52700 1 -- 172.21.15.98:0/135349828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f970802df80 con 0x7f9708108a20 2024-04-25T17:41:06.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.308+0000 7f970ca50700 1 -- 172.21.15.98:0/135349828 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11878 (secure 0 0 0) 0x7f970009e9e0 con 0x7f9708108a20 2024-04-25T17:41:06.308 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:41:06.308 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:40:57.732557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:41:06.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 -- 172.21.15.98:0/135349828 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f96f40a3db0 msgr2=0x7f96f40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:06.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f96f40a3db0 0x7f96f40a6270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f96f800abd0 tx=0x7f96f800a380 comp rx=0 tx=0).stop 2024-04-25T17:41:06.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 -- 172.21.15.98:0/135349828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9708108a20 msgr2=0x7f970819d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:06.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9708108a20 0x7f970819d830 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f97000062a0 tx=0x7f9700004e20 comp rx=0 tx=0).stop 2024-04-25T17:41:06.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 -- 172.21.15.98:0/135349828 shutdown_connections 2024-04-25T17:41:06.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f96f40a3db0 0x7f96f40a6270 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:06.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f970806afe0 0x7f9708198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:06.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9708108a20 0x7f970819d830 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:06.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 --2- 172.21.15.98:0/135349828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f970806b980 0x7f97081994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:06.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 -- 172.21.15.98:0/135349828 >> 172.21.15.98:0/135349828 conn(0x7f97080788b0 msgr2=0x7f97080ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:06.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 -- 172.21.15.98:0/135349828 shutdown_connections 2024-04-25T17:41:06.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:06.310+0000 7f970da52700 1 -- 172.21.15.98:0/135349828 wait complete. 2024-04-25T17:41:06.311 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 33 2024-04-25T17:41:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:06 smithi203 ceph-mon[32834]: pgmap v159: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 21 op/s 2024-04-25T17:41:06.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:06 smithi173 ceph-mon[32856]: pgmap v159: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 33 KiB/s wr, 21 op/s 2024-04-25T17:41:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 275.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:41:07.133 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id 839b44ee --release --await' 2024-04-25T17:41:07.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id 839b44ee --release --await 2024-04-25T17:41:07.382 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:41:07.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:07 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/135349828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:41:07.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:07 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/135349828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:41:07.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:07 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/135349828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:41:08.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:08 smithi098 ceph-mon[29192]: pgmap v160: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 49 KiB/s wr, 29 op/s 2024-04-25T17:41:08.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:08 smithi203 ceph-mon[32834]: pgmap v160: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 49 KiB/s wr, 29 op/s 2024-04-25T17:41:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:08 smithi173 ceph-mon[32856]: pgmap v160: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 49 KiB/s wr, 29 op/s 2024-04-25T17:41:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.228+0000 7fb141bfb700 1 -- 172.21.15.98:0/3102132242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c102a90 msgr2=0x7fb13c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.228+0000 7fb141bfb700 1 --2- 172.21.15.98:0/3102132242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c102a90 0x7fb13c102f10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb130009990 tx=0x7fb130009ca0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.228+0000 7fb141bfb700 1 -- 172.21.15.98:0/3102132242 shutdown_connections 2024-04-25T17:41:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.228+0000 7fb141bfb700 1 --2- 172.21.15.98:0/3102132242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c103450 0x7fb13c10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.228+0000 7fb141bfb700 1 --2- 172.21.15.98:0/3102132242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb13c108a50 0x7fb13c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.228+0000 7fb141bfb700 1 --2- 172.21.15.98:0/3102132242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c102a90 0x7fb13c102f10 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.228+0000 7fb141bfb700 1 -- 172.21.15.98:0/3102132242 >> 172.21.15.98:0/3102132242 conn(0x7fb13c079ba0 msgr2=0x7fb13c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.228+0000 7fb141bfb700 1 -- 172.21.15.98:0/3102132242 shutdown_connections 2024-04-25T17:41:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.228+0000 7fb141bfb700 1 -- 172.21.15.98:0/3102132242 wait complete. 2024-04-25T17:41:09.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.229+0000 7fb141bfb700 1 Processor -- start 2024-04-25T17:41:09.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.229+0000 7fb141bfb700 1 -- start start 2024-04-25T17:41:09.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.229+0000 7fb141bfb700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c102a90 0x7fb13c199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.229+0000 7fb141bfb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c103450 0x7fb13c199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.229+0000 7fb141bfb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb13c108a50 0x7fb13c19da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.229+0000 7fb141bfb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c112be0 con 0x7fb13c103450 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.229+0000 7fb141bfb700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c112a60 con 0x7fb13c108a50 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.229+0000 7fb141bfb700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c112d60 con 0x7fb13c102a90 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c103450 0x7fb13c199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c103450 0x7fb13c199690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50112/0 (socket says 172.21.15.98:50112) 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13affd700 1 -- 172.21.15.98:0/3643850560 learned_addr learned my addr 172.21.15.98:0/3643850560 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c102a90 0x7fb13c199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13bfff700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb13c108a50 0x7fb13c19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13affd700 1 -- 172.21.15.98:0/3643850560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c102a90 msgr2=0x7fb13c199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13affd700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c102a90 0x7fb13c199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13affd700 1 -- 172.21.15.98:0/3643850560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb13c108a50 msgr2=0x7fb13c19da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13affd700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb13c108a50 0x7fb13c19da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13affd700 1 -- 172.21.15.98:0/3643850560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c19e160 con 0x7fb13c103450 2024-04-25T17:41:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb13affd700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c103450 0x7fb13c199690 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb130000c00 tx=0x7fb1300047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb138ff9700 1 -- 172.21.15.98:0/3643850560 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130027070 con 0x7fb13c103450 2024-04-25T17:41:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb138ff9700 1 -- 172.21.15.98:0/3643850560 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb130003f20 con 0x7fb13c103450 2024-04-25T17:41:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb141bfb700 1 -- 172.21.15.98:0/3643850560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c19e3f0 con 0x7fb13c103450 2024-04-25T17:41:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb138ff9700 1 -- 172.21.15.98:0/3643850560 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130022ca0 con 0x7fb13c103450 2024-04-25T17:41:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.230+0000 7fb141bfb700 1 -- 172.21.15.98:0/3643850560 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13c1a5cd0 con 0x7fb13c103450 2024-04-25T17:41:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.232+0000 7fb138ff9700 1 -- 172.21.15.98:0/3643850560 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb13002a070 con 0x7fb13c103450 2024-04-25T17:41:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.233+0000 7fb138ff9700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb1240a3db0 0x7fb1240a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.233+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb1240a3db0 0x7fb1240a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.233+0000 7fb138ff9700 1 -- 172.21.15.98:0/3643850560 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fb1300d1610 con 0x7fb13c103450 2024-04-25T17:41:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb1240a3db0 0x7fb1240a6270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb12c005950 tx=0x7fb12c00b410 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb14c484b80 1 Processor -- start 2024-04-25T17:41:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb14c484b80 1 -- start start 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 0x561fda761dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda7710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda7739e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb14c484b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561fda6b4e30 con 0x561fda762310 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb14c484b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561fda65ba60 con 0x561fda7715f0 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb14c484b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561fda6b0df0 con 0x561fda7619f0 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb13affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda7710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb13affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda7710b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50118/0 (socket says 172.21.15.98:50118) 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb13affd700 1 -- 172.21.15.98:0/41234529 learned_addr learned my addr 172.21.15.98:0/41234529 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.234+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/41234529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 0x561fda761dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb13bfff700 1 --2- 172.21.15.98:0/41234529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda7739e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb13affd700 1 -- 172.21.15.98:0/41234529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 msgr2=0x561fda761dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb13affd700 1 --2- 172.21.15.98:0/41234529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 0x561fda761dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb13affd700 1 -- 172.21.15.98:0/41234529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 msgr2=0x561fda7739e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb13affd700 1 --2- 172.21.15.98:0/41234529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda7739e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb13affd700 1 -- 172.21.15.98:0/41234529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fda6029d0 con 0x561fda762310 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/41234529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 0x561fda761dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb13affd700 1 --2- 172.21.15.98:0/41234529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda7710b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb130099ad0 tx=0x7fb13009d0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170aff2249bcc0c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb121ffb700 1 -- 172.21.15.98:0/41234529 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1300a1a20 con 0x561fda762310 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb121ffb700 1 -- 172.21.15.98:0/41234529 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1300a6250 con 0x561fda762310 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.235+0000 7fb121ffb700 1 -- 172.21.15.98:0/41234529 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13009d8d0 con 0x561fda762310 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 -- 172.21.15.98:0/41234529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 msgr2=0x561fda7710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 --2- 172.21.15.98:0/41234529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda7710b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb130099ad0 tx=0x7fb13009d0a0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 -- 172.21.15.98:0/41234529 shutdown_connections 2024-04-25T17:41:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 --2- 172.21.15.98:0/41234529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 0x561fda761dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 --2- 172.21.15.98:0/41234529 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda7739e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 --2- 172.21.15.98:0/41234529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda7710b0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 -- 172.21.15.98:0/41234529 >> 172.21.15.98:0/41234529 conn(0x561fda68f790 msgr2=0x561fda75ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 -- 172.21.15.98:0/41234529 shutdown_connections 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 -- 172.21.15.98:0/41234529 wait complete. 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 -- start start 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.236+0000 7fb14c484b80 1 Processor -- start 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7619f0 0x561fda770e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda76e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7715f0 0x561fda76e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb14c484b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561fda6b4e30 con 0x561fda762310 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb14c484b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561fda653cd0 con 0x561fda7619f0 2024-04-25T17:41:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb14c484b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561fda773f20 con 0x561fda7715f0 2024-04-25T17:41:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda76e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb14c484b80 10 client.0 _ll_register_callbacks cb 0x561fda64da50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda76e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50130/0 (socket says 172.21.15.98:50130) 2024-04-25T17:41:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13affd700 1 -- 172.21.15.98:0/3801384389 learned_addr learned my addr 172.21.15.98:0/3801384389 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7619f0 0x561fda770e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13bfff700 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7715f0 0x561fda76e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13affd700 1 -- 172.21.15.98:0/3801384389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7715f0 msgr2=0x561fda76e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13affd700 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7715f0 0x561fda76e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13affd700 1 -- 172.21.15.98:0/3801384389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7619f0 msgr2=0x561fda770e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13affd700 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7619f0 0x561fda770e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13affd700 1 -- 172.21.15.98:0/3801384389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fda66c370 con 0x561fda762310 2024-04-25T17:41:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb13affd700 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda76e130 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb1300d1930 tx=0x7fb1300ced30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd4fe9be9dcc17a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb11affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:41:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb11affd700 1 -- 172.21.15.98:0/3801384389 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1300b5030 con 0x561fda762310 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb11affd700 1 -- 172.21.15.98:0/3801384389 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fb13009b970 con 0x561fda762310 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.237+0000 7fb14c484b80 1 -- 172.21.15.98:0/3801384389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1300a6250 con 0x561fda762310 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb11affd700 1 -- 172.21.15.98:0/3801384389 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb1300a1a20 con 0x561fda762310 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb14c484b80 10 client.15894 fetch_fsmap learned FSMap version 33 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb14c484b80 1 -- 172.21.15.98:0/3801384389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561fda77e660 con 0x561fda762310 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb11affd700 1 -- 172.21.15.98:0/3801384389 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1300a6250 con 0x561fda762310 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb11affd700 1 -- 172.21.15.98:0/3801384389 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fb1300bc370 con 0x561fda762310 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb14c484b80 10 client.15894 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb14c484b80 10 client.15894 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb14c484b80 20 client.15894 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb14c484b80 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561fda780b30 0x561fda782f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb14c484b80 4 client.15894 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb14c484b80 1 -- 172.21.15.98:0/3801384389 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fda6c2620 con 0x561fda780b30 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.238+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561fda780b30 0x561fda782f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.239+0000 7fb11affd700 10 client.15894 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.239+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561fda780b30 0x561fda782f20 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bd0db9f8ac3d171 server_cookie=e9f5c558650b1734 in_seq=0 out_seq=0 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.239+0000 7fb11affd700 1 -- 172.21.15.98:0/3801384389 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561fda6c2620 con 0x561fda780b30 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.239+0000 7fb11affd700 10 client.15894 handle_command_reply: tid=0 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.239+0000 7fb14c484b80 1 client.15894 shutdown 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.239+0000 7fb14c484b80 1 -- 172.21.15.98:0/3801384389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 msgr2=0x561fda76e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb14c484b80 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda76e130 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb1300d1930 tx=0x7fb1300ced30 comp rx=0 tx=0).stop 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb14c484b80 1 -- 172.21.15.98:0/3801384389 shutdown_connections 2024-04-25T17:41:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb14c484b80 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561fda780b30 0x561fda782f20 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb14c484b80 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7715f0 0x561fda76e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb14c484b80 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7619f0 0x561fda770e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb14c484b80 1 --2- 172.21.15.98:0/3801384389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda76e130 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb14c484b80 1 -- 172.21.15.98:0/3801384389 >> 172.21.15.98:0/3801384389 conn(0x561fda68f790 msgr2=0x561fda75db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb11affd700 0 client.15894 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb14c484b80 1 -- 172.21.15.98:0/3801384389 shutdown_connections 2024-04-25T17:41:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb14c484b80 1 -- 172.21.15.98:0/3801384389 wait complete. 2024-04-25T17:41:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.240+0000 7fb14c484b80 20 client.15894 trim_cache size 0 max 16384 2024-04-25T17:41:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb14c484b80 1 Processor -- start 2024-04-25T17:41:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb14c484b80 1 -- start start 2024-04-25T17:41:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda7619f0 0x561fda76d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda762310 0x561fda7698e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda77cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb14c484b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c1ad940 con 0x561fda7619f0 2024-04-25T17:41:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb14c484b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c1ab1a0 con 0x561fda7715f0 2024-04-25T17:41:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb14c484b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c1a9010 con 0x561fda762310 2024-04-25T17:41:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb13b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda7619f0 0x561fda76d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb13b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda7619f0 0x561fda76d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50144/0 (socket says 172.21.15.98:50144) 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb13b7fe700 1 -- 172.21.15.98:0/2035448071 learned_addr learned my addr 172.21.15.98:0/2035448071 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb13b7fe700 1 -- 172.21.15.98:0/2035448071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda762310 msgr2=0x561fda7698e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/2035448071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda762310 0x561fda7698e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb13b7fe700 1 -- 172.21.15.98:0/2035448071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 msgr2=0x561fda77cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/2035448071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda77cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.259+0000 7fb13b7fe700 1 -- 172.21.15.98:0/2035448071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fda75e020 con 0x561fda7619f0 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/2035448071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda7619f0 0x561fda76d0b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb12c010e30 tx=0x7fb12c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b9b00abac274839 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb1217fa700 1 -- 172.21.15.98:0/2035448071 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c020c20 con 0x561fda7619f0 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb1217fa700 1 -- 172.21.15.98:0/2035448071 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb12c020dc0 con 0x561fda7619f0 2024-04-25T17:41:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb1217fa700 1 -- 172.21.15.98:0/2035448071 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c004bb0 con 0x561fda7619f0 2024-04-25T17:41:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb14c484b80 1 -- 172.21.15.98:0/2035448071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda7619f0 msgr2=0x561fda76d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb14c484b80 1 --2- 172.21.15.98:0/2035448071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda7619f0 0x561fda76d0b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb12c010e30 tx=0x7fb12c013000 comp rx=0 tx=0).stop 2024-04-25T17:41:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb14c484b80 1 -- 172.21.15.98:0/2035448071 shutdown_connections 2024-04-25T17:41:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb14c484b80 1 --2- 172.21.15.98:0/2035448071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda77cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb14c484b80 1 --2- 172.21.15.98:0/2035448071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda7619f0 0x561fda76d0b0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb14c484b80 1 --2- 172.21.15.98:0/2035448071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda762310 0x561fda7698e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb14c484b80 1 -- 172.21.15.98:0/2035448071 >> 172.21.15.98:0/2035448071 conn(0x561fda68f790 msgr2=0x561fda763990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.260+0000 7fb14c484b80 1 -- 172.21.15.98:0/2035448071 shutdown_connections 2024-04-25T17:41:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb14c484b80 1 -- 172.21.15.98:0/2035448071 wait complete. 2024-04-25T17:41:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb14c484b80 1 -- start start 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb14c484b80 1 Processor -- start 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 0x561fda779330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda776040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb14c484b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda776580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb14c484b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c1a9660 con 0x561fda762310 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb14c484b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c1aaad0 con 0x561fda7715f0 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb14c484b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c1a9d30 con 0x561fda7619f0 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb14c484b80 10 client.0 _ll_register_callbacks cb 0x561fda64da50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb13affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda776040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb13affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda776040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50156/0 (socket says 172.21.15.98:50156) 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb13affd700 1 -- 172.21.15.98:0/2646554351 learned_addr learned my addr 172.21.15.98:0/2646554351 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.261+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 0x561fda779330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb13bfff700 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda776580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb13affd700 1 -- 172.21.15.98:0/2646554351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 msgr2=0x561fda779330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb13affd700 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 0x561fda779330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb13affd700 1 -- 172.21.15.98:0/2646554351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 msgr2=0x561fda776580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb13affd700 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda776580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb13affd700 1 -- 172.21.15.98:0/2646554351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fda776c80 con 0x561fda762310 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb13bfff700 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda776580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb13affd700 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda776040 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb130099ad0 tx=0x7fb130009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af881eb66513057f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb11affd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb11affd700 1 -- 172.21.15.98:0/2646554351 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1300a6250 con 0x561fda762310 2024-04-25T17:41:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb11affd700 1 -- 172.21.15.98:0/2646554351 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fb130022470 con 0x561fda762310 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb14c484b80 1 -- 172.21.15.98:0/2646554351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb12c020dc0 con 0x561fda762310 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb11affd700 1 -- 172.21.15.98:0/2646554351 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb130068020 con 0x561fda762310 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb14c484b80 10 client.15900 fetch_fsmap learned FSMap version 33 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb14c484b80 1 -- 172.21.15.98:0/2646554351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561fda776c80 con 0x561fda762310 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb11affd700 1 -- 172.21.15.98:0/2646554351 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb12c020dc0 con 0x561fda762310 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb11affd700 1 -- 172.21.15.98:0/2646554351 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fb1300bc2e0 con 0x561fda762310 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb14c484b80 10 client.15900 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb14c484b80 10 client.15900 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb14c484b80 20 client.15900 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb14c484b80 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561fda7e81d0 0x561fda7ea5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb14c484b80 4 client.15900 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "839b44ee", "release": true, "await": true}] 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.262+0000 7fb14c484b80 1 -- 172.21.15.98:0/2646554351 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "839b44ee", "release": true, "await": true}) -- 0x561fda6bc6c0 con 0x561fda7e81d0 2024-04-25T17:41:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.263+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561fda7e81d0 0x561fda7ea5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.263+0000 7fb11affd700 10 client.15900 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.263+0000 7fb13b7fe700 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561fda7e81d0 0x561fda7ea5f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51af94af75d8239e server_cookie=f8e981fd71081d3c in_seq=0 out_seq=0 2024-04-25T17:41:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb11affd700 1 -- 172.21.15.98:0/2646554351 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x561fda6bc6c0 con 0x561fda7e81d0 2024-04-25T17:41:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb11affd700 10 client.15900 handle_command_reply: tid=0 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 client.15900 shutdown 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 -- 172.21.15.98:0/2646554351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 msgr2=0x561fda776040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda776040 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb130099ad0 tx=0x7fb130009ca0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 -- 172.21.15.98:0/2646554351 shutdown_connections 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561fda7e81d0 0x561fda7ea5f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561fda7619f0 0x561fda779330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb11affd700 0 client.15900 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561fda7715f0 0x561fda776580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 --2- 172.21.15.98:0/2646554351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561fda762310 0x561fda776040 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 -- 172.21.15.98:0/2646554351 >> 172.21.15.98:0/2646554351 conn(0x561fda68f790 msgr2=0x561fda764220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 -- 172.21.15.98:0/2646554351 shutdown_connections 2024-04-25T17:41:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 1 -- 172.21.15.98:0/2646554351 wait complete. 2024-04-25T17:41:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.271+0000 7fb14c484b80 20 client.15900 trim_cache size 0 max 16384 2024-04-25T17:41:09.273 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:41:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 33, 2024-04-25T17:41:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:41:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 16, 2024-04-25T17:41:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:41:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "839b44ee": { 2024-04-25T17:41:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 16, 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 85.2, 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:41:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:41:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:41:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:41:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:41:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:41:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:41:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:41:09.275 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:41:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.272+0000 7fb141bfb700 1 -- 172.21.15.98:0/3643850560 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb1240a3db0 msgr2=0x7fb1240a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.272+0000 7fb141bfb700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb1240a3db0 0x7fb1240a6270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb12c005950 tx=0x7fb12c00b410 comp rx=0 tx=0).stop 2024-04-25T17:41:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.272+0000 7fb141bfb700 1 -- 172.21.15.98:0/3643850560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c103450 msgr2=0x7fb13c199690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.272+0000 7fb141bfb700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c103450 0x7fb13c199690 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb130000c00 tx=0x7fb1300047b0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.272+0000 7fb141bfb700 1 -- 172.21.15.98:0/3643850560 shutdown_connections 2024-04-25T17:41:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.272+0000 7fb141bfb700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb1240a3db0 0x7fb1240a6270 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.272+0000 7fb141bfb700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c102a90 0x7fb13c199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.272+0000 7fb141bfb700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb13c108a50 0x7fb13c19da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.272+0000 7fb141bfb700 1 --2- 172.21.15.98:0/3643850560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb13c103450 0x7fb13c199690 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.272+0000 7fb141bfb700 1 -- 172.21.15.98:0/3643850560 >> 172.21.15.98:0/3643850560 conn(0x7fb13c079ba0 msgr2=0x7fb13c1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.273+0000 7fb141bfb700 1 -- 172.21.15.98:0/3643850560 shutdown_connections 2024-04-25T17:41:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:41:09.273+0000 7fb141bfb700 1 -- 172.21.15.98:0/3643850560 wait complete. 2024-04-25T17:41:10.030 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '839b44ee', seconds elapsed: 39.0 2024-04-25T17:41:10.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:10 smithi098 ceph-mon[29192]: pgmap v161: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 41 KiB/s wr, 25 op/s 2024-04-25T17:41:10.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:10 smithi203 ceph-mon[32834]: pgmap v161: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 41 KiB/s wr, 25 op/s 2024-04-25T17:41:10.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:10 smithi173 ceph-mon[32856]: pgmap v161: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 41 KiB/s wr, 25 op/s 2024-04-25T17:41:11.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 280.0 s, 1382.6 tps, lat 493.615 ms stddev 3801.191 2024-04-25T17:41:12.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:12 smithi098 ceph-mon[29192]: pgmap v162: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 693 KiB/s wr, 55 op/s 2024-04-25T17:41:12.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:12 smithi203 ceph-mon[32834]: pgmap v162: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 693 KiB/s wr, 55 op/s 2024-04-25T17:41:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:12 smithi173 ceph-mon[32856]: pgmap v162: 97 pgs: 97 active+clean; 2.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 693 KiB/s wr, 55 op/s 2024-04-25T17:41:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.424+0000 7f0b18f68640 1 -- 172.21.15.98:0/2967023404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b14106170 msgr2=0x7f0b1410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.424+0000 7f0b18f68640 1 --2- 172.21.15.98:0/2967023404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b14106170 0x7f0b1410dd40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009ca0 tx=0x7f0b0c02a8c0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.424+0000 7f0b18f68640 1 -- 172.21.15.98:0/2967023404 shutdown_connections 2024-04-25T17:41:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.424+0000 7f0b18f68640 1 --2- 172.21.15.98:0/2967023404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b14106170 0x7f0b1410dd40 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.424+0000 7f0b18f68640 1 --2- 172.21.15.98:0/2967023404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b14101aa0 0x7f0b14105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.424+0000 7f0b18f68640 1 --2- 172.21.15.98:0/2967023404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b141010f0 0x7f0b141014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.424+0000 7f0b18f68640 1 -- 172.21.15.98:0/2967023404 >> 172.21.15.98:0/2967023404 conn(0x7f0b14077fd0 msgr2=0x7f0b140783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:14.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.424+0000 7f0b18f68640 1 -- 172.21.15.98:0/2967023404 shutdown_connections 2024-04-25T17:41:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.424+0000 7f0b18f68640 1 -- 172.21.15.98:0/2967023404 wait complete. 2024-04-25T17:41:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.425+0000 7f0b18f68640 1 Processor -- start 2024-04-25T17:41:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.425+0000 7f0b18f68640 1 -- start start 2024-04-25T17:41:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b18f68640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b141010f0 0x7f0b14199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b18f68640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b14106170 0x7f0b14199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b18f68640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b14199ff0 0x7f0b141a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b18f68640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141110c0 con 0x7f0b14199ff0 2024-04-25T17:41:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b18f68640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0b14110f40 con 0x7f0b14106170 2024-04-25T17:41:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b18f68640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b14111240 con 0x7f0b141010f0 2024-04-25T17:41:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b11d9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b14106170 0x7f0b14199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b11d9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b14106170 0x7f0b14199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56150/0 (socket says 172.21.15.98:56150) 2024-04-25T17:41:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b11d9b640 1 -- 172.21.15.98:0/1826192140 learned_addr learned my addr 172.21.15.98:0/1826192140 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b1259c640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b141010f0 0x7f0b14199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b12d9d640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b14199ff0 0x7f0b141a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b11d9b640 1 -- 172.21.15.98:0/1826192140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b141010f0 msgr2=0x7f0b14199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b141010f0 0x7f0b14199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b11d9b640 1 -- 172.21.15.98:0/1826192140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b14199ff0 msgr2=0x7f0b141a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b14199ff0 0x7f0b141a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.426+0000 7f0b11d9b640 1 -- 172.21.15.98:0/1826192140 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b141a4da0 con 0x7f0b14106170 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.427+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b14106170 0x7f0b14199a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800d770 tx=0x7f0b0800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.427+0000 7f0b037fe640 1 -- 172.21.15.98:0/1826192140 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08017070 con 0x7f0b14106170 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.427+0000 7f0b037fe640 1 -- 172.21.15.98:0/1826192140 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b08009c30 con 0x7f0b14106170 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.427+0000 7f0b037fe640 1 -- 172.21.15.98:0/1826192140 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0800f620 con 0x7f0b14106170 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.427+0000 7f0b18f68640 1 -- 172.21.15.98:0/1826192140 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b141a5090 con 0x7f0b14106170 2024-04-25T17:41:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.427+0000 7f0b18f68640 1 -- 172.21.15.98:0/1826192140 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b141a5630 con 0x7f0b14106170 2024-04-25T17:41:14.432 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.431+0000 7f0b1b922740 1 Processor -- start 2024-04-25T17:41:14.432 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.431+0000 7f0b037fe640 1 -- 172.21.15.98:0/1826192140 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0b0801a070 con 0x7f0b14106170 2024-04-25T17:41:14.432 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.432+0000 7f0b037fe640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0afc0a3ec0 0x7f0afc0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.432 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.432+0000 7f0b037fe640 1 -- 172.21.15.98:0/1826192140 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f0b080d0230 con 0x7f0b14106170 2024-04-25T17:41:14.432 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.432+0000 7f0b12d9d640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0afc0a3ec0 0x7f0afc0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.432 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.432+0000 7f0b1b922740 1 -- start start 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.432+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 0x559bf72d5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.432+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5f40 0x559bf72e02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.432+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 0x559bf72e2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.432+0000 7f0b1b922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559bf723a970 con 0x559bf72d5f40 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.432+0000 7f0b1b922740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559bf71d74a0 con 0x559bf72e0800 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.432+0000 7f0b1b922740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559bf7236930 con 0x559bf72d5620 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b11d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5f40 0x559bf72e02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b12d9d640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0afc0a3ec0 0x7f0afc0a6380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c006010 tx=0x7f0b0c005ef0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b11d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5f40 0x559bf72e02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60570/0 (socket says 172.21.15.98:60570) 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b11d9b640 1 -- 172.21.15.98:0/2412556108 learned_addr learned my addr 172.21.15.98:0/2412556108 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b12d9d640 1 --2- 172.21.15.98:0/2412556108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 0x559bf72e2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b1259c640 1 --2- 172.21.15.98:0/2412556108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 0x559bf72d5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b11d9b640 1 -- 172.21.15.98:0/2412556108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 msgr2=0x559bf72d5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/2412556108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 0x559bf72d5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b11d9b640 1 -- 172.21.15.98:0/2412556108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 msgr2=0x559bf72e2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/2412556108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 0x559bf72e2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b11d9b640 1 -- 172.21.15.98:0/2412556108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bf6f4d210 con 0x559bf72d5f40 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b12d9d640 1 --2- 172.21.15.98:0/2412556108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 0x559bf72e2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/2412556108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5f40 0x559bf72e02c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0b08098600 tx=0x7f0b0809bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360e4066eb140d85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b00ff9640 1 -- 172.21.15.98:0/2412556108 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08021070 con 0x559bf72d5f40 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b00ff9640 1 -- 172.21.15.98:0/2412556108 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b080a0550 con 0x559bf72d5f40 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.433+0000 7f0b00ff9640 1 -- 172.21.15.98:0/2412556108 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b080a0320 con 0x559bf72d5f40 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.434+0000 7f0b1b922740 1 -- 172.21.15.98:0/2412556108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5f40 msgr2=0x559bf72e02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.434+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2412556108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5f40 0x559bf72e02c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f0b08098600 tx=0x7f0b0809bc00 comp rx=0 tx=0).stop 2024-04-25T17:41:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.434+0000 7f0b1b922740 1 -- 172.21.15.98:0/2412556108 shutdown_connections 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.434+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2412556108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 0x559bf72e2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.434+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2412556108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5f40 0x559bf72e02c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.434+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2412556108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 0x559bf72d5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.434+0000 7f0b1b922740 1 -- 172.21.15.98:0/2412556108 >> 172.21.15.98:0/2412556108 conn(0x559bf72d1b30 msgr2=0x559bf72d1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.434+0000 7f0b1b922740 1 -- 172.21.15.98:0/2412556108 shutdown_connections 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.434+0000 7f0b1b922740 1 -- 172.21.15.98:0/2412556108 wait complete. 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.434+0000 7f0b1b922740 1 -- start start 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b1b922740 1 Processor -- start 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5620 0x559bf72e01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 0x559bf72dda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b1b922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559bf72e33d0 con 0x559bf72d5620 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b1b922740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559bf71d50b0 con 0x559bf72e0800 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b1b922740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559bf72e3550 con 0x559bf72d5f40 2024-04-25T17:41:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b1b922740 10 client.0 _ll_register_callbacks cb 0x559bf71c2d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b11d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b1259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5620 0x559bf72e01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b11d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:57908/0 (socket says 172.21.15.98:57908) 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.435+0000 7f0b11d9b640 1 -- 172.21.15.98:0/1643476639 learned_addr learned my addr 172.21.15.98:0/1643476639 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0b12d9d640 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 0x559bf72dda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0b11d9b640 1 -- 172.21.15.98:0/1643476639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 msgr2=0x559bf72dda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 0x559bf72dda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0b11d9b640 1 -- 172.21.15.98:0/1643476639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5620 msgr2=0x559bf72e01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5620 0x559bf72e01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0b11d9b640 1 -- 172.21.15.98:0/1643476639 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bf7019620 con 0x559bf72d5f40 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0b1259c640 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5620 0x559bf72e01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dd4a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0b0809bea0 tx=0x7f0b08098a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68479303c4ecfcb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0af9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:41:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0af9ffb640 1 -- 172.21.15.98:0/1643476639 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b080b9070 con 0x559bf72d5f40 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0b1b922740 1 -- 172.21.15.98:0/1643476639 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b080a0550 con 0x559bf72d5f40 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.436+0000 7f0af9ffb640 1 -- 172.21.15.98:0/1643476639 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f0b080a0120 con 0x559bf72d5f40 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0af9ffb640 1 -- 172.21.15.98:0/1643476639 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0b08021070 con 0x559bf72d5f40 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0b1b922740 10 client.25420 fetch_fsmap learned FSMap version 33 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0b1b922740 1 -- 172.21.15.98:0/1643476639 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559bf72e33d0 con 0x559bf72d5f40 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0af9ffb640 1 -- 172.21.15.98:0/1643476639 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b080a0550 con 0x559bf72d5f40 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0af9ffb640 1 -- 172.21.15.98:0/1643476639 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f0b08098080 con 0x559bf72d5f40 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0b1b922740 10 client.25420 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0b1b922740 10 client.25420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0b1b922740 20 client.25420 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0b1b922740 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf72f90c0 0x559bf72fb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0b1b922740 4 client.25420 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0b1b922740 1 -- 172.21.15.98:0/1643476639 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bf7248390 con 0x559bf72f90c0 2024-04-25T17:41:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.437+0000 7f0b1259c640 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf72f90c0 0x559bf72fb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.438+0000 7f0af9ffb640 10 client.25420 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.438+0000 7f0b1259c640 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf72f90c0 0x559bf72fb4e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b259f7e117dc8e2c server_cookie=45b3e5d4b49d7b05 in_seq=0 out_seq=0 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.438+0000 7f0af9ffb640 1 -- 172.21.15.98:0/1643476639 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559bf7248390 con 0x559bf72f90c0 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0af9ffb640 10 client.25420 handle_command_reply: tid=0 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 client.25420 shutdown 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 -- 172.21.15.98:0/1643476639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 msgr2=0x559bf72dd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dd4a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0b0809bea0 tx=0x7f0b08098a40 comp rx=0 tx=0).stop 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 -- 172.21.15.98:0/1643476639 shutdown_connections 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf72f90c0 0x559bf72fb4e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72e0800 0x559bf72dda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dd4a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 --2- 172.21.15.98:0/1643476639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72d5620 0x559bf72e01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 -- 172.21.15.98:0/1643476639 >> 172.21.15.98:0/1643476639 conn(0x559bf72d1b30 msgr2=0x559bf72d0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0af9ffb640 0 client.25420 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 -- 172.21.15.98:0/1643476639 shutdown_connections 2024-04-25T17:41:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 1 -- 172.21.15.98:0/1643476639 wait complete. 2024-04-25T17:41:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.439+0000 7f0b1b922740 20 client.25420 trim_cache size 0 max 16384 2024-04-25T17:41:14.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:14 smithi098 ceph-mon[29192]: pgmap v163: 97 pgs: 97 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 81 op/s 2024-04-25T17:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.461+0000 7f0b1b922740 1 Processor -- start 2024-04-25T17:41:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.461+0000 7f0b1b922740 1 -- start start 2024-04-25T17:41:14.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:14 smithi203 ceph-mon[32834]: pgmap v163: 97 pgs: 97 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 81 op/s 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5620 0x559bf72ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72e0800 0x559bf7300d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b1b922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141ae1f0 con 0x559bf72e0800 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b1b922740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141aba90 con 0x559bf72d5620 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b1b922740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141a98e0 con 0x559bf72d5f40 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b1259c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5620 0x559bf72ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b1259c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5620 0x559bf72ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56176/0 (socket says 172.21.15.98:56176) 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b1259c640 1 -- 172.21.15.98:0/2286617720 learned_addr learned my addr 172.21.15.98:0/2286617720 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/2286617720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.462+0000 7f0b1259c640 1 -- 172.21.15.98:0/2286617720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 msgr2=0x559bf72dfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.463+0000 7f0b1259c640 1 --2- 172.21.15.98:0/2286617720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.463+0000 7f0b1259c640 1 -- 172.21.15.98:0/2286617720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72e0800 msgr2=0x559bf7300d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:14.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.463+0000 7f0b1259c640 1 --2- 172.21.15.98:0/2286617720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72e0800 0x559bf7300d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.463+0000 7f0b1259c640 1 -- 172.21.15.98:0/2286617720 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bf72d5d70 con 0x559bf72d5620 2024-04-25T17:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.463+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/2286617720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.463+0000 7f0b1259c640 1 --2- 172.21.15.98:0/2286617720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5620 0x559bf72ccb10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400bfd0 tx=0x7f0b04013830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35cde8306b56aa02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.463+0000 7f0afbfff640 1 -- 172.21.15.98:0/2286617720 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b04023070 con 0x559bf72d5620 2024-04-25T17:41:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.463+0000 7f0afbfff640 1 -- 172.21.15.98:0/2286617720 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b0401d4e0 con 0x559bf72d5620 2024-04-25T17:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.463+0000 7f0afbfff640 1 -- 172.21.15.98:0/2286617720 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0401c9c0 con 0x559bf72d5620 2024-04-25T17:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.464+0000 7f0b1b922740 1 -- 172.21.15.98:0/2286617720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5620 msgr2=0x559bf72ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.464+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2286617720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5620 0x559bf72ccb10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400bfd0 tx=0x7f0b04013830 comp rx=0 tx=0).stop 2024-04-25T17:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.465+0000 7f0b1b922740 1 -- 172.21.15.98:0/2286617720 shutdown_connections 2024-04-25T17:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.465+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2286617720 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72e0800 0x559bf7300d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.465+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2286617720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5f40 0x559bf72dfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.465+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2286617720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5620 0x559bf72ccb10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.465+0000 7f0b1b922740 1 -- 172.21.15.98:0/2286617720 >> 172.21.15.98:0/2286617720 conn(0x559bf72d1b30 msgr2=0x559bf72d0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.465+0000 7f0b1b922740 1 -- 172.21.15.98:0/2286617720 shutdown_connections 2024-04-25T17:41:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.465+0000 7f0b1b922740 1 -- 172.21.15.98:0/2286617720 wait complete. 2024-04-25T17:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.465+0000 7f0b1b922740 1 -- start start 2024-04-25T17:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.465+0000 7f0b1b922740 1 Processor -- start 2024-04-25T17:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.466+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 0x559bf72f12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.466+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5f40 0x559bf72d9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.466+0000 7f0b1b922740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72e0800 0x559bf72da270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.466+0000 7f0b1b922740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141a9f50 con 0x559bf72e0800 2024-04-25T17:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.466+0000 7f0b1b922740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141ab3c0 con 0x559bf72d5f40 2024-04-25T17:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.466+0000 7f0b1b922740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141aa620 con 0x559bf72d5620 2024-04-25T17:41:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.466+0000 7f0b1b922740 10 client.0 _ll_register_callbacks cb 0x7f0b0809bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:14.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b12d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72e0800 0x559bf72da270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b12d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72e0800 0x559bf72da270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60588/0 (socket says 172.21.15.98:60588) 2024-04-25T17:41:14.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b12d9d640 1 -- 172.21.15.98:0/2529386311 learned_addr learned my addr 172.21.15.98:0/2529386311 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b1259c640 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 0x559bf72f12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5f40 0x559bf72d9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b1259c640 1 -- 172.21.15.98:0/2529386311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5f40 msgr2=0x559bf72d9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b1259c640 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5f40 0x559bf72d9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b1259c640 1 -- 172.21.15.98:0/2529386311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72e0800 msgr2=0x559bf72da270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b1259c640 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72e0800 0x559bf72da270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b1259c640 1 -- 172.21.15.98:0/2529386311 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bf72da930 con 0x559bf72d5620 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b1259c640 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 0x559bf72f12c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400bfd0 tx=0x7f0b04013300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59443d30390ae3af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0af9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0af9ffb640 1 -- 172.21.15.98:0/2529386311 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0402c030 con 0x559bf72d5620 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0af9ffb640 1 -- 172.21.15.98:0/2529386311 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7f0b0401d690 con 0x559bf72d5620 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.467+0000 7f0b1b922740 1 -- 172.21.15.98:0/2529386311 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b0401d4e0 con 0x559bf72d5620 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0af9ffb640 1 -- 172.21.15.98:0/2529386311 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0b04023040 con 0x559bf72d5620 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0b1b922740 10 client.25426 fetch_fsmap learned FSMap version 33 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0b1b922740 1 -- 172.21.15.98:0/2529386311 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559bf735f4a0 con 0x559bf72d5620 2024-04-25T17:41:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0af9ffb640 1 -- 172.21.15.98:0/2529386311 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b0401d4e0 con 0x559bf72d5620 2024-04-25T17:41:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0af9ffb640 1 -- 172.21.15.98:0/2529386311 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f0b04003890 con 0x559bf72d5620 2024-04-25T17:41:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0b1b922740 10 client.25426 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T17:41:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0b1b922740 10 client.25426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:41:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0b1b922740 20 client.25426 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf735fb70 0x559bf7361f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0b1b922740 4 client.25426 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:41:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0b1b922740 1 -- 172.21.15.98:0/2529386311 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559bf7242200 con 0x559bf735fb70 2024-04-25T17:41:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.468+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf735fb70 0x559bf7361f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.469+0000 7f0b11d9b640 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf735fb70 0x559bf7361f60 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b4fec222af20245 server_cookie=e4ea627f1c2b32a8 in_seq=0 out_seq=0 2024-04-25T17:41:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.469+0000 7f0af9ffb640 10 client.25426 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0af9ffb640 1 -- 172.21.15.98:0/2529386311 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559bf7242200 con 0x559bf735fb70 2024-04-25T17:41:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0af9ffb640 10 client.25426 handle_command_reply: tid=0 2024-04-25T17:41:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0b1b922740 1 client.25426 shutdown 2024-04-25T17:41:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0b1b922740 1 -- 172.21.15.98:0/2529386311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 msgr2=0x559bf72f12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 0x559bf72f12c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0b0400bfd0 tx=0x7f0b04013300 comp rx=0 tx=0).stop 2024-04-25T17:41:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0b1b922740 1 -- 172.21.15.98:0/2529386311 shutdown_connections 2024-04-25T17:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf735fb70 0x559bf7361f60 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0af9ffb640 0 client.25426 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf72e0800 0x559bf72da270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf72d5f40 0x559bf72d9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0b1b922740 1 --2- 172.21.15.98:0/2529386311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf72d5620 0x559bf72f12c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.480+0000 7f0b1b922740 1 -- 172.21.15.98:0/2529386311 >> 172.21.15.98:0/2529386311 conn(0x559bf72d1b30 msgr2=0x559bf72d6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.481+0000 7f0b1b922740 1 -- 172.21.15.98:0/2529386311 shutdown_connections 2024-04-25T17:41:14.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.481+0000 7f0b1b922740 1 -- 172.21.15.98:0/2529386311 wait complete. 2024-04-25T17:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.481+0000 7f0b1b922740 20 client.25426 trim_cache size 0 max 16384 2024-04-25T17:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:41:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 -- 172.21.15.98:0/1826192140 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0afc0a3ec0 msgr2=0x7f0afc0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0afc0a3ec0 0x7f0afc0a6380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c006010 tx=0x7f0b0c005ef0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 -- 172.21.15.98:0/1826192140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b14106170 msgr2=0x7f0b14199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b14106170 0x7f0b14199a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800d770 tx=0x7f0b0800dc30 comp rx=0 tx=0).stop 2024-04-25T17:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 -- 172.21.15.98:0/1826192140 shutdown_connections 2024-04-25T17:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0afc0a3ec0 0x7f0afc0a6380 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b14199ff0 0x7f0b141a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0b14106170 0x7f0b14199a60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 --2- 172.21.15.98:0/1826192140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b141010f0 0x7f0b14199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 -- 172.21.15.98:0/1826192140 >> 172.21.15.98:0/1826192140 conn(0x7f0b14077fd0 msgr2=0x7f0b140ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 -- 172.21.15.98:0/1826192140 shutdown_connections 2024-04-25T17:41:14.484 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:14.482+0000 7f0b18f68640 1 -- 172.21.15.98:0/1826192140 wait complete. 2024-04-25T17:41:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:14 smithi173 ceph-mon[32856]: pgmap v163: 97 pgs: 97 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 2.0 MiB/s wr, 81 op/s 2024-04-25T17:41:16.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:16 smithi098 ceph-mon[29192]: pgmap v164: 97 pgs: 97 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.0 MiB/s wr, 80 op/s 2024-04-25T17:41:16.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:16 smithi203 ceph-mon[32834]: pgmap v164: 97 pgs: 97 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.0 MiB/s wr, 80 op/s 2024-04-25T17:41:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:16 smithi173 ceph-mon[32856]: pgmap v164: 97 pgs: 97 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 2.0 MiB/s wr, 80 op/s 2024-04-25T17:41:16.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 285.0 s, 3191.0 tps, lat 31.292 ms stddev 32.329 2024-04-25T17:41:18.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:18 smithi098 ceph-mon[29192]: pgmap v165: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 5.0 MiB/s wr, 388 op/s 2024-04-25T17:41:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:18 smithi203 ceph-mon[32834]: pgmap v165: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 5.0 MiB/s wr, 388 op/s 2024-04-25T17:41:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:18 smithi173 ceph-mon[32856]: pgmap v165: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 5.0 MiB/s wr, 388 op/s 2024-04-25T17:41:20.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:20 smithi098 ceph-mon[29192]: pgmap v166: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.0 MiB/s wr, 380 op/s 2024-04-25T17:41:20.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:20 smithi203 ceph-mon[32834]: pgmap v166: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.0 MiB/s wr, 380 op/s 2024-04-25T17:41:20.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:20 smithi173 ceph-mon[32856]: pgmap v166: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.0 MiB/s wr, 380 op/s 2024-04-25T17:41:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 290.0 s, 2927.6 tps, lat 34.118 ms stddev 39.437 2024-04-25T17:41:22.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:22 smithi098 ceph-mon[29192]: pgmap v167: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.3 MiB/s wr, 425 op/s 2024-04-25T17:41:22.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:22 smithi203 ceph-mon[32834]: pgmap v167: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.3 MiB/s wr, 425 op/s 2024-04-25T17:41:22.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:22 smithi173 ceph-mon[32856]: pgmap v167: 97 pgs: 97 active+clean; 2.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.3 MiB/s wr, 425 op/s 2024-04-25T17:41:24.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:24 smithi098 ceph-mon[29192]: pgmap v168: 97 pgs: 97 active+clean; 2.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 648 op/s 2024-04-25T17:41:24.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:24 smithi203 ceph-mon[32834]: pgmap v168: 97 pgs: 97 active+clean; 2.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 648 op/s 2024-04-25T17:41:24.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:24 smithi173 ceph-mon[32856]: pgmap v168: 97 pgs: 97 active+clean; 2.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.2 MiB/s wr, 648 op/s 2024-04-25T17:41:26.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:26 smithi098 ceph-mon[29192]: pgmap v169: 97 pgs: 97 active+clean; 2.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.9 MiB/s wr, 605 op/s 2024-04-25T17:41:26.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:26 smithi203 ceph-mon[32834]: pgmap v169: 97 pgs: 97 active+clean; 2.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.9 MiB/s wr, 605 op/s 2024-04-25T17:41:26.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:26 smithi173 ceph-mon[32856]: pgmap v169: 97 pgs: 97 active+clean; 2.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.9 MiB/s wr, 605 op/s 2024-04-25T17:41:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 295.0 s, 2976.2 tps, lat 33.517 ms stddev 34.882 2024-04-25T17:41:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 -- 172.21.15.98:0/4013775990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffba01021a0 msgr2=0x7ffba0102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 --2- 172.21.15.98:0/4013775990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffba01021a0 0x7ffba0102580 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ffb94009900 tx=0x7ffb9402e820 comp rx=0 tx=0).stop 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 -- 172.21.15.98:0/4013775990 shutdown_connections 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 --2- 172.21.15.98:0/4013775990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba0107270 0x7ffba010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 --2- 172.21.15.98:0/4013775990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0102b50 0x7ffba0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 --2- 172.21.15.98:0/4013775990 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffba01021a0 0x7ffba0102580 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 -- 172.21.15.98:0/4013775990 >> 172.21.15.98:0/4013775990 conn(0x7ffba00fda00 msgr2=0x7ffba00ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 -- 172.21.15.98:0/4013775990 shutdown_connections 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 -- 172.21.15.98:0/4013775990 wait complete. 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 Processor -- start 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.616+0000 7ffba6e49640 1 -- start start 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba6e49640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba01021a0 0x7ffba0196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba6e49640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0107270 0x7ffba0196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba6e49640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffba019add0 0x7ffba019b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba4be5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba01021a0 0x7ffba0196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba4be5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba01021a0 0x7ffba0196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56044/0 (socket says 172.21.15.98:56044) 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba4be5640 1 -- 172.21.15.98:0/3126593837 learned_addr learned my addr 172.21.15.98:0/3126593837 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffb9ffff640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0107270 0x7ffba0196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffba0110010 con 0x7ffba019add0 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffba010fe90 con 0x7ffba01021a0 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffba0110190 con 0x7ffba0107270 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba53e6640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffba019add0 0x7ffba019b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.617+0000 7ffba4be5640 1 -- 172.21.15.98:0/3126593837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0107270 msgr2=0x7ffba0196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.618+0000 7ffba4be5640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0107270 0x7ffba0196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.618+0000 7ffba4be5640 1 -- 172.21.15.98:0/3126593837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffba019add0 msgr2=0x7ffba019b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.618+0000 7ffba4be5640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffba019add0 0x7ffba019b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.618+0000 7ffba4be5640 1 -- 172.21.15.98:0/3126593837 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba019b8c0 con 0x7ffba01021a0 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.618+0000 7ffba4be5640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba01021a0 0x7ffba0196510 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffb94005ab0 tx=0x7ffb940366c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.618+0000 7ffb9dffb640 1 -- 172.21.15.98:0/3126593837 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb94056020 con 0x7ffba01021a0 2024-04-25T17:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.618+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffba019bb50 con 0x7ffba01021a0 2024-04-25T17:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.618+0000 7ffb9dffb640 1 -- 172.21.15.98:0/3126593837 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb940368b0 con 0x7ffba01021a0 2024-04-25T17:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.618+0000 7ffb9dffb640 1 -- 172.21.15.98:0/3126593837 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb94051900 con 0x7ffba01021a0 2024-04-25T17:41:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.619+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffba006bc60 con 0x7ffba01021a0 2024-04-25T17:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.622+0000 7ffb9dffb640 1 -- 172.21.15.98:0/3126593837 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffb94059020 con 0x7ffba01021a0 2024-04-25T17:41:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.622+0000 7ffba9c35740 1 Processor -- start 2024-04-25T17:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffb9dffb640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb880a3dc0 0x7ffb880a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba9c35740 1 -- start start 2024-04-25T17:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffb9dffb640 1 -- 172.21.15.98:0/3126593837 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7ffb941054d0 con 0x7ffba01021a0 2024-04-25T17:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a1c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a2fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a32400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba9c35740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fdc8981970 con 0x55fdc8a1c520 2024-04-25T17:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba9c35740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fdc891e4a0 con 0x55fdc8a1ce40 2024-04-25T17:41:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba9c35740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fdc897d930 con 0x55fdc8a30010 2024-04-25T17:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba53e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a32400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba53e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a32400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:33068/0 (socket says 172.21.15.98:33068) 2024-04-25T17:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba53e6640 1 -- 172.21.15.98:0/4231418123 learned_addr learned my addr 172.21.15.98:0/4231418123 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.623+0000 7ffba53e6640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb880a3dc0 0x7ffb880a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.624+0000 7ffba53e6640 1 -- 172.21.15.98:0/4231418123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a1ce40 msgr2=0x55fdc8a2fad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T17:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.624+0000 7ffba53e6640 1 --2- 172.21.15.98:0/4231418123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a2fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.624+0000 7ffba53e6640 1 -- 172.21.15.98:0/4231418123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 msgr2=0x55fdc8a1c900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T17:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.624+0000 7ffba53e6640 1 --2- 172.21.15.98:0/4231418123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a1c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.624+0000 7ffba53e6640 1 -- 172.21.15.98:0/4231418123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdc8694210 con 0x55fdc8a30010 2024-04-25T17:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.624+0000 7ffba53e6640 1 --2- 172.21.15.98:0/4231418123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a32400 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffb900129d0 tx=0x7ffb90012e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3052b60bcb117856 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.624+0000 7ffba53e6640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb880a3dc0 0x7ffb880a6280 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffb900107a0 tx=0x7ffb90008000 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.624+0000 7ffb86ffd640 1 -- 172.21.15.98:0/4231418123 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90012570 con 0x55fdc8a30010 2024-04-25T17:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.624+0000 7ffb86ffd640 1 -- 172.21.15.98:0/4231418123 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb90009e00 con 0x55fdc8a30010 2024-04-25T17:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.624+0000 7ffb86ffd640 1 -- 172.21.15.98:0/4231418123 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb900035f0 con 0x55fdc8a30010 2024-04-25T17:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.625+0000 7ffba9c35740 1 -- 172.21.15.98:0/4231418123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 msgr2=0x55fdc8a32400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.625+0000 7ffba9c35740 1 --2- 172.21.15.98:0/4231418123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a32400 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffb900129d0 tx=0x7ffb90012e90 comp rx=0 tx=0).stop 2024-04-25T17:41:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.625+0000 7ffba9c35740 1 -- 172.21.15.98:0/4231418123 shutdown_connections 2024-04-25T17:41:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.625+0000 7ffba9c35740 1 --2- 172.21.15.98:0/4231418123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a32400 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.625+0000 7ffba9c35740 1 --2- 172.21.15.98:0/4231418123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a2fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.625+0000 7ffba9c35740 1 --2- 172.21.15.98:0/4231418123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a1c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.625+0000 7ffba9c35740 1 -- 172.21.15.98:0/4231418123 >> 172.21.15.98:0/4231418123 conn(0x55fdc8a18b30 msgr2=0x55fdc8a18f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.626+0000 7ffba9c35740 1 -- 172.21.15.98:0/4231418123 shutdown_connections 2024-04-25T17:41:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.626+0000 7ffba9c35740 1 -- 172.21.15.98:0/4231418123 wait complete. 2024-04-25T17:41:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.626+0000 7ffba9c35740 1 -- start start 2024-04-25T17:41:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.626+0000 7ffba9c35740 1 Processor -- start 2024-04-25T17:41:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a297f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a29d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a30010 0x55fdc8a2f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffba9c35740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fdc8a1f500 con 0x55fdc8a1c520 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffba9c35740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fdc891c0b0 con 0x55fdc8a30010 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffba9c35740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fdc8a1f680 con 0x55fdc8a1ce40 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffb9ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a29d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffb9ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a29d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:33078/0 (socket says 172.21.15.98:33078) 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffb9ffff640 1 -- 172.21.15.98:0/4089886421 learned_addr learned my addr 172.21.15.98:0/4089886421 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffba9c35740 10 client.0 _ll_register_callbacks cb 0x55fdc8909d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffba4be5640 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a297f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffba53e6640 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a30010 0x55fdc8a2f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffb9ffff640 1 -- 172.21.15.98:0/4089886421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a30010 msgr2=0x55fdc8a2f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffb9ffff640 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a30010 0x55fdc8a2f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffb9ffff640 1 -- 172.21.15.98:0/4089886421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 msgr2=0x55fdc8a297f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:27.629 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffb9ffff640 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a297f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.627+0000 7ffb9ffff640 1 -- 172.21.15.98:0/4089886421 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdc8760620 con 0x55fdc8a1ce40 2024-04-25T17:41:27.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.628+0000 7ffb9ffff640 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a29d30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb98011950 tx=0x7ffb98011e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb6549e6001adbe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:27.630 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.628+0000 7ffb7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:41:27.631 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.628+0000 7ffb7ffff640 1 -- 172.21.15.98:0/4089886421 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb98003c40 con 0x55fdc8a1ce40 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.628+0000 7ffba9c35740 1 -- 172.21.15.98:0/4089886421 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb90009e00 con 0x55fdc8a1ce40 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.628+0000 7ffb7ffff640 1 -- 172.21.15.98:0/4089886421 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7ffb98003de0 con 0x55fdc8a1ce40 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.628+0000 7ffb7ffff640 1 -- 172.21.15.98:0/4089886421 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffb98004b20 con 0x55fdc8a1ce40 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.628+0000 7ffba9c35740 10 client.25438 fetch_fsmap learned FSMap version 33 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.629+0000 7ffba9c35740 1 -- 172.21.15.98:0/4089886421 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fdc8a474b0 con 0x55fdc8a1ce40 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.629+0000 7ffb7ffff640 1 -- 172.21.15.98:0/4089886421 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb90009e00 con 0x55fdc8a1ce40 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.629+0000 7ffb7ffff640 1 -- 172.21.15.98:0/4089886421 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7ffb98004300 con 0x55fdc8a1ce40 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.629+0000 7ffba9c35740 10 client.25438 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.629+0000 7ffba9c35740 10 client.25438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.629+0000 7ffba9c35740 20 client.25438 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.629+0000 7ffba9c35740 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fdc8a48920 0x55fdc8a4ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.629+0000 7ffba9c35740 4 client.25438 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.629+0000 7ffba9c35740 1 -- 172.21.15.98:0/4089886421 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdc898f390 con 0x55fdc8a48920 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.630+0000 7ffba4be5640 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fdc8a48920 0x55fdc8a4ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.630+0000 7ffba4be5640 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fdc8a48920 0x55fdc8a4ad70 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b0f5c254e984129 server_cookie=8e8e4e688744817a in_seq=0 out_seq=0 2024-04-25T17:41:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.631+0000 7ffb7ffff640 10 client.25438 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.631+0000 7ffb7ffff640 1 -- 172.21.15.98:0/4089886421 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fdc898f390 con 0x55fdc8a48920 2024-04-25T17:41:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.631+0000 7ffb7ffff640 10 client.25438 handle_command_reply: tid=0 2024-04-25T17:41:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.632+0000 7ffba9c35740 1 client.25438 shutdown 2024-04-25T17:41:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.632+0000 7ffba9c35740 1 -- 172.21.15.98:0/4089886421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 msgr2=0x55fdc8a29d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.632+0000 7ffba9c35740 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a29d30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb98011950 tx=0x7ffb98011e10 comp rx=0 tx=0).stop 2024-04-25T17:41:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.634+0000 7ffba9c35740 1 -- 172.21.15.98:0/4089886421 shutdown_connections 2024-04-25T17:41:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.634+0000 7ffba9c35740 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fdc8a48920 0x55fdc8a4ad70 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.634+0000 7ffba9c35740 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a30010 0x55fdc8a2f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.634+0000 7ffba9c35740 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a29d30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.634+0000 7ffba9c35740 1 --2- 172.21.15.98:0/4089886421 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a297f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.634+0000 7ffb7ffff640 0 client.25438 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.634+0000 7ffba9c35740 1 -- 172.21.15.98:0/4089886421 >> 172.21.15.98:0/4089886421 conn(0x55fdc8a18b30 msgr2=0x55fdc8a1e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.635+0000 7ffba9c35740 1 -- 172.21.15.98:0/4089886421 shutdown_connections 2024-04-25T17:41:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.635+0000 7ffba9c35740 1 -- 172.21.15.98:0/4089886421 wait complete. 2024-04-25T17:41:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.636+0000 7ffba9c35740 20 client.25438 trim_cache size 0 max 16384 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba9c35740 1 Processor -- start 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba9c35740 1 -- start start 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a1c520 0x55fdc8a2f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a50680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a50bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba9c35740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffba01a59a0 con 0x55fdc8a1ce40 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba9c35740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffba01a3240 con 0x55fdc8a1c520 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba9c35740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffba00721b0 con 0x55fdc8a30010 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba53e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a50bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba53e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a50bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:33092/0 (socket says 172.21.15.98:33092) 2024-04-25T17:41:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba53e6640 1 -- 172.21.15.98:0/3369697706 learned_addr learned my addr 172.21.15.98:0/3369697706 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffba4be5640 1 --2- 172.21.15.98:0/3369697706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a1c520 0x55fdc8a2f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.665+0000 7ffb9ffff640 1 --2- 172.21.15.98:0/3369697706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a50680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.666+0000 7ffba53e6640 1 -- 172.21.15.98:0/3369697706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a1c520 msgr2=0x55fdc8a2f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.666+0000 7ffba53e6640 1 --2- 172.21.15.98:0/3369697706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a1c520 0x55fdc8a2f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.666+0000 7ffba53e6640 1 -- 172.21.15.98:0/3369697706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1ce40 msgr2=0x55fdc8a50680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.666+0000 7ffba53e6640 1 --2- 172.21.15.98:0/3369697706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a50680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.666+0000 7ffba53e6640 1 -- 172.21.15.98:0/3369697706 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdc8a18960 con 0x55fdc8a30010 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.666+0000 7ffba53e6640 1 --2- 172.21.15.98:0/3369697706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a50bc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffb900129d0 tx=0x7ffb90005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a87fa706077ab8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.666+0000 7ffb867fc640 1 -- 172.21.15.98:0/3369697706 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9001b040 con 0x55fdc8a30010 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.666+0000 7ffb867fc640 1 -- 172.21.15.98:0/3369697706 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb9002c030 con 0x55fdc8a30010 2024-04-25T17:41:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.666+0000 7ffb867fc640 1 -- 172.21.15.98:0/3369697706 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb900035f0 con 0x55fdc8a30010 2024-04-25T17:41:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.667+0000 7ffba9c35740 1 -- 172.21.15.98:0/3369697706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 msgr2=0x55fdc8a50bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.667+0000 7ffba9c35740 1 --2- 172.21.15.98:0/3369697706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a50bc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffb900129d0 tx=0x7ffb90005f90 comp rx=0 tx=0).stop 2024-04-25T17:41:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.667+0000 7ffba9c35740 1 -- 172.21.15.98:0/3369697706 shutdown_connections 2024-04-25T17:41:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.667+0000 7ffba9c35740 1 --2- 172.21.15.98:0/3369697706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a30010 0x55fdc8a50bc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.667+0000 7ffba9c35740 1 --2- 172.21.15.98:0/3369697706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a50680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.667+0000 7ffba9c35740 1 --2- 172.21.15.98:0/3369697706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a1c520 0x55fdc8a2f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.667+0000 7ffba9c35740 1 -- 172.21.15.98:0/3369697706 >> 172.21.15.98:0/3369697706 conn(0x55fdc8a18b30 msgr2=0x55fdc8a20190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.667+0000 7ffba9c35740 1 -- 172.21.15.98:0/3369697706 shutdown_connections 2024-04-25T17:41:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.667+0000 7ffba9c35740 1 -- 172.21.15.98:0/3369697706 wait complete. 2024-04-25T17:41:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.668+0000 7ffba9c35740 1 -- start start 2024-04-25T17:41:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.668+0000 7ffba9c35740 1 Processor -- start 2024-04-25T17:41:27.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.668+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a21a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a21fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffba9c35740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a30010 0x55fdc8a224f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffba9c35740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffba0072800 con 0x55fdc8a1c520 2024-04-25T17:41:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffba9c35740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffba0073c70 con 0x55fdc8a30010 2024-04-25T17:41:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffba9c35740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffba0072ed0 con 0x55fdc8a1ce40 2024-04-25T17:41:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffba9c35740 10 client.0 _ll_register_callbacks cb 0x7ffb98011560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffba4be5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a21a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffba4be5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a21a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44678/0 (socket says 172.21.15.98:44678) 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffba4be5640 1 -- 172.21.15.98:0/3238221782 learned_addr learned my addr 172.21.15.98:0/3238221782 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffb9ffff640 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a21fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffb9ffff640 1 -- 172.21.15.98:0/3238221782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a30010 msgr2=0x55fdc8a224f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffb9ffff640 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a30010 0x55fdc8a224f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffb9ffff640 1 -- 172.21.15.98:0/3238221782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 msgr2=0x55fdc8a21a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffb9ffff640 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a21a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.669+0000 7ffb9ffff640 1 -- 172.21.15.98:0/3238221782 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdc8aadc20 con 0x55fdc8a1ce40 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.670+0000 7ffb9ffff640 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a21fb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffb980047a0 tx=0x7ffb98011270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1400ae0b21424659 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.670+0000 7ffb7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.670+0000 7ffb7ffff640 1 -- 172.21.15.98:0/3238221782 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9800fd70 con 0x55fdc8a1ce40 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.670+0000 7ffb7ffff640 1 -- 172.21.15.98:0/3238221782 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7ffb98003c40 con 0x55fdc8a1ce40 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.670+0000 7ffba9c35740 1 -- 172.21.15.98:0/3238221782 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb9002c030 con 0x55fdc8a1ce40 2024-04-25T17:41:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.670+0000 7ffb7ffff640 1 -- 172.21.15.98:0/3238221782 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffb98003e40 con 0x55fdc8a1ce40 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.670+0000 7ffba9c35740 10 client.25444 fetch_fsmap learned FSMap version 33 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.670+0000 7ffba9c35740 1 -- 172.21.15.98:0/3238221782 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fdc8aaef30 con 0x55fdc8a1ce40 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.670+0000 7ffb7ffff640 1 -- 172.21.15.98:0/3238221782 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb9002c030 con 0x55fdc8a1ce40 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.670+0000 7ffb7ffff640 1 -- 172.21.15.98:0/3238221782 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7ffb98034030 con 0x55fdc8a1ce40 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.671+0000 7ffba9c35740 10 client.25444 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.671+0000 7ffba9c35740 10 client.25444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.671+0000 7ffba9c35740 20 client.25444 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.671+0000 7ffba9c35740 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fdc8aaf6d0 0x55fdc8ab1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.671+0000 7ffba9c35740 4 client.25444 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.671+0000 7ffba9c35740 1 -- 172.21.15.98:0/3238221782 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fdc8989200 con 0x55fdc8aaf6d0 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.671+0000 7ffba4be5640 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fdc8aaf6d0 0x55fdc8ab1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.671+0000 7ffba4be5640 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fdc8aaf6d0 0x55fdc8ab1af0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=704238e75644a4d0 server_cookie=6e3c544fbc96c6cc in_seq=0 out_seq=0 2024-04-25T17:41:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.671+0000 7ffb7ffff640 10 client.25444 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:27.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.691+0000 7ffb7ffff640 1 -- 172.21.15.98:0/3238221782 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fdc8989200 con 0x55fdc8aaf6d0 2024-04-25T17:41:27.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.691+0000 7ffb7ffff640 10 client.25444 handle_command_reply: tid=0 2024-04-25T17:41:27.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.691+0000 7ffba9c35740 1 client.25444 shutdown 2024-04-25T17:41:27.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 1 -- 172.21.15.98:0/3238221782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 msgr2=0x55fdc8a21fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:27.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a21fb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffb980047a0 tx=0x7ffb98011270 comp rx=0 tx=0).stop 2024-04-25T17:41:27.693 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 1 -- 172.21.15.98:0/3238221782 shutdown_connections 2024-04-25T17:41:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55fdc8aaf6d0 0x55fdc8ab1af0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffb7ffff640 0 client.25444 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fdc8a30010 0x55fdc8a224f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fdc8a1ce40 0x55fdc8a21fb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 1 --2- 172.21.15.98:0/3238221782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fdc8a1c520 0x55fdc8a21a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 1 -- 172.21.15.98:0/3238221782 >> 172.21.15.98:0/3238221782 conn(0x55fdc8a18b30 msgr2=0x55fdc8a17420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:41:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:41:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:41:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:41:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 1 -- 172.21.15.98:0/3238221782 shutdown_connections 2024-04-25T17:41:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 1 -- 172.21.15.98:0/3238221782 wait complete. 2024-04-25T17:41:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.692+0000 7ffba9c35740 20 client.25444 trim_cache size 0 max 16384 2024-04-25T17:41:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.694+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb880a3dc0 msgr2=0x7ffb880a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.694+0000 7ffba6e49640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb880a3dc0 0x7ffb880a6280 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffb900107a0 tx=0x7ffb90008000 comp rx=0 tx=0).stop 2024-04-25T17:41:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.694+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba01021a0 msgr2=0x7ffba0196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.694+0000 7ffba6e49640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba01021a0 0x7ffba0196510 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ffb94005ab0 tx=0x7ffb940366c0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.694+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 shutdown_connections 2024-04-25T17:41:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.694+0000 7ffba6e49640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb880a3dc0 0x7ffb880a6280 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.694+0000 7ffba6e49640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffba019add0 0x7ffba019b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.694+0000 7ffba6e49640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0107270 0x7ffba0196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.694+0000 7ffba6e49640 1 --2- 172.21.15.98:0/3126593837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba01021a0 0x7ffba0196510 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.694+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 >> 172.21.15.98:0/3126593837 conn(0x7ffba00fda00 msgr2=0x7ffba00ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.695+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 shutdown_connections 2024-04-25T17:41:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:27.695+0000 7ffba6e49640 1 -- 172.21.15.98:0/3126593837 wait complete. 2024-04-25T17:41:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:28 smithi098 ceph-mon[29192]: pgmap v170: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 10 MiB/s wr, 900 op/s 2024-04-25T17:41:28.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:28 smithi203 ceph-mon[32834]: pgmap v170: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 10 MiB/s wr, 900 op/s 2024-04-25T17:41:28.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:28 smithi173 ceph-mon[32856]: pgmap v170: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 10 MiB/s wr, 900 op/s 2024-04-25T17:41:30.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:30 smithi098 ceph-mon[29192]: pgmap v171: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 592 op/s 2024-04-25T17:41:30.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:30 smithi203 ceph-mon[32834]: pgmap v171: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 592 op/s 2024-04-25T17:41:30.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:30 smithi173 ceph-mon[32856]: pgmap v171: 97 pgs: 97 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 592 op/s 2024-04-25T17:41:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 300.0 s, 3714.4 tps, lat 26.964 ms stddev 31.962 2024-04-25T17:41:32.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:32 smithi098 ceph-mon[29192]: pgmap v172: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 13 MiB/s wr, 706 op/s 2024-04-25T17:41:32.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:32 smithi203 ceph-mon[32834]: pgmap v172: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 13 MiB/s wr, 706 op/s 2024-04-25T17:41:32.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:32 smithi173 ceph-mon[32856]: pgmap v172: 97 pgs: 97 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 13 MiB/s wr, 706 op/s 2024-04-25T17:41:34.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:34 smithi098 ceph-mon[29192]: pgmap v173: 97 pgs: 97 active+clean; 2.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 31 MiB/s wr, 986 op/s 2024-04-25T17:41:34.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:34 smithi203 ceph-mon[32834]: pgmap v173: 97 pgs: 97 active+clean; 2.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 31 MiB/s wr, 986 op/s 2024-04-25T17:41:34.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:34 smithi173 ceph-mon[32856]: pgmap v173: 97 pgs: 97 active+clean; 2.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 31 MiB/s wr, 986 op/s 2024-04-25T17:41:36.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:36 smithi098 ceph-mon[29192]: pgmap v174: 97 pgs: 97 active+clean; 2.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 28 MiB/s wr, 733 op/s 2024-04-25T17:41:36.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:36 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:41:36.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:36 smithi203 ceph-mon[32834]: pgmap v174: 97 pgs: 97 active+clean; 2.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 28 MiB/s wr, 733 op/s 2024-04-25T17:41:36.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:36 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:41:36.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:36 smithi173 ceph-mon[32856]: pgmap v174: 97 pgs: 97 active+clean; 2.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 28 MiB/s wr, 733 op/s 2024-04-25T17:41:36.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:36 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:41:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 305.0 s, 3943.4 tps, lat 25.474 ms stddev 32.169 2024-04-25T17:41:38.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:38 smithi098 ceph-mon[29192]: pgmap v175: 97 pgs: 97 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 47 MiB/s wr, 1.07k op/s 2024-04-25T17:41:38.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:38 smithi203 ceph-mon[32834]: pgmap v175: 97 pgs: 97 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 47 MiB/s wr, 1.07k op/s 2024-04-25T17:41:38.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:38 smithi173 ceph-mon[32856]: pgmap v175: 97 pgs: 97 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 47 MiB/s wr, 1.07k op/s 2024-04-25T17:41:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:40 smithi098 ceph-mon[29192]: pgmap v176: 97 pgs: 97 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 43 MiB/s wr, 779 op/s 2024-04-25T17:41:40.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:40 smithi203 ceph-mon[32834]: pgmap v176: 97 pgs: 97 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 43 MiB/s wr, 779 op/s 2024-04-25T17:41:40.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:40 smithi173 ceph-mon[32856]: pgmap v176: 97 pgs: 97 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 43 MiB/s wr, 779 op/s 2024-04-25T17:41:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.833+0000 7fc809223640 1 -- 172.21.15.98:0/86551519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc804101aa0 msgr2=0x7fc804105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.833+0000 7fc809223640 1 --2- 172.21.15.98:0/86551519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc804101aa0 0x7fc804105a90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009900 tx=0x7fc7f802a410 comp rx=0 tx=0).stop 2024-04-25T17:41:40.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.833+0000 7fc809223640 1 -- 172.21.15.98:0/86551519 shutdown_connections 2024-04-25T17:41:40.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.833+0000 7fc809223640 1 --2- 172.21.15.98:0/86551519 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc804106170 0x7fc80410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.833+0000 7fc809223640 1 --2- 172.21.15.98:0/86551519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc804101aa0 0x7fc804105a90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.833+0000 7fc809223640 1 --2- 172.21.15.98:0/86551519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8041010f0 0x7fc8041014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.833+0000 7fc809223640 1 -- 172.21.15.98:0/86551519 >> 172.21.15.98:0/86551519 conn(0x7fc804077fd0 msgr2=0x7fc8040783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:40.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.834+0000 7fc809223640 1 -- 172.21.15.98:0/86551519 shutdown_connections 2024-04-25T17:41:40.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.834+0000 7fc809223640 1 -- 172.21.15.98:0/86551519 wait complete. 2024-04-25T17:41:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.835+0000 7fc809223640 1 Processor -- start 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.835+0000 7fc809223640 1 -- start start 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.835+0000 7fc809223640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8041010f0 0x7fc8041a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.835+0000 7fc809223640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc804106170 0x7fc8041a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.835+0000 7fc802d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8041010f0 0x7fc8041a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc809223640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80419a690 0x7fc80419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc802d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8041010f0 0x7fc8041a05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46804/0 (socket says 172.21.15.98:46804) 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc802d9d640 1 -- 172.21.15.98:0/3484158925 learned_addr learned my addr 172.21.15.98:0/3484158925 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc80410fdd0 con 0x7fc8041010f0 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc80410fc50 con 0x7fc80419a690 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc80410ff50 con 0x7fc804106170 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc80259c640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc804106170 0x7fc8041a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.836 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc80359e640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80419a690 0x7fc80419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc80359e640 1 -- 172.21.15.98:0/3484158925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc804106170 msgr2=0x7fc8041a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:40.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc80359e640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc804106170 0x7fc8041a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc80359e640 1 -- 172.21.15.98:0/3484158925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8041010f0 msgr2=0x7fc8041a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:40.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc80359e640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8041010f0 0x7fc8041a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.836+0000 7fc80359e640 1 -- 172.21.15.98:0/3484158925 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80419b290 con 0x7fc80419a690 2024-04-25T17:41:40.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.837+0000 7fc80359e640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80419a690 0x7fc80419ab10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc009fd0 tx=0x7fc7fc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:40.837 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.837+0000 7fc802d9d640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8041010f0 0x7fc8041a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:41:40.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.838+0000 7fc7ebfff640 1 -- 172.21.15.98:0/3484158925 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc00ce20 con 0x7fc80419a690 2024-04-25T17:41:40.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.838+0000 7fc7ebfff640 1 -- 172.21.15.98:0/3484158925 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7fc005020 con 0x7fc80419a690 2024-04-25T17:41:40.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.838+0000 7fc7ebfff640 1 -- 172.21.15.98:0/3484158925 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc010670 con 0x7fc80419a690 2024-04-25T17:41:40.838 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.838+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8041a7370 con 0x7fc80419a690 2024-04-25T17:41:40.839 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.838+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8041a7720 con 0x7fc80419a690 2024-04-25T17:41:40.840 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.840+0000 7fc7ebfff640 1 -- 172.21.15.98:0/3484158925 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc7fc010810 con 0x7fc80419a690 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.841+0000 7fc80c012740 1 Processor -- start 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.841+0000 7fc7ebfff640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc7ec0a3f20 0x7fc7ec0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.841+0000 7fc7ebfff640 1 -- 172.21.15.98:0/3484158925 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fc7fc0d0c20 con 0x7fc80419a690 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80259c640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc7ec0a3f20 0x7fc7ec0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80c012740 1 -- start start 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80c012740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 0x55851a20a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80c012740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a214ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80c012740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a217400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80c012740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55851a166970 con 0x55851a215010 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80c012740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55851a1034a0 con 0x55851a209e30 2024-04-25T17:41:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80c012740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55851a162930 con 0x55851a20a750 2024-04-25T17:41:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc802d9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 0x55851a20a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a217400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a217400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46812/0 (socket says 172.21.15.98:46812) 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.842+0000 7fc80359e640 1 -- 172.21.15.98:0/2140341656 learned_addr learned my addr 172.21.15.98:0/2140341656 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.843+0000 7fc80259c640 1 --2- 172.21.15.98:0/2140341656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a214ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.843+0000 7fc80259c640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc7ec0a3f20 0x7fc7ec0a63e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8006010 tx=0x7fc7f8005d50 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.843+0000 7fc802d9d640 1 -- 172.21.15.98:0/2140341656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 msgr2=0x55851a214ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.843+0000 7fc802d9d640 1 --2- 172.21.15.98:0/2140341656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a214ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.843+0000 7fc802d9d640 1 -- 172.21.15.98:0/2140341656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 msgr2=0x55851a217400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.843+0000 7fc802d9d640 1 --2- 172.21.15.98:0/2140341656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a217400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.843+0000 7fc802d9d640 1 -- 172.21.15.98:0/2140341656 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558519e79210 con 0x55851a209e30 2024-04-25T17:41:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.843+0000 7fc80359e640 1 --2- 172.21.15.98:0/2140341656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a217400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:41:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.843+0000 7fc802d9d640 1 --2- 172.21.15.98:0/2140341656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 0x55851a20a210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4010480 tx=0x7fc7f4010940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fa4fe7113e6036c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.844+0000 7fc7e97fa640 1 -- 172.21.15.98:0/2140341656 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4011820 con 0x55851a209e30 2024-04-25T17:41:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.844+0000 7fc7e97fa640 1 -- 172.21.15.98:0/2140341656 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f401c480 con 0x55851a209e30 2024-04-25T17:41:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.844+0000 7fc7e97fa640 1 -- 172.21.15.98:0/2140341656 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f400e690 con 0x55851a209e30 2024-04-25T17:41:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.845+0000 7fc80c012740 1 -- 172.21.15.98:0/2140341656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 msgr2=0x55851a20a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.845+0000 7fc80c012740 1 --2- 172.21.15.98:0/2140341656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 0x55851a20a210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4010480 tx=0x7fc7f4010940 comp rx=0 tx=0).stop 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.845+0000 7fc80c012740 1 -- 172.21.15.98:0/2140341656 shutdown_connections 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.845+0000 7fc80c012740 1 --2- 172.21.15.98:0/2140341656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a217400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.845+0000 7fc80c012740 1 --2- 172.21.15.98:0/2140341656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a214ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.845+0000 7fc80c012740 1 --2- 172.21.15.98:0/2140341656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 0x55851a20a210 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.845+0000 7fc80c012740 1 -- 172.21.15.98:0/2140341656 >> 172.21.15.98:0/2140341656 conn(0x55851a206340 msgr2=0x55851a206720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.845+0000 7fc80c012740 1 -- 172.21.15.98:0/2140341656 shutdown_connections 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.845+0000 7fc80c012740 1 -- 172.21.15.98:0/2140341656 wait complete. 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.846+0000 7fc80c012740 1 -- start start 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.846+0000 7fc80c012740 1 Processor -- start 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.846+0000 7fc80c012740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a209e30 0x55851a214900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.846+0000 7fc80c012740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a211bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.846+0000 7fc80c012740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a215010 0x55851a212130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.846+0000 7fc80c012740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55851a217c40 con 0x55851a209e30 2024-04-25T17:41:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.846+0000 7fc80c012740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55851a1010b0 con 0x55851a215010 2024-04-25T17:41:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.846+0000 7fc80c012740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55851a217dc0 con 0x55851a20a750 2024-04-25T17:41:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.846+0000 7fc80c012740 10 client.0 _ll_register_callbacks cb 0x55851a0eed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80359e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a215010 0x55851a212130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a211bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80359e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a215010 0x55851a212130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48998/0 (socket says 172.21.15.98:48998) 2024-04-25T17:41:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a211bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:37214/0 (socket says 172.21.15.98:37214) 2024-04-25T17:41:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80359e640 1 -- 172.21.15.98:0/2170797727 learned_addr learned my addr 172.21.15.98:0/2170797727 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc802d9d640 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a209e30 0x55851a214900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80259c640 1 -- 172.21.15.98:0/2170797727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a215010 msgr2=0x55851a212130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80259c640 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a215010 0x55851a212130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80259c640 1 -- 172.21.15.98:0/2170797727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a209e30 msgr2=0x55851a214900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80259c640 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a209e30 0x55851a214900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80259c640 1 -- 172.21.15.98:0/2170797727 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558519f45620 con 0x55851a20a750 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc80259c640 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a211bf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8005ab0 tx=0x7fc7f802a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9417ee7c8df708f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.847+0000 7fc7e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.848+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/2170797727 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8035020 con 0x55851a20a750 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.848+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/2170797727 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fc7f8004db0 con 0x55851a20a750 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.848+0000 7fc80c012740 1 -- 172.21.15.98:0/2170797727 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f401c480 con 0x55851a20a750 2024-04-25T17:41:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.848+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/2170797727 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc7f8004030 con 0x55851a20a750 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.848+0000 7fc80c012740 10 client.25456 fetch_fsmap learned FSMap version 34 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.848+0000 7fc80c012740 1 -- 172.21.15.98:0/2170797727 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55851a223b50 con 0x55851a20a750 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.848+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/2170797727 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f401c480 con 0x55851a20a750 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.848+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/2170797727 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 34) ==== 8338+0+0 (secure 0 0 0) 0x7fc7f803a040 con 0x55851a20a750 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.848+0000 7fc80c012740 10 client.25456 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.848+0000 7fc80c012740 10 client.25456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.849+0000 7fc80c012740 20 client.25456 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.849+0000 7fc80c012740 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55851a225080 0x55851a2274d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.849+0000 7fc80c012740 4 client.25456 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.849+0000 7fc80c012740 1 -- 172.21.15.98:0/2170797727 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55851a174390 con 0x55851a225080 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.849+0000 7fc802d9d640 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55851a225080 0x55851a2274d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.849+0000 7fc7e6ff5640 10 client.25456 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.849+0000 7fc802d9d640 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55851a225080 0x55851a2274d0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29376e15c75b804a server_cookie=93b3ffd7cdf5df7d in_seq=0 out_seq=0 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.850+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/2170797727 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55851a174390 con 0x55851a225080 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.850+0000 7fc7e6ff5640 10 client.25456 handle_command_reply: tid=0 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc80c012740 1 client.25456 shutdown 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc80c012740 1 -- 172.21.15.98:0/2170797727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 msgr2=0x55851a211bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc80c012740 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a211bf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8005ab0 tx=0x7fc7f802a410 comp rx=0 tx=0).stop 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc80c012740 1 -- 172.21.15.98:0/2170797727 shutdown_connections 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc80c012740 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55851a225080 0x55851a2274d0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc80c012740 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a215010 0x55851a212130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc80c012740 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a211bf0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc7e6ff5640 0 client.25456 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc80c012740 1 --2- 172.21.15.98:0/2170797727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a209e30 0x55851a214900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc80c012740 1 -- 172.21.15.98:0/2170797727 >> 172.21.15.98:0/2170797727 conn(0x55851a206340 msgr2=0x55851a1fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.851+0000 7fc80c012740 1 -- 172.21.15.98:0/2170797727 shutdown_connections 2024-04-25T17:41:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.852+0000 7fc80c012740 1 -- 172.21.15.98:0/2170797727 wait complete. 2024-04-25T17:41:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.852+0000 7fc80c012740 20 client.25456 trim_cache size 0 max 16384 2024-04-25T17:41:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.901+0000 7fc80c012740 1 Processor -- start 2024-04-25T17:41:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.903+0000 7fc80c012740 1 -- start start 2024-04-25T17:41:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.903+0000 7fc80c012740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a209e30 0x55851a220b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.903+0000 7fc80c012740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a20a750 0x55851a22ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.903+0000 7fc80c012740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a22d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.903+0000 7fc802d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a209e30 0x55851a220b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.903+0000 7fc802d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a209e30 0x55851a220b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:37226/0 (socket says 172.21.15.98:37226) 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.903+0000 7fc80259c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a20a750 0x55851a22ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.903+0000 7fc80259c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a20a750 0x55851a22ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49012/0 (socket says 172.21.15.98:49012) 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.903+0000 7fc802d9d640 1 -- 172.21.15.98:0/2998851069 learned_addr learned my addr 172.21.15.98:0/2998851069 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.904+0000 7fc80c012740 1 -- 172.21.15.98:0/2998851069 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f0010f90 con 0x55851a215010 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.904+0000 7fc80c012740 1 -- 172.21.15.98:0/2998851069 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f000e860 con 0x55851a20a750 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.904+0000 7fc80c012740 1 -- 172.21.15.98:0/2998851069 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f000c6b0 con 0x55851a209e30 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.904+0000 7fc80359e640 1 --2- 172.21.15.98:0/2998851069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a22d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.904+0000 7fc802d9d640 1 -- 172.21.15.98:0/2998851069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a20a750 msgr2=0x55851a22ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.904+0000 7fc802d9d640 1 --2- 172.21.15.98:0/2998851069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a20a750 0x55851a22ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.904+0000 7fc802d9d640 1 -- 172.21.15.98:0/2998851069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 msgr2=0x55851a22d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.904+0000 7fc802d9d640 1 --2- 172.21.15.98:0/2998851069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a22d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.904+0000 7fc802d9d640 1 -- 172.21.15.98:0/2998851069 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55851a20d7f0 con 0x55851a209e30 2024-04-25T17:41:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.904+0000 7fc802d9d640 1 --2- 172.21.15.98:0/2998851069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a209e30 0x55851a220b00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4010450 tx=0x7fc7f401ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa9a7fad800283d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.905+0000 7fc7e8ff9640 1 -- 172.21.15.98:0/2998851069 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f401c8b0 con 0x55851a209e30 2024-04-25T17:41:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.905+0000 7fc7e8ff9640 1 -- 172.21.15.98:0/2998851069 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f40054e0 con 0x55851a209e30 2024-04-25T17:41:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.905+0000 7fc7e8ff9640 1 -- 172.21.15.98:0/2998851069 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f40049e0 con 0x55851a209e30 2024-04-25T17:41:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.906+0000 7fc80c012740 1 -- 172.21.15.98:0/2998851069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a209e30 msgr2=0x55851a220b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.906+0000 7fc80c012740 1 --2- 172.21.15.98:0/2998851069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a209e30 0x55851a220b00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4010450 tx=0x7fc7f401ceb0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.906+0000 7fc80c012740 1 -- 172.21.15.98:0/2998851069 shutdown_connections 2024-04-25T17:41:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.906+0000 7fc80c012740 1 --2- 172.21.15.98:0/2998851069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a22d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.906+0000 7fc80c012740 1 --2- 172.21.15.98:0/2998851069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a20a750 0x55851a22ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.906+0000 7fc80c012740 1 --2- 172.21.15.98:0/2998851069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a209e30 0x55851a220b00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.906+0000 7fc80c012740 1 -- 172.21.15.98:0/2998851069 >> 172.21.15.98:0/2998851069 conn(0x55851a206340 msgr2=0x55851a1fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.906+0000 7fc80c012740 1 -- 172.21.15.98:0/2998851069 shutdown_connections 2024-04-25T17:41:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.906+0000 7fc80c012740 1 -- 172.21.15.98:0/2998851069 wait complete. 2024-04-25T17:41:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.907+0000 7fc80c012740 1 -- start start 2024-04-25T17:41:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.907+0000 7fc80c012740 1 Processor -- start 2024-04-25T17:41:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc80c012740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 0x55851a21bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc80c012740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a21c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc80c012740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a21c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc80c012740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f000cd20 con 0x55851a215010 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc80c012740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f000e190 con 0x55851a209e30 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc80c012740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f000d3f0 con 0x55851a20a750 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc80c012740 10 client.0 _ll_register_callbacks cb 0x7fc7f8004760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc80359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a21c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc80359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a21c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46838/0 (socket says 172.21.15.98:46838) 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc80359e640 1 -- 172.21.15.98:0/1445327129 learned_addr learned my addr 172.21.15.98:0/1445327129 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.908+0000 7fc802d9d640 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 0x55851a21bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.909+0000 7fc80259c640 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a21c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.909+0000 7fc80259c640 1 -- 172.21.15.98:0/1445327129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 msgr2=0x55851a21bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.909+0000 7fc80259c640 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 0x55851a21bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.909+0000 7fc80259c640 1 -- 172.21.15.98:0/1445327129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 msgr2=0x55851a21c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.909+0000 7fc80259c640 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a21c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.909+0000 7fc80259c640 1 -- 172.21.15.98:0/1445327129 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55851a2046f0 con 0x55851a20a750 2024-04-25T17:41:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.909+0000 7fc80259c640 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a21c140 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80035f0 tx=0x7fc7f80092b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f265bdac2181052 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.909+0000 7fc7e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:41:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/1445327129 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f805f070 con 0x55851a20a750 2024-04-25T17:41:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/1445327129 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fc7f80038e0 con 0x55851a20a750 2024-04-25T17:41:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc80c012740 1 -- 172.21.15.98:0/1445327129 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7f40054e0 con 0x55851a20a750 2024-04-25T17:41:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/1445327129 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc7f8069050 con 0x55851a20a750 2024-04-25T17:41:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc80c012740 10 client.25462 fetch_fsmap learned FSMap version 34 2024-04-25T17:41:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc80c012740 1 -- 172.21.15.98:0/1445327129 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55851a205a00 con 0x55851a20a750 2024-04-25T17:41:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/1445327129 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7f80316d0 con 0x55851a20a750 2024-04-25T17:41:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/1445327129 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 34) ==== 8338+0+0 (secure 0 0 0) 0x7fc7f8052470 con 0x55851a20a750 2024-04-25T17:41:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc80c012740 10 client.25462 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T17:41:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc80c012740 10 client.25462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:41:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.910+0000 7fc80c012740 20 client.25462 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:40.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.911+0000 7fc80c012740 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55851a294a30 0x55851a296e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:40.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.911+0000 7fc80c012740 4 client.25462 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:41:40.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.911+0000 7fc80c012740 1 -- 172.21.15.98:0/1445327129 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fc7f40054e0 con 0x55851a294a30 2024-04-25T17:41:40.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.911+0000 7fc802d9d640 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55851a294a30 0x55851a296e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:40.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.911+0000 7fc7e6ff5640 10 client.25462 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:40.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.911+0000 7fc802d9d640 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55851a294a30 0x55851a296e50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b07385b81ef8783 server_cookie=8fecdc07cf14606 in_seq=0 out_seq=0 2024-04-25T17:41:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.930+0000 7fc7e6ff5640 1 -- 172.21.15.98:0/1445327129 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fc7f40054e0 con 0x55851a294a30 2024-04-25T17:41:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.930+0000 7fc7e6ff5640 10 client.25462 handle_command_reply: tid=0 2024-04-25T17:41:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.930+0000 7fc80c012740 1 client.25462 shutdown 2024-04-25T17:41:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.930+0000 7fc80c012740 1 -- 172.21.15.98:0/1445327129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 msgr2=0x55851a21c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:40.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.930+0000 7fc80c012740 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a21c140 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80035f0 tx=0x7fc7f80092b0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.931+0000 7fc80c012740 1 -- 172.21.15.98:0/1445327129 shutdown_connections 2024-04-25T17:41:40.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.931+0000 7fc80c012740 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55851a294a30 0x55851a296e50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.931+0000 7fc7e6ff5640 0 client.25462 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.931+0000 7fc80c012740 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55851a215010 0x55851a21c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.931+0000 7fc80c012740 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55851a20a750 0x55851a21c140 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.931+0000 7fc80c012740 1 --2- 172.21.15.98:0/1445327129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55851a209e30 0x55851a21bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.931+0000 7fc80c012740 1 -- 172.21.15.98:0/1445327129 >> 172.21.15.98:0/1445327129 conn(0x55851a206340 msgr2=0x55851a213420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.931+0000 7fc80c012740 1 -- 172.21.15.98:0/1445327129 shutdown_connections 2024-04-25T17:41:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.931+0000 7fc80c012740 1 -- 172.21.15.98:0/1445327129 wait complete. 2024-04-25T17:41:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.931+0000 7fc80c012740 20 client.25462 trim_cache size 0 max 16384 2024-04-25T17:41:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:41:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:41:40.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:41:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:41:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc7ec0a3f20 msgr2=0x7fc7ec0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc7ec0a3f20 0x7fc7ec0a63e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8006010 tx=0x7fc7f8005d50 comp rx=0 tx=0).stop 2024-04-25T17:41:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80419a690 msgr2=0x7fc80419ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80419a690 0x7fc80419ab10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc009fd0 tx=0x7fc7fc00ef10 comp rx=0 tx=0).stop 2024-04-25T17:41:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 shutdown_connections 2024-04-25T17:41:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc7ec0a3f20 0x7fc7ec0a63e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc80419a690 0x7fc80419ab10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc804106170 0x7fc8041a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 --2- 172.21.15.98:0/3484158925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8041010f0 0x7fc8041a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 >> 172.21.15.98:0/3484158925 conn(0x7fc804077fd0 msgr2=0x7fc8040fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 shutdown_connections 2024-04-25T17:41:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:40.932+0000 7fc809223640 1 -- 172.21.15.98:0/3484158925 wait complete. 2024-04-25T17:41:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 310.0 s, 4214.8 tps, lat 23.654 ms stddev 28.024 2024-04-25T17:41:42.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:42 smithi098 ceph-mon[29192]: pgmap v177: 97 pgs: 97 active+clean; 2.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 50 MiB/s wr, 891 op/s 2024-04-25T17:41:42.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:42 smithi203 ceph-mon[32834]: pgmap v177: 97 pgs: 97 active+clean; 2.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 50 MiB/s wr, 891 op/s 2024-04-25T17:41:42.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:42 smithi173 ceph-mon[32856]: pgmap v177: 97 pgs: 97 active+clean; 2.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 50 MiB/s wr, 891 op/s 2024-04-25T17:41:44.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:44 smithi203 ceph-mon[32834]: pgmap v178: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 54 MiB/s wr, 1.02k op/s 2024-04-25T17:41:44.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:44 smithi173 ceph-mon[32856]: pgmap v178: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 54 MiB/s wr, 1.02k op/s 2024-04-25T17:41:44.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:44 smithi098 ceph-mon[29192]: pgmap v178: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 54 MiB/s wr, 1.02k op/s 2024-04-25T17:41:46.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:46 smithi203 ceph-mon[32834]: pgmap v179: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 35 MiB/s wr, 694 op/s 2024-04-25T17:41:46.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:46 smithi173 ceph-mon[32856]: pgmap v179: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 35 MiB/s wr, 694 op/s 2024-04-25T17:41:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 315.0 s, 3180.6 tps, lat 31.468 ms stddev 55.265 2024-04-25T17:41:46.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:46 smithi098 ceph-mon[29192]: pgmap v179: 97 pgs: 97 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 35 MiB/s wr, 694 op/s 2024-04-25T17:41:48.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:48 smithi098 ceph-mon[29192]: pgmap v180: 97 pgs: 97 active+clean; 3.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 73 MiB/s wr, 2.01k op/s 2024-04-25T17:41:48.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:48 smithi173 ceph-mon[32856]: pgmap v180: 97 pgs: 97 active+clean; 3.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 73 MiB/s wr, 2.01k op/s 2024-04-25T17:41:48.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:48 smithi203 ceph-mon[32834]: pgmap v180: 97 pgs: 97 active+clean; 3.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 73 MiB/s wr, 2.01k op/s 2024-04-25T17:41:50.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:50 smithi173 ceph-mon[32856]: pgmap v181: 97 pgs: 97 active+clean; 3.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 54 MiB/s wr, 1.67k op/s 2024-04-25T17:41:50.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:50 smithi098 ceph-mon[29192]: pgmap v181: 97 pgs: 97 active+clean; 3.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 54 MiB/s wr, 1.67k op/s 2024-04-25T17:41:50.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:50 smithi203 ceph-mon[32834]: pgmap v181: 97 pgs: 97 active+clean; 3.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 54 MiB/s wr, 1.67k op/s 2024-04-25T17:41:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 320.0 s, 4128.6 tps, lat 24.194 ms stddev 26.615 2024-04-25T17:41:52.514 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:52 smithi203 ceph-mon[32834]: pgmap v182: 97 pgs: 97 active+clean; 3.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 55 MiB/s wr, 1.74k op/s 2024-04-25T17:41:52.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:52 smithi173 ceph-mon[32856]: pgmap v182: 97 pgs: 97 active+clean; 3.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 55 MiB/s wr, 1.74k op/s 2024-04-25T17:41:52.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:52 smithi098 ceph-mon[29192]: pgmap v182: 97 pgs: 97 active+clean; 3.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 55 MiB/s wr, 1.74k op/s 2024-04-25T17:41:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.057+0000 7fdc00820640 1 -- 172.21.15.98:0/103031325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc106f30 msgr2=0x7fdbfc107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.057+0000 7fdc00820640 1 --2- 172.21.15.98:0/103031325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc106f30 0x7fdbfc107310 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009930 tx=0x7fdbec02a420 comp rx=0 tx=0).stop 2024-04-25T17:41:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.057+0000 7fdc00820640 1 -- 172.21.15.98:0/103031325 shutdown_connections 2024-04-25T17:41:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.057+0000 7fdc00820640 1 --2- 172.21.15.98:0/103031325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbfc100e10 0x7fdbfc10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.057+0000 7fdc00820640 1 --2- 172.21.15.98:0/103031325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbfc100450 0x7fdbfc1008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.057+0000 7fdc00820640 1 --2- 172.21.15.98:0/103031325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc106f30 0x7fdbfc107310 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.057+0000 7fdc00820640 1 -- 172.21.15.98:0/103031325 >> 172.21.15.98:0/103031325 conn(0x7fdbfc0773e0 msgr2=0x7fdbfc0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.057+0000 7fdc00820640 1 -- 172.21.15.98:0/103031325 shutdown_connections 2024-04-25T17:41:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.057+0000 7fdc00820640 1 -- 172.21.15.98:0/103031325 wait complete. 2024-04-25T17:41:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdc00820640 1 Processor -- start 2024-04-25T17:41:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdc00820640 1 -- start start 2024-04-25T17:41:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdc00820640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbfc100450 0x7fdbfc196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdc00820640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbfc100e10 0x7fdbfc196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdc00820640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc106f30 0x7fdbfc19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdc00820640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc110130 con 0x7fdbfc106f30 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdc00820640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc10ffb0 con 0x7fdbfc100e10 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdc00820640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1102b0 con 0x7fdbfc100450 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdbfad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc106f30 0x7fdbfc19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdbfad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc106f30 0x7fdbfc19ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42792/0 (socket says 172.21.15.98:42792) 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdbfa59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbfc100450 0x7fdbfc196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.058+0000 7fdbfad9d640 1 -- 172.21.15.98:0/1599401958 learned_addr learned my addr 172.21.15.98:0/1599401958 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbf9d9b640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbfc100e10 0x7fdbfc196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbfad9d640 1 -- 172.21.15.98:0/1599401958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbfc100450 msgr2=0x7fdbfc196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbfad9d640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbfc100450 0x7fdbfc196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbfad9d640 1 -- 172.21.15.98:0/1599401958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbfc100e10 msgr2=0x7fdbfc196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbfad9d640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbfc100e10 0x7fdbfc196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbfad9d640 1 -- 172.21.15.98:0/1599401958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbfc19b540 con 0x7fdbfc106f30 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbfc100450 0x7fdbfc196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:41:54.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbfad9d640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc106f30 0x7fdbfc19ae10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400efd0 tx=0x7fdbf400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbeb7fe640 1 -- 172.21.15.98:0/1599401958 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4003b70 con 0x7fdbfc106f30 2024-04-25T17:41:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdc00820640 1 -- 172.21.15.98:0/1599401958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbfc19b830 con 0x7fdbfc106f30 2024-04-25T17:41:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbeb7fe640 1 -- 172.21.15.98:0/1599401958 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbf4003d10 con 0x7fdbfc106f30 2024-04-25T17:41:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdc00820640 1 -- 172.21.15.98:0/1599401958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbfc1a30c0 con 0x7fdbfc106f30 2024-04-25T17:41:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.059+0000 7fdbeb7fe640 1 -- 172.21.15.98:0/1599401958 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40108a0 con 0x7fdbfc106f30 2024-04-25T17:41:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.063+0000 7fdc03610740 1 Processor -- start 2024-04-25T17:41:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.063+0000 7fdc03610740 1 -- start start 2024-04-25T17:41:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.063+0000 7fdc03610740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613ceed1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.063+0000 7fdc03610740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1e90 0x5613ceee4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.063+0000 7fdc03610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceee5020 0x5613ceee7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.063+0000 7fdc03610740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5613cee36970 con 0x5613ceed1e90 2024-04-25T17:41:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.063+0000 7fdc03610740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5613cedd34a0 con 0x5613ceed1570 2024-04-25T17:41:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.063+0000 7fdc03610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5613cee32930 con 0x5613ceee5020 2024-04-25T17:41:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.063+0000 7fdbfa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1e90 0x5613ceee4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.063+0000 7fdbf9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceee5020 0x5613ceee7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.064+0000 7fdbfa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1e90 0x5613ceee4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42802/0 (socket says 172.21.15.98:42802) 2024-04-25T17:41:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.064+0000 7fdbfa59c640 1 -- 172.21.15.98:0/1199912941 learned_addr learned my addr 172.21.15.98:0/1199912941 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.064+0000 7fdbf9d9b640 1 --2- 172.21.15.98:0/1199912941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613ceed1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.064+0000 7fdbfa59c640 1 -- 172.21.15.98:0/1199912941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceee5020 msgr2=0x5613ceee7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.064+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/1199912941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceee5020 0x5613ceee7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.064+0000 7fdbfa59c640 1 -- 172.21.15.98:0/1199912941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 msgr2=0x5613ceed1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.064+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/1199912941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613ceed1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.064+0000 7fdbfa59c640 1 -- 172.21.15.98:0/1199912941 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613ceb49210 con 0x5613ceed1e90 2024-04-25T17:41:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.065+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/1199912941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1e90 0x5613ceee4ae0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009930 tx=0x7fdbec004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ac3474849f1bfb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.065+0000 7fdbeb7fe640 1 -- 172.21.15.98:0/1599401958 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdbf4004020 con 0x7fdbfc106f30 2024-04-25T17:41:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.066+0000 7fdbeb7fe640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbe40a3ea0 0x7fdbe40a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.066+0000 7fdbe8ff9640 1 -- 172.21.15.98:0/1199912941 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec035040 con 0x5613ceed1e90 2024-04-25T17:41:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.066+0000 7fdbf9d9b640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbe40a3ea0 0x7fdbe40a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.066+0000 7fdbeb7fe640 1 -- 172.21.15.98:0/1599401958 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fdbf40d0b60 con 0x7fdbfc106f30 2024-04-25T17:41:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.066+0000 7fdbe8ff9640 1 -- 172.21.15.98:0/1199912941 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbec004760 con 0x5613ceed1e90 2024-04-25T17:41:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.066+0000 7fdbe8ff9640 1 -- 172.21.15.98:0/1199912941 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec003a20 con 0x5613ceed1e90 2024-04-25T17:41:54.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.067+0000 7fdc03610740 1 -- 172.21.15.98:0/1199912941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1e90 msgr2=0x5613ceee4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:54.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.067+0000 7fdc03610740 1 --2- 172.21.15.98:0/1199912941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1e90 0x5613ceee4ae0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009930 tx=0x7fdbec004530 comp rx=0 tx=0).stop 2024-04-25T17:41:54.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.067+0000 7fdbf9d9b640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbe40a3ea0 0x7fdbe40a6360 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5613ceececd0 tx=0x7fdbf000b390 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:54.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.068+0000 7fdc03610740 1 -- 172.21.15.98:0/1199912941 shutdown_connections 2024-04-25T17:41:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.068+0000 7fdc03610740 1 --2- 172.21.15.98:0/1199912941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceee5020 0x5613ceee7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.068+0000 7fdc03610740 1 --2- 172.21.15.98:0/1199912941 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1e90 0x5613ceee4ae0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.068+0000 7fdc03610740 1 --2- 172.21.15.98:0/1199912941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613ceed1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.068+0000 7fdc03610740 1 -- 172.21.15.98:0/1199912941 >> 172.21.15.98:0/1199912941 conn(0x5613ceecdb40 msgr2=0x5613ceecdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.068+0000 7fdc03610740 1 -- 172.21.15.98:0/1199912941 shutdown_connections 2024-04-25T17:41:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.068+0000 7fdc03610740 1 -- 172.21.15.98:0/1199912941 wait complete. 2024-04-25T17:41:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdc03610740 1 -- start start 2024-04-25T17:41:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdc03610740 1 Processor -- start 2024-04-25T17:41:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdc03610740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1570 0x5613ceede7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdc03610740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1e90 0x5613ceedecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdbfa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1570 0x5613ceede7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdbfa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1570 0x5613ceede7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42816/0 (socket says 172.21.15.98:42816) 2024-04-25T17:41:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdbfa59c640 1 -- 172.21.15.98:0/265168993 learned_addr learned my addr 172.21.15.98:0/265168993 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdc03610740 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceee5020 0x5613ceee44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5613ceed44b0 con 0x5613ceed1570 2024-04-25T17:41:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5613cedd10b0 con 0x5613ceed1e90 2024-04-25T17:41:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.069+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5613ceed4630 con 0x5613ceee5020 2024-04-25T17:41:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbf9d9b640 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1e90 0x5613ceedecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbfad9d640 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceee5020 0x5613ceee44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdc03610740 10 client.0 _ll_register_callbacks cb 0x5613cedbed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbf9d9b640 1 -- 172.21.15.98:0/265168993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceee5020 msgr2=0x5613ceee44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbf9d9b640 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceee5020 0x5613ceee44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbf9d9b640 1 -- 172.21.15.98:0/265168993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1570 msgr2=0x5613ceede7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbf9d9b640 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1570 0x5613ceede7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbf9d9b640 1 -- 172.21.15.98:0/265168993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613cec15620 con 0x5613ceed1e90 2024-04-25T17:41:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbec004760 con 0x5613ceed1e90 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbf9d9b640 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1e90 0x5613ceedecf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0005d00 tx=0x7fdbf00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45752d648a1ada65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbe1ffb640 10 client.25490 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/265168993 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf001e870 con 0x5613ceed1e90 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.070+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/265168993 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fdbf001ea10 con 0x5613ceed1e90 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.071+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/265168993 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fdbf001ec30 con 0x5613ceed1e90 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.071+0000 7fdc03610740 10 client.25490 fetch_fsmap learned FSMap version 34 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.071+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=34}) -- 0x5613ceefc140 con 0x5613ceed1e90 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.071+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/265168993 <== mon.2 v2:172.21.15.173:3300/0 4 ==== fsmap(e 34) ==== 8338+0+0 (secure 0 0 0) 0x7fdbf0027070 con 0x5613ceed1e90 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.072+0000 7fdc03610740 10 client.25490 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.072+0000 7fdc03610740 10 client.25490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.072+0000 7fdc03610740 20 client.25490 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.072+0000 7fdc03610740 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5613ceefda00 0x5613ceeffe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.072+0000 7fdc03610740 4 client.25490 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:41:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.072+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613cee44390 con 0x5613ceefda00 2024-04-25T17:41:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.072+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5613ceefda00 0x5613ceeffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.073+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5613ceefda00 0x5613ceeffe90 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c0a45dedcb9ade0 server_cookie=dfd17590e1b9aba in_seq=0 out_seq=0 2024-04-25T17:41:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.073+0000 7fdbe1ffb640 10 client.25490 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.073+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/265168993 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613cee44390 con 0x5613ceefda00 2024-04-25T17:41:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.073+0000 7fdbe1ffb640 10 client.25490 handle_command_reply: tid=0 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.073+0000 7fdc03610740 1 client.25490 shutdown 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.073+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1e90 msgr2=0x5613ceedecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.073+0000 7fdc03610740 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1e90 0x5613ceedecf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0005d00 tx=0x7fdbf00046c0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.074+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 shutdown_connections 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.074+0000 7fdc03610740 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5613ceefda00 0x5613ceeffe90 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.074+0000 7fdc03610740 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceee5020 0x5613ceee44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.074+0000 7fdc03610740 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1e90 0x5613ceedecf0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.074+0000 7fdc03610740 1 --2- 172.21.15.98:0/265168993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceed1570 0x5613ceede7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.074+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 >> 172.21.15.98:0/265168993 conn(0x5613ceecdb40 msgr2=0x5613ceed38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.074+0000 7fdbe1ffb640 0 client.25490 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.074+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 shutdown_connections 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.074+0000 7fdc03610740 1 -- 172.21.15.98:0/265168993 wait complete. 2024-04-25T17:41:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.074+0000 7fdc03610740 20 client.25490 trim_cache size 0 max 16384 2024-04-25T17:41:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.099+0000 7fdc03610740 1 Processor -- start 2024-04-25T17:41:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdc03610740 1 -- start start 2024-04-25T17:41:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdc03610740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613ceee4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdc03610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 0x5613cef05770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdc03610740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 0x5613cef05cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdc03610740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1ae110 con 0x5613ceee5020 2024-04-25T17:41:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdc03610740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1ab9e0 con 0x5613ceed1570 2024-04-25T17:41:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdc03610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1a9830 con 0x5613ceed1e90 2024-04-25T17:41:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdbfad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 0x5613cef05cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdbf9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 0x5613cef05770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdbf9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 0x5613cef05770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:55958/0 (socket says 172.21.15.98:55958) 2024-04-25T17:41:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdbfad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 0x5613cef05cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42828/0 (socket says 172.21.15.98:42828) 2024-04-25T17:41:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdbf9d9b640 1 -- 172.21.15.98:0/3369462666 learned_addr learned my addr 172.21.15.98:0/3369462666 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.100+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/3369462666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613ceee4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.101+0000 7fdbfa59c640 1 -- 172.21.15.98:0/3369462666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 msgr2=0x5613cef05770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.101+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/3369462666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 0x5613cef05770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.101+0000 7fdbfa59c640 1 -- 172.21.15.98:0/3369462666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 msgr2=0x5613cef05cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.101+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/3369462666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 0x5613cef05cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.101+0000 7fdbfa59c640 1 -- 172.21.15.98:0/3369462666 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613ceed4620 con 0x5613ceed1570 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.101+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/3369462666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613ceee4320 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x5613cef00710 tx=0x7fdbec005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82e7a94962999beb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.101+0000 7fdbe3fff640 1 -- 172.21.15.98:0/3369462666 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec035040 con 0x5613ceed1570 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.101+0000 7fdbe3fff640 1 -- 172.21.15.98:0/3369462666 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbec005480 con 0x5613ceed1570 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.101+0000 7fdbe3fff640 1 -- 172.21.15.98:0/3369462666 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec054640 con 0x5613ceed1570 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 -- 172.21.15.98:0/3369462666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 msgr2=0x5613ceee4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 --2- 172.21.15.98:0/3369462666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613ceee4320 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x5613cef00710 tx=0x7fdbec005e70 comp rx=0 tx=0).stop 2024-04-25T17:41:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 -- 172.21.15.98:0/3369462666 shutdown_connections 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 --2- 172.21.15.98:0/3369462666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 0x5613cef05cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 --2- 172.21.15.98:0/3369462666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 0x5613cef05770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 --2- 172.21.15.98:0/3369462666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613ceee4320 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 -- 172.21.15.98:0/3369462666 >> 172.21.15.98:0/3369462666 conn(0x5613ceecdb40 msgr2=0x5613ceed5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 -- 172.21.15.98:0/3369462666 shutdown_connections 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 -- 172.21.15.98:0/3369462666 wait complete. 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 -- start start 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.102+0000 7fdc03610740 1 Processor -- start 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdc03610740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613cef10f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdc03610740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 0x5613ceed63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdc03610740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 0x5613ceed68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdc03610740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1a9ea0 con 0x5613ceee5020 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdc03610740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1ab310 con 0x5613ceed1570 2024-04-25T17:41:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdc03610740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1aa570 con 0x5613ceed1e90 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdc03610740 10 client.0 _ll_register_callbacks cb 0x7fdbf0003c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbf9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 0x5613ceed63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbfa59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613cef10f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbf9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 0x5613ceed63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:55974/0 (socket says 172.21.15.98:55974) 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbf9d9b640 1 -- 172.21.15.98:0/3285566089 learned_addr learned my addr 172.21.15.98:0/3285566089 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbfad9d640 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 0x5613ceed68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbfa59c640 1 -- 172.21.15.98:0/3285566089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 msgr2=0x5613ceed63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 0x5613ceed63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbfa59c640 1 -- 172.21.15.98:0/3285566089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 msgr2=0x5613ceed68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 0x5613ceed68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbfa59c640 1 -- 172.21.15.98:0/3285566089 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613ceed6f20 con 0x5613ceed1570 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbfa59c640 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613cef10f50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdbec02abd0 tx=0x7fdbec0545f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d9c18642f9c43a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:41:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.103+0000 7fdbe1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.104+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/3285566089 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec035030 con 0x5613ceed1570 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.104+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/3285566089 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8469+0+0 (secure 0 0 0) 0x7fdbec004910 con 0x5613ceed1570 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.104+0000 7fdc03610740 1 -- 172.21.15.98:0/3285566089 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbec005480 con 0x5613ceed1570 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.105+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/3285566089 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fdbec054170 con 0x5613ceed1570 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.105+0000 7fdc03610740 10 client.25496 fetch_fsmap learned FSMap version 34 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.105+0000 7fdc03610740 1 -- 172.21.15.98:0/3285566089 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5613cef63940 con 0x5613ceed1570 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.105+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/3285566089 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbec005480 con 0x5613ceed1570 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.106+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/3285566089 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 34) ==== 8338+0+0 (secure 0 0 0) 0x7fdbec003b70 con 0x5613ceed1570 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.106+0000 7fdc03610740 10 client.25496 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.106+0000 7fdc03610740 10 client.25496 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.106+0000 7fdc03610740 20 client.25496 populate_metadata read hostname 'smithi098' 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.106+0000 7fdc03610740 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5613cef63bc0 0x5613cef65fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:41:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.106+0000 7fdc03610740 4 client.25496 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:41:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.106+0000 7fdc03610740 1 -- 172.21.15.98:0/3285566089 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5613cee3e200 con 0x5613cef63bc0 2024-04-25T17:41:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.106+0000 7fdbf9d9b640 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5613cef63bc0 0x5613cef65fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:41:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.107+0000 7fdbe1ffb640 10 client.25496 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:54.106+0000 7fdbf9d9b640 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5613cef63bc0 0x5613cef65fb0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61d11b35c75fea8b server_cookie=b2c55116a58ab891 in_seq=0 out_seq=0 2024-04-25T17:41:54.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:54 smithi173 ceph-mon[32856]: pgmap v183: 97 pgs: 97 active+clean; 3.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 54 MiB/s wr, 1.88k op/s 2024-04-25T17:41:54.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:54 smithi098 ceph-mon[29192]: pgmap v183: 97 pgs: 97 active+clean; 3.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 54 MiB/s wr, 1.88k op/s 2024-04-25T17:41:54.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:54 smithi203 ceph-mon[32834]: pgmap v183: 97 pgs: 97 active+clean; 3.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 54 MiB/s wr, 1.88k op/s 2024-04-25T17:41:56.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:56 smithi173 ceph-mon[32856]: pgmap v184: 97 pgs: 97 active+clean; 3.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 44 MiB/s wr, 1.64k op/s 2024-04-25T17:41:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 325.0 s, 2025.9 tps, lat 27.151 ms stddev 29.670 2024-04-25T17:41:56.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:56 smithi098 ceph-mon[29192]: pgmap v184: 97 pgs: 97 active+clean; 3.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 44 MiB/s wr, 1.64k op/s 2024-04-25T17:41:56.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:56 smithi203 ceph-mon[32834]: pgmap v184: 97 pgs: 97 active+clean; 3.2 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 44 MiB/s wr, 1.64k op/s 2024-04-25T17:41:58.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:41:58 smithi173 ceph-mon[32856]: pgmap v185: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 64 MiB/s wr, 3.06k op/s 2024-04-25T17:41:58.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:41:58 smithi098 ceph-mon[29192]: pgmap v185: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 64 MiB/s wr, 3.06k op/s 2024-04-25T17:41:58.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:41:58 smithi203 ceph-mon[32834]: pgmap v185: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 64 MiB/s wr, 3.06k op/s 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdbe1ffb640 1 -- 172.21.15.98:0/3285566089 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5613cee3e200 con 0x5613cef63bc0 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdbe1ffb640 10 client.25496 handle_command_reply: tid=0 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 client.25496 shutdown 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 -- 172.21.15.98:0/3285566089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 msgr2=0x5613cef10f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613cef10f50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdbec02abd0 tx=0x7fdbec0545f0 comp rx=0 tx=0).stop 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 -- 172.21.15.98:0/3285566089 shutdown_connections 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5613cef63bc0 0x5613cef65fb0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5613ceee5020 0x5613ceed68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613ceed1e90 0x5613ceed63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdbe1ffb640 0 client.25496 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 --2- 172.21.15.98:0/3285566089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5613ceed1570 0x5613cef10f50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 -- 172.21.15.98:0/3285566089 >> 172.21.15.98:0/3285566089 conn(0x5613ceecdb40 msgr2=0x5613ceecc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 -- 172.21.15.98:0/3285566089 shutdown_connections 2024-04-25T17:41:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 1 -- 172.21.15.98:0/3285566089 wait complete. 2024-04-25T17:41:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.968+0000 7fdc03610740 20 client.25496 trim_cache size 0 max 16384 2024-04-25T17:41:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:41:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:41:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:41:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 -- 172.21.15.98:0/1599401958 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbe40a3ea0 msgr2=0x7fdbe40a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbe40a3ea0 0x7fdbe40a6360 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5613ceececd0 tx=0x7fdbf000b390 comp rx=0 tx=0).stop 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 -- 172.21.15.98:0/1599401958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc106f30 msgr2=0x7fdbfc19ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc106f30 0x7fdbfc19ae10 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400efd0 tx=0x7fdbf400c6a0 comp rx=0 tx=0).stop 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 -- 172.21.15.98:0/1599401958 shutdown_connections 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbe40a3ea0 0x7fdbe40a6360 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbfc106f30 0x7fdbfc19ae10 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbfc100e10 0x7fdbfc196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 --2- 172.21.15.98:0/1599401958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbfc100450 0x7fdbfc196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 -- 172.21.15.98:0/1599401958 >> 172.21.15.98:0/1599401958 conn(0x7fdbfc0773e0 msgr2=0x7fdbfc0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:41:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 -- 172.21.15.98:0/1599401958 shutdown_connections 2024-04-25T17:41:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:41:59.969+0000 7fdc00820640 1 -- 172.21.15.98:0/1599401958 wait complete. 2024-04-25T17:42:00.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:00 smithi173 ceph-mon[32856]: pgmap v186: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 26 MiB/s wr, 1.75k op/s 2024-04-25T17:42:00.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:00 smithi098 ceph-mon[29192]: pgmap v186: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 26 MiB/s wr, 1.75k op/s 2024-04-25T17:42:00.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:00 smithi203 ceph-mon[32834]: pgmap v186: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 26 MiB/s wr, 1.75k op/s 2024-04-25T17:42:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 330.0 s, 1064.1 tps, lat 135.834 ms stddev 766.368 2024-04-25T17:42:02.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:02 smithi173 ceph-mon[32856]: pgmap v187: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 27 MiB/s wr, 1.79k op/s 2024-04-25T17:42:02.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:02 smithi098 ceph-mon[29192]: pgmap v187: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 27 MiB/s wr, 1.79k op/s 2024-04-25T17:42:02.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:02 smithi203 ceph-mon[32834]: pgmap v187: 97 pgs: 97 active+clean; 3.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 27 MiB/s wr, 1.79k op/s 2024-04-25T17:42:04.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:04 smithi173 ceph-mon[32856]: pgmap v188: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 27 MiB/s wr, 1.85k op/s 2024-04-25T17:42:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:04 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:42:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:04 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:42:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:04 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:42:04.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:04 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:42:04.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:04 smithi098 ceph-mon[29192]: pgmap v188: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 27 MiB/s wr, 1.85k op/s 2024-04-25T17:42:04.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:04 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:42:04.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:04 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:42:04.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:04 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:42:04.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:04 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:42:04.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:04 smithi203 ceph-mon[32834]: pgmap v188: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 27 MiB/s wr, 1.85k op/s 2024-04-25T17:42:04.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:04 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:42:04.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:04 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:42:04.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:04 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:42:04.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:04 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:42:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:06 smithi173 ceph-mon[32856]: pgmap v189: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 22 MiB/s wr, 1.59k op/s 2024-04-25T17:42:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:06 smithi173 ceph-mon[32856]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T17:42:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 335.0 s, 3078.0 tps, lat 32.713 ms stddev 36.081 2024-04-25T17:42:06.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:06 smithi098 ceph-mon[29192]: pgmap v189: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 22 MiB/s wr, 1.59k op/s 2024-04-25T17:42:06.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:06 smithi098 ceph-mon[29192]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T17:42:06.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:06 smithi203 ceph-mon[32834]: pgmap v189: 97 pgs: 97 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 22 MiB/s wr, 1.59k op/s 2024-04-25T17:42:06.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:06 smithi203 ceph-mon[32834]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T17:42:08.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:08 smithi173 ceph-mon[32856]: pgmap v191: 97 pgs: 97 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 6.8 MiB/s wr, 566 op/s 2024-04-25T17:42:08.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:08 smithi098 ceph-mon[29192]: pgmap v191: 97 pgs: 97 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 6.8 MiB/s wr, 566 op/s 2024-04-25T17:42:08.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:08 smithi203 ceph-mon[32834]: pgmap v191: 97 pgs: 97 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 6.8 MiB/s wr, 566 op/s 2024-04-25T17:42:10.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:10 smithi173 ceph-mon[32856]: pgmap v192: 97 pgs: 97 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 6.8 MiB/s wr, 566 op/s 2024-04-25T17:42:10.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:10 smithi173 ceph-mon[32856]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T17:42:10.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:10 smithi098 ceph-mon[29192]: pgmap v192: 97 pgs: 97 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 6.8 MiB/s wr, 566 op/s 2024-04-25T17:42:10.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:10 smithi098 ceph-mon[29192]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T17:42:10.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:10 smithi203 ceph-mon[32834]: pgmap v192: 97 pgs: 97 active+clean; 3.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 6.8 MiB/s wr, 566 op/s 2024-04-25T17:42:10.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:10 smithi203 ceph-mon[32834]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T17:42:11.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 340.0 s, 3678.8 tps, lat 26.927 ms stddev 30.428 2024-04-25T17:42:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:12 smithi173 ceph-mon[32856]: pgmap v194: 97 pgs: 97 active+clean; 3.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 8.0 MiB/s wr, 592 op/s 2024-04-25T17:42:12.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:12 smithi098 ceph-mon[29192]: pgmap v194: 97 pgs: 97 active+clean; 3.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 8.0 MiB/s wr, 592 op/s 2024-04-25T17:42:12.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:12 smithi203 ceph-mon[32834]: pgmap v194: 97 pgs: 97 active+clean; 3.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 8.0 MiB/s wr, 592 op/s 2024-04-25T17:42:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.082+0000 7f4f76460640 1 -- 172.21.15.98:0/551156892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f70102b80 msgr2=0x7f4f70106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.082+0000 7f4f76460640 1 --2- 172.21.15.98:0/551156892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f70102b80 0x7f4f70106bc0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f4f64009900 tx=0x7f4f6402a410 comp rx=0 tx=0).stop 2024-04-25T17:42:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.083+0000 7f4f76460640 1 -- 172.21.15.98:0/551156892 shutdown_connections 2024-04-25T17:42:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.083+0000 7f4f76460640 1 --2- 172.21.15.98:0/551156892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f701072a0 0x7f4f7010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.083+0000 7f4f76460640 1 --2- 172.21.15.98:0/551156892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f70102b80 0x7f4f70106bc0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.083+0000 7f4f76460640 1 --2- 172.21.15.98:0/551156892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f701021d0 0x7f4f701025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.083+0000 7f4f76460640 1 -- 172.21.15.98:0/551156892 >> 172.21.15.98:0/551156892 conn(0x7f4f700fda10 msgr2=0x7f4f700ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.083+0000 7f4f76460640 1 -- 172.21.15.98:0/551156892 shutdown_connections 2024-04-25T17:42:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.083+0000 7f4f76460640 1 -- 172.21.15.98:0/551156892 wait complete. 2024-04-25T17:42:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.083+0000 7f4f76460640 1 Processor -- start 2024-04-25T17:42:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f76460640 1 -- start start 2024-04-25T17:42:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f76460640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f701021d0 0x7f4f7010b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f76460640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f701072a0 0x7f4f7010bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f76460640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f701a89a0 0x7f4f701aad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f76460640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f70075380 con 0x7f4f701021d0 2024-04-25T17:42:13.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f76460640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4f70075200 con 0x7f4f701072a0 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f76460640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4f70075500 con 0x7f4f701a89a0 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f6f7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f701072a0 0x7f4f7010bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f701021d0 0x7f4f7010b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f749fd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f701a89a0 0x7f4f701aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f6f7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f701072a0 0x7f4f7010bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:44704/0 (socket says 172.21.15.98:44704) 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f6f7fe640 1 -- 172.21.15.98:0/2489417236 learned_addr learned my addr 172.21.15.98:0/2489417236 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f749fd640 1 -- 172.21.15.98:0/2489417236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f701072a0 msgr2=0x7f4f7010bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f749fd640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f701072a0 0x7f4f7010bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f749fd640 1 -- 172.21.15.98:0/2489417236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f701021d0 msgr2=0x7f4f7010b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f749fd640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f701021d0 0x7f4f7010b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.084+0000 7f4f749fd640 1 -- 172.21.15.98:0/2489417236 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f701ab350 con 0x7f4f701a89a0 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.085+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f701021d0 0x7f4f7010b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.085+0000 7f4f749fd640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f701a89a0 0x7f4f701aad90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800da10 tx=0x7f4f6800ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.085+0000 7f4f6d7fa640 1 -- 172.21.15.98:0/2489417236 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6800bd70 con 0x7f4f701a89a0 2024-04-25T17:42:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.085+0000 7f4f76460640 1 -- 172.21.15.98:0/2489417236 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f701ab640 con 0x7f4f701a89a0 2024-04-25T17:42:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.085+0000 7f4f6d7fa640 1 -- 172.21.15.98:0/2489417236 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f68003e20 con 0x7f4f701a89a0 2024-04-25T17:42:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.085+0000 7f4f6d7fa640 1 -- 172.21.15.98:0/2489417236 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f680106c0 con 0x7f4f701a89a0 2024-04-25T17:42:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.085+0000 7f4f76460640 1 -- 172.21.15.98:0/2489417236 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f701abb60 con 0x7f4f701a89a0 2024-04-25T17:42:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.087+0000 7f4f79174740 1 Processor -- start 2024-04-25T17:42:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f6d7fa640 1 -- 172.21.15.98:0/2489417236 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4f68004270 con 0x7f4f701a89a0 2024-04-25T17:42:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f79174740 1 -- start start 2024-04-25T17:42:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f79174740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929620 0x558f9a929a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f79174740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a9342c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f79174740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a936bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f79174740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558f9a88e970 con 0x558f9a929f40 2024-04-25T17:42:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f79174740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558f9a82b4a0 con 0x558f9a929620 2024-04-25T17:42:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f79174740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558f9a88a930 con 0x558f9a934800 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a9342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a9342c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38248/0 (socket says 172.21.15.98:38248) 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f6ffff640 1 -- 172.21.15.98:0/3401544372 learned_addr learned my addr 172.21.15.98:0/3401544372 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f6d7fa640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4f580a3e10 0x7f4f580a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/3401544372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929620 0x558f9a929a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.088+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/3401544372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a936bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.089+0000 7f4f6d7fa640 1 -- 172.21.15.98:0/2489417236 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f4f680d0b40 con 0x7f4f701a89a0 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.089+0000 7f4f749fd640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4f580a3e10 0x7f4f580a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.089+0000 7f4f6f7fe640 1 -- 172.21.15.98:0/3401544372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 msgr2=0x558f9a936bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.089+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/3401544372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a936bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.089+0000 7f4f6f7fe640 1 -- 172.21.15.98:0/3401544372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 msgr2=0x558f9a9342c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.089+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/3401544372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a9342c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.089+0000 7f4f6f7fe640 1 -- 172.21.15.98:0/3401544372 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f9a5a1210 con 0x558f9a929620 2024-04-25T17:42:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.090+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/3401544372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929620 0x558f9a929a00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4f64006010 tx=0x7f4f64005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35d60d7c18c7145d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:13.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.090+0000 7f4f567fc640 1 -- 172.21.15.98:0/3401544372 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f64035070 con 0x558f9a929620 2024-04-25T17:42:13.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.090+0000 7f4f567fc640 1 -- 172.21.15.98:0/3401544372 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f64003dc0 con 0x558f9a929620 2024-04-25T17:42:13.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.090+0000 7f4f567fc640 1 -- 172.21.15.98:0/3401544372 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f640049e0 con 0x558f9a929620 2024-04-25T17:42:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.091+0000 7f4f749fd640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4f580a3e10 0x7f4f580a62d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4f680a5220 tx=0x7f4f6800bf10 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.091+0000 7f4f79174740 1 -- 172.21.15.98:0/3401544372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929620 msgr2=0x558f9a929a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.092+0000 7f4f79174740 1 --2- 172.21.15.98:0/3401544372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929620 0x558f9a929a00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4f64006010 tx=0x7f4f64005b50 comp rx=0 tx=0).stop 2024-04-25T17:42:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.092+0000 7f4f79174740 1 -- 172.21.15.98:0/3401544372 shutdown_connections 2024-04-25T17:42:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.092+0000 7f4f79174740 1 --2- 172.21.15.98:0/3401544372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a936bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.092+0000 7f4f79174740 1 --2- 172.21.15.98:0/3401544372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a9342c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.092+0000 7f4f79174740 1 --2- 172.21.15.98:0/3401544372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929620 0x558f9a929a00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.092+0000 7f4f79174740 1 -- 172.21.15.98:0/3401544372 >> 172.21.15.98:0/3401544372 conn(0x558f9a925b30 msgr2=0x558f9a925f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.092+0000 7f4f79174740 1 -- 172.21.15.98:0/3401544372 shutdown_connections 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.092+0000 7f4f79174740 1 -- 172.21.15.98:0/3401544372 wait complete. 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.092+0000 7f4f79174740 1 -- start start 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f79174740 1 Processor -- start 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f79174740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9341d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f79174740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 0x558f9a9314c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9341d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38254/0 (socket says 172.21.15.98:38254) 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f79174740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a931ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f6f7fe640 1 -- 172.21.15.98:0/1930471827 learned_addr learned my addr 172.21.15.98:0/1930471827 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558f9a9373d0 con 0x558f9a929620 2024-04-25T17:42:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558f9a8290b0 con 0x558f9a929f40 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558f9a937550 con 0x558f9a934800 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 0x558f9a9314c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.093+0000 7f4f79174740 10 client.0 _ll_register_callbacks cb 0x558f9a816d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a931ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f6ffff640 1 -- 172.21.15.98:0/1930471827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 msgr2=0x558f9a931ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a931ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f6ffff640 1 -- 172.21.15.98:0/1930471827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 msgr2=0x558f9a9341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f6ffff640 1 -- 172.21.15.98:0/1930471827 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f9a66d620 con 0x558f9a929f40 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 0x558f9a9314c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x558f9a92abc0 tx=0x7f4f6000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb98665db6738562 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/1930471827 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f600120d0 con 0x558f9a929f40 2024-04-25T17:42:13.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f64003dc0 con 0x558f9a929f40 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/1930471827 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f4f60005020 con 0x558f9a929f40 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/1930471827 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4f60012270 con 0x558f9a929f40 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f79174740 10 client.25508 fetch_fsmap learned FSMap version 34 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x558f9a9431c0 con 0x558f9a929f40 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.094+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/1930471827 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f64003dc0 con 0x558f9a929f40 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.095+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/1930471827 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 34) ==== 8338+0+0 (secure 0 0 0) 0x7f4f60025480 con 0x558f9a929f40 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.095+0000 7f4f79174740 10 client.25508 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.095+0000 7f4f79174740 10 client.25508 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.095+0000 7f4f79174740 20 client.25508 populate_metadata read hostname 'smithi098' 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.095+0000 7f4f79174740 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558f9a944630 0x558f9a946a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.095+0000 7f4f79174740 4 client.25508 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.095+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f9a89c390 con 0x558f9a944630 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.095+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558f9a944630 0x558f9a946a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.095+0000 7f4f4f7fe640 10 client.25508 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.095+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558f9a944630 0x558f9a946a20 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a437ed391691731d server_cookie=cc591cd4ab85b02 in_seq=0 out_seq=0 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/1930471827 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f9a89c390 con 0x558f9a944630 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f4f7fe640 10 client.25508 handle_command_reply: tid=0 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f79174740 1 client.25508 shutdown 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 msgr2=0x558f9a9314c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f79174740 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 0x558f9a9314c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x558f9a92abc0 tx=0x7f4f6000f790 comp rx=0 tx=0).stop 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 shutdown_connections 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f79174740 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558f9a944630 0x558f9a946a20 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f79174740 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a931ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f79174740 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 0x558f9a9314c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f79174740 1 --2- 172.21.15.98:0/1930471827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.096+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 >> 172.21.15.98:0/1930471827 conn(0x558f9a925b30 msgr2=0x558f9a92b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.097+0000 7f4f4f7fe640 0 client.25508 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.097+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 shutdown_connections 2024-04-25T17:42:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.097+0000 7f4f79174740 1 -- 172.21.15.98:0/1930471827 wait complete. 2024-04-25T17:42:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.097+0000 7f4f79174740 20 client.25508 trim_cache size 0 max 16384 2024-04-25T17:42:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f79174740 1 Processor -- start 2024-04-25T17:42:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f79174740 1 -- start start 2024-04-25T17:42:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f79174740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a929620 0x558f9a940290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f79174740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a92b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f79174740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a934800 0x558f9a933d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f79174740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f701b69d0 con 0x558f9a929f40 2024-04-25T17:42:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f79174740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4f701b42a0 con 0x558f9a934800 2024-04-25T17:42:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f79174740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4f701b2110 con 0x558f9a929620 2024-04-25T17:42:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a92b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a92b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38268/0 (socket says 172.21.15.98:38268) 2024-04-25T17:42:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.116+0000 7f4f6ffff640 1 -- 172.21.15.98:0/2781286637 learned_addr learned my addr 172.21.15.98:0/2781286637 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.117+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/2781286637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a929620 0x558f9a940290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.117+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/2781286637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a934800 0x558f9a933d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.117+0000 7f4f6ffff640 1 -- 172.21.15.98:0/2781286637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a929620 msgr2=0x558f9a940290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.117+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/2781286637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a929620 0x558f9a940290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.117+0000 7f4f6ffff640 1 -- 172.21.15.98:0/2781286637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a934800 msgr2=0x558f9a933d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.117+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/2781286637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a934800 0x558f9a933d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.117+0000 7f4f6ffff640 1 -- 172.21.15.98:0/2781286637 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f9a9293e0 con 0x558f9a929f40 2024-04-25T17:42:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.118+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/2781286637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a92b730 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4f60025a40 tx=0x7f4f6000fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ac884f0da02bb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.118+0000 7f4f55ffb640 1 -- 172.21.15.98:0/2781286637 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60025480 con 0x558f9a929f40 2024-04-25T17:42:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.118+0000 7f4f55ffb640 1 -- 172.21.15.98:0/2781286637 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f60025620 con 0x558f9a929f40 2024-04-25T17:42:13.119 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.118+0000 7f4f55ffb640 1 -- 172.21.15.98:0/2781286637 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60003ba0 con 0x558f9a929f40 2024-04-25T17:42:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.119+0000 7f4f79174740 1 -- 172.21.15.98:0/2781286637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 msgr2=0x558f9a92b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.119+0000 7f4f79174740 1 --2- 172.21.15.98:0/2781286637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a92b730 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4f60025a40 tx=0x7f4f6000fd10 comp rx=0 tx=0).stop 2024-04-25T17:42:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.119+0000 7f4f79174740 1 -- 172.21.15.98:0/2781286637 shutdown_connections 2024-04-25T17:42:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.119+0000 7f4f79174740 1 --2- 172.21.15.98:0/2781286637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a934800 0x558f9a933d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.119+0000 7f4f79174740 1 --2- 172.21.15.98:0/2781286637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929f40 0x558f9a92b730 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.119+0000 7f4f79174740 1 --2- 172.21.15.98:0/2781286637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a929620 0x558f9a940290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.119+0000 7f4f79174740 1 -- 172.21.15.98:0/2781286637 >> 172.21.15.98:0/2781286637 conn(0x558f9a925b30 msgr2=0x558f9a92b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.119+0000 7f4f79174740 1 -- 172.21.15.98:0/2781286637 shutdown_connections 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.119+0000 7f4f79174740 1 -- 172.21.15.98:0/2781286637 wait complete. 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.119+0000 7f4f79174740 1 -- start start 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f79174740 1 Processor -- start 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f79174740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f79174740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 0x558f9a9a33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f79174740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a937b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f79174740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4f701b2760 con 0x558f9a929620 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f79174740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4f701b3bd0 con 0x558f9a929f40 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f79174740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4f701b2e30 con 0x558f9a934800 2024-04-25T17:42:13.122 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f79174740 10 client.0 _ll_register_callbacks cb 0x7f4f60012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f6f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9a2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38284/0 (socket says 172.21.15.98:38284) 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f6f7fe640 1 -- 172.21.15.98:0/333862580 learned_addr learned my addr 172.21.15.98:0/333862580 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 0x558f9a9a33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.120+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a937b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.121+0000 7f4f6f7fe640 1 -- 172.21.15.98:0/333862580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 msgr2=0x558f9a9a33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.121+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 0x558f9a9a33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.121+0000 7f4f6f7fe640 1 -- 172.21.15.98:0/333862580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 msgr2=0x558f9a9a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.121+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.121+0000 7f4f6f7fe640 1 -- 172.21.15.98:0/333862580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f9a938260 con 0x558f9a934800 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.121+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 0x558f9a9a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.121+0000 7f4f6f7fe640 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a937b70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f640057c0 tx=0x7f4f64005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57507593e96e18f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.121+0000 7f4f4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.121+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/333862580 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f64035040 con 0x558f9a934800 2024-04-25T17:42:13.123 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.122+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/333862580 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f4f64003ac0 con 0x558f9a934800 2024-04-25T17:42:13.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.122+0000 7f4f79174740 1 -- 172.21.15.98:0/333862580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f60025620 con 0x558f9a934800 2024-04-25T17:42:13.124 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.124+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/333862580 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4f64003cc0 con 0x558f9a934800 2024-04-25T17:42:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.127+0000 7f4f79174740 10 client.25498 fetch_fsmap learned FSMap version 34 2024-04-25T17:42:13.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.127+0000 7f4f79174740 1 -- 172.21.15.98:0/333862580 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x558f9a9aa630 con 0x558f9a934800 2024-04-25T17:42:13.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.134+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/333862580 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f64003e60 con 0x558f9a934800 2024-04-25T17:42:13.134 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.134+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/333862580 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 34) ==== 8338+0+0 (secure 0 0 0) 0x7f4f640315c0 con 0x558f9a934800 2024-04-25T17:42:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.134+0000 7f4f79174740 10 client.25498 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T17:42:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.134+0000 7f4f79174740 10 client.25498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:42:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.134+0000 7f4f79174740 20 client.25498 populate_metadata read hostname 'smithi098' 2024-04-25T17:42:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.134+0000 7f4f79174740 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558f9a9aa910 0x558f9a9acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:13.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.135+0000 7f4f79174740 4 client.25498 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:42:13.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.135+0000 7f4f79174740 1 -- 172.21.15.98:0/333862580 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f4f60025620 con 0x558f9a9aa910 2024-04-25T17:42:13.137 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.135+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558f9a9aa910 0x558f9a9acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:13.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.138+0000 7f4f6ffff640 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558f9a9aa910 0x558f9a9acd00 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=437a180cfb9ae692 server_cookie=24fbaa3ff8de9fbc in_seq=0 out_seq=0 2024-04-25T17:42:13.138 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.138+0000 7f4f4f7fe640 10 client.25498 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f4f7fe640 1 -- 172.21.15.98:0/333862580 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f4f60025620 con 0x558f9a9aa910 2024-04-25T17:42:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f4f7fe640 10 client.25498 handle_command_reply: tid=0 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 client.25498 shutdown 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 -- 172.21.15.98:0/333862580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 msgr2=0x558f9a937b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a937b70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f640057c0 tx=0x7f4f64005b50 comp rx=0 tx=0).stop 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 -- 172.21.15.98:0/333862580 shutdown_connections 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558f9a9aa910 0x558f9a9acd00 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f9a934800 0x558f9a937b70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558f9a929f40 0x558f9a9a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 --2- 172.21.15.98:0/333862580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f9a929620 0x558f9a9a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 -- 172.21.15.98:0/333862580 >> 172.21.15.98:0/333862580 conn(0x558f9a925b30 msgr2=0x558f9a924cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f4f7fe640 0 client.25498 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 -- 172.21.15.98:0/333862580 shutdown_connections 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 1 -- 172.21.15.98:0/333862580 wait complete. 2024-04-25T17:42:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.174+0000 7f4f79174740 20 client.25498 trim_cache size 0 max 16384 2024-04-25T17:42:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:42:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:42:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:42:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 -- 172.21.15.98:0/2489417236 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4f580a3e10 msgr2=0x7f4f580a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4f580a3e10 0x7f4f580a62d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4f680a5220 tx=0x7f4f6800bf10 comp rx=0 tx=0).stop 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 -- 172.21.15.98:0/2489417236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f701a89a0 msgr2=0x7f4f701aad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f701a89a0 0x7f4f701aad90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800da10 tx=0x7f4f6800ded0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 -- 172.21.15.98:0/2489417236 shutdown_connections 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f4f580a3e10 0x7f4f580a62d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f701a89a0 0x7f4f701aad90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f701072a0 0x7f4f7010bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 --2- 172.21.15.98:0/2489417236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4f701021d0 0x7f4f7010b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 -- 172.21.15.98:0/2489417236 >> 172.21.15.98:0/2489417236 conn(0x7f4f700fda10 msgr2=0x7f4f700fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 -- 172.21.15.98:0/2489417236 shutdown_connections 2024-04-25T17:42:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:13.175+0000 7f4f76460640 1 -- 172.21.15.98:0/2489417236 wait complete. 2024-04-25T17:42:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:14 smithi173 ceph-mon[32856]: pgmap v195: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 12 MiB/s wr, 939 op/s 2024-04-25T17:42:14.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:14 smithi098 ceph-mon[29192]: pgmap v195: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 12 MiB/s wr, 939 op/s 2024-04-25T17:42:14.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:14 smithi203 ceph-mon[32834]: pgmap v195: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 12 MiB/s wr, 939 op/s 2024-04-25T17:42:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:16 smithi173 ceph-mon[32856]: pgmap v196: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 11 MiB/s wr, 819 op/s 2024-04-25T17:42:16.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 345.0 s, 2895.6 tps, lat 34.853 ms stddev 37.904 2024-04-25T17:42:16.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:16 smithi098 ceph-mon[29192]: pgmap v196: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 11 MiB/s wr, 819 op/s 2024-04-25T17:42:16.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:16 smithi203 ceph-mon[32834]: pgmap v196: 97 pgs: 97 active+clean; 3.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 11 MiB/s wr, 819 op/s 2024-04-25T17:42:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:18 smithi173 ceph-mon[32856]: pgmap v197: 97 pgs: 97 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 879 op/s 2024-04-25T17:42:18.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:18 smithi098 ceph-mon[29192]: pgmap v197: 97 pgs: 97 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 879 op/s 2024-04-25T17:42:18.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:18 smithi203 ceph-mon[32834]: pgmap v197: 97 pgs: 97 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 879 op/s 2024-04-25T17:42:20.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:20 smithi173 ceph-mon[32856]: pgmap v198: 97 pgs: 97 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 879 op/s 2024-04-25T17:42:20.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:20 smithi098 ceph-mon[29192]: pgmap v198: 97 pgs: 97 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 879 op/s 2024-04-25T17:42:20.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:20 smithi203 ceph-mon[32834]: pgmap v198: 97 pgs: 97 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 879 op/s 2024-04-25T17:42:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 350.0 s, 4029.8 tps, lat 24.703 ms stddev 27.783 2024-04-25T17:42:21.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:21 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:21.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:21 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:21.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:21 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:22.612 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:22 smithi203 ceph-mon[32834]: pgmap v199: 97 pgs: 97 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 12 MiB/s wr, 996 op/s 2024-04-25T17:42:22.672 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:22 smithi173 ceph-mon[32856]: pgmap v199: 97 pgs: 97 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 12 MiB/s wr, 996 op/s 2024-04-25T17:42:22.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:22 smithi098 ceph-mon[29192]: pgmap v199: 97 pgs: 97 active+clean; 3.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 12 MiB/s wr, 996 op/s 2024-04-25T17:42:24.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:24 smithi098 ceph-mon[29192]: pgmap v200: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 12 MiB/s wr, 970 op/s 2024-04-25T17:42:24.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:24 smithi203 ceph-mon[32834]: pgmap v200: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 12 MiB/s wr, 970 op/s 2024-04-25T17:42:24.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:24 smithi173 ceph-mon[32856]: pgmap v200: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 12 MiB/s wr, 970 op/s 2024-04-25T17:42:25.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:25 smithi098 ceph-mon[29192]: pgmap v201: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.9 MiB/s wr, 738 op/s 2024-04-25T17:42:25.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:25 smithi203 ceph-mon[32834]: pgmap v201: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.9 MiB/s wr, 738 op/s 2024-04-25T17:42:25.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:25 smithi173 ceph-mon[32856]: pgmap v201: 97 pgs: 97 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.9 MiB/s wr, 738 op/s 2024-04-25T17:42:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.309+0000 7f1047aae640 1 -- 172.21.15.98:0/3033138627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040100fd0 msgr2=0x7f10401013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.309+0000 7f1047aae640 1 --2- 172.21.15.98:0/3033138627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040100fd0 0x7f10401013b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1038009ca0 tx=0x7f103802a9f0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.310+0000 7f1047aae640 1 -- 172.21.15.98:0/3033138627 shutdown_connections 2024-04-25T17:42:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.310+0000 7f1047aae640 1 --2- 172.21.15.98:0/3033138627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10401062f0 0x7f1040106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.310+0000 7f1047aae640 1 --2- 172.21.15.98:0/3033138627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10401018f0 0x7f1040105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.310+0000 7f1047aae640 1 --2- 172.21.15.98:0/3033138627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040100fd0 0x7f10401013b0 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1038009ca0 tx=0x7f103802a9f0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.310+0000 7f1047aae640 1 -- 172.21.15.98:0/3033138627 >> 172.21.15.98:0/3033138627 conn(0x7f1040077fd0 msgr2=0x7f10400783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.310+0000 7f1047aae640 1 -- 172.21.15.98:0/3033138627 shutdown_connections 2024-04-25T17:42:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.310+0000 7f1047aae640 1 -- 172.21.15.98:0/3033138627 wait complete. 2024-04-25T17:42:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.311+0000 7f1047aae640 1 Processor -- start 2024-04-25T17:42:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f1047aae640 1 -- start start 2024-04-25T17:42:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f1047aae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10401018f0 0x7f104019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f1047aae640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10401062f0 0x7f104019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f1047aae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10401a3820 0x7f10401a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f1047aae640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1040118880 con 0x7f10401a3820 2024-04-25T17:42:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f1047aae640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1040118700 con 0x7f10401062f0 2024-04-25T17:42:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f1047aae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1040118a00 con 0x7f10401018f0 2024-04-25T17:42:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f104584a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10401018f0 0x7f104019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f104584a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10401018f0 0x7f104019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53986/0 (socket says 172.21.15.98:53986) 2024-04-25T17:42:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f104584a640 1 -- 172.21.15.98:0/2683752712 learned_addr learned my addr 172.21.15.98:0/2683752712 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.312+0000 7f1045049640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10401062f0 0x7f104019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.313+0000 7f1045049640 1 -- 172.21.15.98:0/2683752712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10401018f0 msgr2=0x7f104019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.313+0000 7f104604b640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10401a3820 0x7f10401a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.313+0000 7f1045049640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10401018f0 0x7f104019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.313+0000 7f1045049640 1 -- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10401a3820 msgr2=0x7f10401a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.313+0000 7f1045049640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10401a3820 0x7f10401a3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.313+0000 7f1045049640 1 -- 172.21.15.98:0/2683752712 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10401a4310 con 0x7f10401062f0 2024-04-25T17:42:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.313+0000 7f104604b640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10401a3820 0x7f10401a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:42:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.313+0000 7f1045049640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10401062f0 0x7f104019f4a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1030009fd0 tx=0x7f103000b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.313+0000 7f1036ffd640 1 -- 172.21.15.98:0/2683752712 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030015020 con 0x7f10401062f0 2024-04-25T17:42:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.314+0000 7f1036ffd640 1 -- 172.21.15.98:0/2683752712 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1030011ce0 con 0x7f10401062f0 2024-04-25T17:42:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.314+0000 7f1036ffd640 1 -- 172.21.15.98:0/2683752712 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030010e40 con 0x7f10401062f0 2024-04-25T17:42:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.314+0000 7f1047aae640 1 -- 172.21.15.98:0/2683752712 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10401a4600 con 0x7f10401062f0 2024-04-25T17:42:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.314+0000 7f1047aae640 1 -- 172.21.15.98:0/2683752712 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10401abdf0 con 0x7f10401062f0 2024-04-25T17:42:26.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.318+0000 7f1036ffd640 1 -- 172.21.15.98:0/2683752712 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1030018020 con 0x7f10401062f0 2024-04-25T17:42:26.319 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.318+0000 7f104a7c9740 1 Processor -- start 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.318+0000 7f104a7c9740 1 -- start start 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.318+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104a7c9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555982328970 con 0x5559823c3620 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104a7c9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5559822c54a0 con 0x5559823c3f40 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104a7c9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555982324930 con 0x5559823ce800 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104604b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104584a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104584a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823ce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:39818/0 (socket says 172.21.15.98:39818) 2024-04-25T17:42:26.320 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104604b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823c3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52078/0 (socket says 172.21.15.98:52078) 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104584a640 1 -- 172.21.15.98:0/60568232 learned_addr learned my addr 172.21.15.98:0/60568232 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f1045049640 1 --2- 172.21.15.98:0/60568232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f1036ffd640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f102c0a3e40 0x7f102c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104584a640 1 -- 172.21.15.98:0/60568232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 msgr2=0x5559823d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104584a640 1 --2- 172.21.15.98:0/60568232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823d0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104604b640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f102c0a3e40 0x7f102c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104584a640 1 -- 172.21.15.98:0/60568232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 msgr2=0x5559823c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104584a640 1 --2- 172.21.15.98:0/60568232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104584a640 1 -- 172.21.15.98:0/60568232 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55598203b210 con 0x5559823c3f40 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f1036ffd640 1 -- 172.21.15.98:0/2683752712 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f10300cff70 con 0x7f10401062f0 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104584a640 1 --2- 172.21.15.98:0/60568232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823ce2c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1038035000 tx=0x7f1038005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=194913ef3453ad52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.319+0000 7f104604b640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f102c0a3e40 0x7f102c0a6300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f103c005fd0 tx=0x7f103c004730 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.320+0000 7f102bfff640 1 -- 172.21.15.98:0/60568232 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038033050 con 0x5559823c3f40 2024-04-25T17:42:26.321 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.320+0000 7f1045049640 1 --2- 172.21.15.98:0/60568232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:42:26.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.320+0000 7f102bfff640 1 -- 172.21.15.98:0/60568232 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1038047070 con 0x5559823c3f40 2024-04-25T17:42:26.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.320+0000 7f102bfff640 1 -- 172.21.15.98:0/60568232 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103802f8b0 con 0x5559823c3f40 2024-04-25T17:42:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.320+0000 7f104a7c9740 1 -- 172.21.15.98:0/60568232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 msgr2=0x5559823ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.320+0000 7f104a7c9740 1 --2- 172.21.15.98:0/60568232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823ce2c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1038035000 tx=0x7f1038005d80 comp rx=0 tx=0).stop 2024-04-25T17:42:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.321+0000 7f104a7c9740 1 -- 172.21.15.98:0/60568232 shutdown_connections 2024-04-25T17:42:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.321+0000 7f104a7c9740 1 --2- 172.21.15.98:0/60568232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.321+0000 7f104a7c9740 1 --2- 172.21.15.98:0/60568232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823ce2c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.321+0000 7f104a7c9740 1 --2- 172.21.15.98:0/60568232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823c3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.321+0000 7f104a7c9740 1 -- 172.21.15.98:0/60568232 >> 172.21.15.98:0/60568232 conn(0x5559823bfb30 msgr2=0x5559823bff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.321+0000 7f104a7c9740 1 -- 172.21.15.98:0/60568232 shutdown_connections 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.321+0000 7f104a7c9740 1 -- 172.21.15.98:0/60568232 wait complete. 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.322+0000 7f104a7c9740 1 -- start start 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.322+0000 7f104a7c9740 1 Processor -- start 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.323+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.323+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.323+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.323+0000 7f104a7c9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5559823d13d0 con 0x5559823c3620 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.323+0000 7f104a7c9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5559822c30b0 con 0x5559823c3f40 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.323+0000 7f104a7c9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5559823d1550 con 0x5559823ce800 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.323+0000 7f104a7c9740 10 client.0 _ll_register_callbacks cb 0x5559822b0d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:42:26.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.323+0000 7f104604b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.323+0000 7f104604b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:54000/0 (socket says 172.21.15.98:54000) 2024-04-25T17:42:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.323+0000 7f104604b640 1 -- 172.21.15.98:0/4174564068 learned_addr learned my addr 172.21.15.98:0/4174564068 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.324+0000 7f1045049640 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.324+0000 7f1045049640 1 -- 172.21.15.98:0/4174564068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 msgr2=0x5559823cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.324+0000 7f1045049640 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.324+0000 7f1045049640 1 -- 172.21.15.98:0/4174564068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 msgr2=0x5559823ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.324+0000 7f1045049640 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.324+0000 7f1045049640 1 -- 172.21.15.98:0/4174564068 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555982107620 con 0x5559823c3f40 2024-04-25T17:42:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.325+0000 7f1045049640 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823cb440 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f10300cd770 tx=0x7f103009bb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b85953597f7baea8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.325+0000 7f10297fa640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:42:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.325+0000 7f10297fa640 1 -- 172.21.15.98:0/4174564068 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10300a0370 con 0x5559823c3f40 2024-04-25T17:42:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.325+0000 7f10297fa640 1 -- 172.21.15.98:0/4174564068 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f10300cd8e0 con 0x5559823c3f40 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.325+0000 7f104a7c9740 1 -- 172.21.15.98:0/4174564068 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1038047070 con 0x5559823c3f40 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.325+0000 7f10297fa640 1 -- 172.21.15.98:0/4174564068 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f10300cdae0 con 0x5559823c3f40 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.325+0000 7f104a7c9740 10 client.25523 fetch_fsmap learned FSMap version 36 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.325+0000 7f104a7c9740 1 -- 172.21.15.98:0/4174564068 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5559823dd360 con 0x5559823c3f40 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.325+0000 7f10297fa640 1 -- 172.21.15.98:0/4174564068 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1038047070 con 0x5559823c3f40 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.326+0000 7f10297fa640 1 -- 172.21.15.98:0/4174564068 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f103009d6c0 con 0x5559823c3f40 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.326+0000 7f104a7c9740 10 client.25523 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.326+0000 7f104a7c9740 10 client.25523 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.326+0000 7f104a7c9740 20 client.25523 populate_metadata read hostname 'smithi098' 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.326+0000 7f104a7c9740 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5559823de8c0 0x5559823e0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.326+0000 7f104a7c9740 4 client.25523 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.326+0000 7f104a7c9740 1 -- 172.21.15.98:0/4174564068 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555982336390 con 0x5559823de8c0 2024-04-25T17:42:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.326+0000 7f104584a640 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5559823de8c0 0x5559823e0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.327+0000 7f104584a640 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5559823de8c0 0x5559823e0d60 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c023e02be61d440 server_cookie=1ca5fbd037d6dacf in_seq=0 out_seq=0 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.327+0000 7f10297fa640 10 client.25523 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.327+0000 7f10297fa640 1 -- 172.21.15.98:0/4174564068 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555982336390 con 0x5559823de8c0 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.327+0000 7f10297fa640 10 client.25523 handle_command_reply: tid=0 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.327+0000 7f104a7c9740 1 client.25523 shutdown 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.327+0000 7f104a7c9740 1 -- 172.21.15.98:0/4174564068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 msgr2=0x5559823cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.327+0000 7f104a7c9740 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823cb440 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f10300cd770 tx=0x7f103009bb80 comp rx=0 tx=0).stop 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.328+0000 7f104a7c9740 1 -- 172.21.15.98:0/4174564068 shutdown_connections 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.328+0000 7f104a7c9740 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5559823de8c0 0x5559823e0d60 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.328+0000 7f104a7c9740 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.328+0000 7f104a7c9740 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823cb440 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.328+0000 7f104a7c9740 1 --2- 172.21.15.98:0/4174564068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.328+0000 7f10297fa640 0 client.25523 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.328+0000 7f104a7c9740 1 -- 172.21.15.98:0/4174564068 >> 172.21.15.98:0/4174564068 conn(0x5559823bfb30 msgr2=0x5559823c5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.328+0000 7f104a7c9740 1 -- 172.21.15.98:0/4174564068 shutdown_connections 2024-04-25T17:42:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.328+0000 7f104a7c9740 1 -- 172.21.15.98:0/4174564068 wait complete. 2024-04-25T17:42:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.328+0000 7f104a7c9740 20 client.25523 trim_cache size 0 max 16384 2024-04-25T17:42:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.348+0000 7f104a7c9740 1 Processor -- start 2024-04-25T17:42:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.348+0000 7f104a7c9740 1 -- start start 2024-04-25T17:42:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.348+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823da290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.348+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823beb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.348+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823e6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.348+0000 7f104a7c9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10401b7070 con 0x5559823c3620 2024-04-25T17:42:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.348+0000 7f104a7c9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10401b48e0 con 0x5559823c3f40 2024-04-25T17:42:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.348+0000 7f104a7c9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f10401b2750 con 0x5559823ce800 2024-04-25T17:42:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f104584a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823da290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f104584a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823da290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52106/0 (socket says 172.21.15.98:52106) 2024-04-25T17:42:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f1045049640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823beb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f104584a640 1 -- 172.21.15.98:0/769083975 learned_addr learned my addr 172.21.15.98:0/769083975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f104604b640 1 --2- 172.21.15.98:0/769083975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823e6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f104584a640 1 -- 172.21.15.98:0/769083975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 msgr2=0x5559823e6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f104584a640 1 --2- 172.21.15.98:0/769083975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823e6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f104584a640 1 -- 172.21.15.98:0/769083975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 msgr2=0x5559823beb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f104584a640 1 --2- 172.21.15.98:0/769083975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823beb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f104584a640 1 -- 172.21.15.98:0/769083975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559823cbd10 con 0x5559823c3620 2024-04-25T17:42:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f1045049640 1 --2- 172.21.15.98:0/769083975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823beb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:42:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f104584a640 1 --2- 172.21.15.98:0/769083975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823da290 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f103802afe0 tx=0x7f103802e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c66b403e407f5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f102b7fe640 1 -- 172.21.15.98:0/769083975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038033040 con 0x5559823c3620 2024-04-25T17:42:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f102b7fe640 1 -- 172.21.15.98:0/769083975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1038047040 con 0x5559823c3620 2024-04-25T17:42:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.349+0000 7f102b7fe640 1 -- 172.21.15.98:0/769083975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103802f9d0 con 0x5559823c3620 2024-04-25T17:42:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.350+0000 7f104a7c9740 1 -- 172.21.15.98:0/769083975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 msgr2=0x5559823da290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.350+0000 7f104a7c9740 1 --2- 172.21.15.98:0/769083975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823da290 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f103802afe0 tx=0x7f103802e780 comp rx=0 tx=0).stop 2024-04-25T17:42:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.350+0000 7f104a7c9740 1 -- 172.21.15.98:0/769083975 shutdown_connections 2024-04-25T17:42:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.350+0000 7f104a7c9740 1 --2- 172.21.15.98:0/769083975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823e6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.350+0000 7f104a7c9740 1 --2- 172.21.15.98:0/769083975 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823beb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.350+0000 7f104a7c9740 1 --2- 172.21.15.98:0/769083975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823da290 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.350+0000 7f104a7c9740 1 -- 172.21.15.98:0/769083975 >> 172.21.15.98:0/769083975 conn(0x5559823bfb30 msgr2=0x5559823be420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.350+0000 7f104a7c9740 1 -- 172.21.15.98:0/769083975 shutdown_connections 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.350+0000 7f104a7c9740 1 -- 172.21.15.98:0/769083975 wait complete. 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.350+0000 7f104a7c9740 1 -- start start 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104a7c9740 1 Processor -- start 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823d6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823d3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104a7c9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823d4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104584a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823d6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104584a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823d6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52120/0 (socket says 172.21.15.98:52120) 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104584a640 1 -- 172.21.15.98:0/2033420645 learned_addr learned my addr 172.21.15.98:0/2033420645 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104a7c9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10401b2da0 con 0x5559823c3620 2024-04-25T17:42:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104a7c9740 1 -- 172.21.15.98:0/2033420645 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10401b4210 con 0x5559823c3f40 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104a7c9740 1 -- 172.21.15.98:0/2033420645 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f10401b3470 con 0x5559823ce800 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f1045049640 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823d3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104a7c9740 10 client.0 _ll_register_callbacks cb 0x7f103009bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.351+0000 7f104604b640 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823d4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f104604b640 1 -- 172.21.15.98:0/2033420645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 msgr2=0x5559823d3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f104604b640 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823d3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f104604b640 1 -- 172.21.15.98:0/2033420645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 msgr2=0x5559823d6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f104604b640 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823d6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f104604b640 1 -- 172.21.15.98:0/2033420645 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559823d4c50 con 0x5559823ce800 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f104584a640 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823d6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f104604b640 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823d4450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f103c009c50 tx=0x7f103c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28fb6d0b817b1529 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f10297fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f10297fa640 1 -- 172.21.15.98:0/2033420645 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c02f060 con 0x5559823ce800 2024-04-25T17:42:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f10297fa640 1 -- 172.21.15.98:0/2033420645 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f103c022a80 con 0x5559823ce800 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.352+0000 7f104a7c9740 1 -- 172.21.15.98:0/2033420645 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1038047040 con 0x5559823ce800 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.353+0000 7f10297fa640 1 -- 172.21.15.98:0/2033420645 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f103c022c80 con 0x5559823ce800 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.353+0000 7f104a7c9740 10 client.25516 fetch_fsmap learned FSMap version 36 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.353+0000 7f104a7c9740 1 -- 172.21.15.98:0/2033420645 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555982444f30 con 0x5559823ce800 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.353+0000 7f10297fa640 1 -- 172.21.15.98:0/2033420645 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1038047040 con 0x5559823ce800 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.353+0000 7f10297fa640 1 -- 172.21.15.98:0/2033420645 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f103c029030 con 0x5559823ce800 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.354+0000 7f104a7c9740 10 client.25516 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.354+0000 7f104a7c9740 10 client.25516 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.354+0000 7f104a7c9740 20 client.25516 populate_metadata read hostname 'smithi098' 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.354+0000 7f104a7c9740 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5559824456c0 0x555982447ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.354+0000 7f104a7c9740 4 client.25516 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.354+0000 7f104a7c9740 1 -- 172.21.15.98:0/2033420645 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555982330200 con 0x5559824456c0 2024-04-25T17:42:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.354+0000 7f104584a640 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5559824456c0 0x555982447ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.354+0000 7f10297fa640 10 client.25516 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:26.359 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.354+0000 7f104584a640 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5559824456c0 0x555982447ae0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b5217ab67d8fe58 server_cookie=c5ef78fd4225cba7 in_seq=0 out_seq=0 2024-04-25T17:42:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f10297fa640 1 -- 172.21.15.98:0/2033420645 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555982330200 con 0x5559824456c0 2024-04-25T17:42:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f10297fa640 10 client.25516 handle_command_reply: tid=0 2024-04-25T17:42:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f104a7c9740 1 client.25516 shutdown 2024-04-25T17:42:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f104a7c9740 1 -- 172.21.15.98:0/2033420645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 msgr2=0x5559823d4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f104a7c9740 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823d4450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f103c009c50 tx=0x7f103c0046c0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f104a7c9740 1 -- 172.21.15.98:0/2033420645 shutdown_connections 2024-04-25T17:42:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f104a7c9740 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5559824456c0 0x555982447ae0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f10297fa640 0 client.25516 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f104a7c9740 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5559823ce800 0x5559823d4450 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f104a7c9740 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5559823c3f40 0x5559823d3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f104a7c9740 1 --2- 172.21.15.98:0/2033420645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5559823c3620 0x5559823d6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.366+0000 7f104a7c9740 1 -- 172.21.15.98:0/2033420645 >> 172.21.15.98:0/2033420645 conn(0x5559823bfb30 msgr2=0x5559823becb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f104a7c9740 1 -- 172.21.15.98:0/2033420645 shutdown_connections 2024-04-25T17:42:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f104a7c9740 1 -- 172.21.15.98:0/2033420645 wait complete. 2024-04-25T17:42:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f104a7c9740 20 client.25516 trim_cache size 0 max 16384 2024-04-25T17:42:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:42:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:42:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:42:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:42:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 -- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f102c0a3e40 msgr2=0x7f102c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f102c0a3e40 0x7f102c0a6300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f103c005fd0 tx=0x7f103c004730 comp rx=0 tx=0).stop 2024-04-25T17:42:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 -- 172.21.15.98:0/2683752712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10401062f0 msgr2=0x7f104019f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10401062f0 0x7f104019f4a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1030009fd0 tx=0x7f103000b870 comp rx=0 tx=0).stop 2024-04-25T17:42:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 -- 172.21.15.98:0/2683752712 shutdown_connections 2024-04-25T17:42:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f102c0a3e40 0x7f102c0a6300 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10401a3820 0x7f10401a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10401062f0 0x7f104019f4a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 --2- 172.21.15.98:0/2683752712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10401018f0 0x7f104019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 -- 172.21.15.98:0/2683752712 >> 172.21.15.98:0/2683752712 conn(0x7f1040077fd0 msgr2=0x7f1040103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.367+0000 7f1047aae640 1 -- 172.21.15.98:0/2683752712 shutdown_connections 2024-04-25T17:42:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:26.368+0000 7f1047aae640 1 -- 172.21.15.98:0/2683752712 wait complete. 2024-04-25T17:42:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 355.0 s, 2728.0 tps, lat 36.750 ms stddev 77.320 2024-04-25T17:42:26.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:26 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:26.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:26 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:26.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:26 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:27.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:27 smithi098 ceph-mon[29192]: pgmap v202: 97 pgs: 97 active+clean; 3.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 33 MiB/s wr, 1.98k op/s 2024-04-25T17:42:27.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:27 smithi203 ceph-mon[32834]: pgmap v202: 97 pgs: 97 active+clean; 3.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 33 MiB/s wr, 1.98k op/s 2024-04-25T17:42:27.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:27 smithi173 ceph-mon[32856]: pgmap v202: 97 pgs: 97 active+clean; 3.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 33 MiB/s wr, 1.98k op/s 2024-04-25T17:42:29.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:29 smithi173 ceph-mon[32856]: pgmap v203: 97 pgs: 97 active+clean; 3.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 MiB/s wr, 1.56k op/s 2024-04-25T17:42:29.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:29 smithi098 ceph-mon[29192]: pgmap v203: 97 pgs: 97 active+clean; 3.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 MiB/s wr, 1.56k op/s 2024-04-25T17:42:29.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:29 smithi203 ceph-mon[32834]: pgmap v203: 97 pgs: 97 active+clean; 3.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 MiB/s wr, 1.56k op/s 2024-04-25T17:42:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 360.0 s, 3490.6 tps, lat 28.535 ms stddev 30.732 2024-04-25T17:42:32.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:32 smithi098 ceph-mon[29192]: pgmap v204: 97 pgs: 97 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.66k op/s 2024-04-25T17:42:32.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:32 smithi203 ceph-mon[32834]: pgmap v204: 97 pgs: 97 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.66k op/s 2024-04-25T17:42:32.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:32 smithi173 ceph-mon[32856]: pgmap v204: 97 pgs: 97 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.66k op/s 2024-04-25T17:42:34.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:34 smithi098 ceph-mon[29192]: pgmap v205: 97 pgs: 97 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 33 MiB/s wr, 1.84k op/s 2024-04-25T17:42:34.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:34 smithi203 ceph-mon[32834]: pgmap v205: 97 pgs: 97 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 33 MiB/s wr, 1.84k op/s 2024-04-25T17:42:34.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:34 smithi173 ceph-mon[32856]: pgmap v205: 97 pgs: 97 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 33 MiB/s wr, 1.84k op/s 2024-04-25T17:42:36.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:36 smithi098 ceph-mon[29192]: pgmap v206: 97 pgs: 97 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.64k op/s 2024-04-25T17:42:36.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:36 smithi203 ceph-mon[32834]: pgmap v206: 97 pgs: 97 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.64k op/s 2024-04-25T17:42:36.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:36 smithi173 ceph-mon[32856]: pgmap v206: 97 pgs: 97 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.64k op/s 2024-04-25T17:42:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 365.0 s, 3717.8 tps, lat 26.966 ms stddev 30.181 2024-04-25T17:42:38.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:38 smithi098 ceph-mon[29192]: pgmap v207: 97 pgs: 97 active+clean; 3.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 35 MiB/s wr, 1.97k op/s 2024-04-25T17:42:38.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:38 smithi203 ceph-mon[32834]: pgmap v207: 97 pgs: 97 active+clean; 3.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 35 MiB/s wr, 1.97k op/s 2024-04-25T17:42:38.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:38 smithi173 ceph-mon[32856]: pgmap v207: 97 pgs: 97 active+clean; 3.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 35 MiB/s wr, 1.97k op/s 2024-04-25T17:42:39.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.492+0000 7f421fd44640 1 -- 172.21.15.98:0/2977385328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4218103220 msgr2=0x7f4218103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:39.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.492+0000 7f421fd44640 1 --2- 172.21.15.98:0/2977385328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4218103220 0x7f4218103600 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4204009900 tx=0x7f420402a480 comp rx=0 tx=0).stop 2024-04-25T17:42:39.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.493+0000 7f421fd44640 1 -- 172.21.15.98:0/2977385328 shutdown_connections 2024-04-25T17:42:39.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.493+0000 7f421fd44640 1 --2- 172.21.15.98:0/2977385328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f421810c560 0x7f421810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.493+0000 7f421fd44640 1 --2- 172.21.15.98:0/2977385328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4218103b40 0x7f421810c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.493+0000 7f421fd44640 1 --2- 172.21.15.98:0/2977385328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4218103220 0x7f4218103600 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.493+0000 7f421fd44640 1 -- 172.21.15.98:0/2977385328 >> 172.21.15.98:0/2977385328 conn(0x7f4218077fd0 msgr2=0x7f42180783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:39.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.493+0000 7f421fd44640 1 -- 172.21.15.98:0/2977385328 shutdown_connections 2024-04-25T17:42:39.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.494+0000 7f421fd44640 1 -- 172.21.15.98:0/2977385328 wait complete. 2024-04-25T17:42:39.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.494+0000 7f421fd44640 1 Processor -- start 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.494+0000 7f421fd44640 1 -- start start 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421fd44640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4218103220 0x7f42181080d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421fd44640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4218103b40 0x7f4218108610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421fd44640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f421810c560 0x7f4218108b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421fd44640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42181121a0 con 0x7f4218103220 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421fd44640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4218112020 con 0x7f4218103b40 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421fd44640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4218112320 con 0x7f421810c560 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421dae0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4218103220 0x7f42181080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421dae0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4218103220 0x7f42181080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55022/0 (socket says 172.21.15.98:55022) 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421dae0640 1 -- 172.21.15.98:0/1659934926 learned_addr learned my addr 172.21.15.98:0/1659934926 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421e2e1640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f421810c560 0x7f4218108b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421d2df640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4218103b40 0x7f4218108610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421dae0640 1 -- 172.21.15.98:0/1659934926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f421810c560 msgr2=0x7f4218108b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421dae0640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f421810c560 0x7f4218108b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421dae0640 1 -- 172.21.15.98:0/1659934926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4218103b40 msgr2=0x7f4218108610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421dae0640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4218103b40 0x7f4218108610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.495+0000 7f421dae0640 1 -- 172.21.15.98:0/1659934926 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42181aafb0 con 0x7f4218103220 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.496+0000 7f421dae0640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4218103220 0x7f42181080d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4204005940 tx=0x7f4204004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.496+0000 7f4216ffd640 1 -- 172.21.15.98:0/1659934926 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420403f030 con 0x7f4218103220 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.496+0000 7f4216ffd640 1 -- 172.21.15.98:0/1659934926 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4204003aa0 con 0x7f4218103220 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.496+0000 7f421fd44640 1 -- 172.21.15.98:0/1659934926 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42181ab240 con 0x7f4218103220 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.496+0000 7f4216ffd640 1 -- 172.21.15.98:0/1659934926 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4204035070 con 0x7f4218103220 2024-04-25T17:42:39.498 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.496+0000 7f421fd44640 1 -- 172.21.15.98:0/1659934926 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42181ab730 con 0x7f4218103220 2024-04-25T17:42:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.499+0000 7f4216ffd640 1 -- 172.21.15.98:0/1659934926 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4204042030 con 0x7f4218103220 2024-04-25T17:42:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.499+0000 7f4222a65740 1 Processor -- start 2024-04-25T17:42:39.499 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.499+0000 7f4222a65740 1 -- start start 2024-04-25T17:42:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f4216ffd640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f42080a3dc0 0x7f42080a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f4222a65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 0x5612c858da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f4222a65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 0x5612c85982c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f4222a65740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c8598800 0x5612c859abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f4222a65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5612c84f2970 con 0x5612c858d620 2024-04-25T17:42:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f4222a65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5612c848f4a0 con 0x5612c858df40 2024-04-25T17:42:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f4222a65740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5612c84ee930 con 0x5612c8598800 2024-04-25T17:42:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f4216ffd640 1 -- 172.21.15.98:0/1659934926 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f42040f0a50 con 0x7f4218103220 2024-04-25T17:42:39.500 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f421e2e1640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f42080a3dc0 0x7f42080a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f421dae0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 0x5612c858da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f421dae0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 0x5612c858da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55030/0 (socket says 172.21.15.98:55030) 2024-04-25T17:42:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f421dae0640 1 -- 172.21.15.98:0/577098357 learned_addr learned my addr 172.21.15.98:0/577098357 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f421d2df640 1 --2- 172.21.15.98:0/577098357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 0x5612c85982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.500+0000 7f421e2e1640 1 --2- 172.21.15.98:0/577098357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c8598800 0x5612c859abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.501+0000 7f421d2df640 1 -- 172.21.15.98:0/577098357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c8598800 msgr2=0x5612c859abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:39.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.501+0000 7f421d2df640 1 --2- 172.21.15.98:0/577098357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c8598800 0x5612c859abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.501+0000 7f421d2df640 1 -- 172.21.15.98:0/577098357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 msgr2=0x5612c858da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.501+0000 7f421d2df640 1 --2- 172.21.15.98:0/577098357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 0x5612c858da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.501+0000 7f421d2df640 1 -- 172.21.15.98:0/577098357 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612c8205210 con 0x5612c858df40 2024-04-25T17:42:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.501+0000 7f421d2df640 1 --2- 172.21.15.98:0/577098357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 0x5612c85982c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f42181098d0 tx=0x7f420c003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e9cdadd3d570dfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.501+0000 7f421e2e1640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f42080a3dc0 0x7f42080a6280 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4210009fd0 tx=0x7f4210009ec0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.501+0000 7f4203fff640 1 -- 172.21.15.98:0/577098357 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c004b20 con 0x5612c858df40 2024-04-25T17:42:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.501+0000 7f4203fff640 1 -- 172.21.15.98:0/577098357 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f420c004cc0 con 0x5612c858df40 2024-04-25T17:42:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.501+0000 7f4203fff640 1 -- 172.21.15.98:0/577098357 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c0126b0 con 0x5612c858df40 2024-04-25T17:42:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.503+0000 7f4222a65740 1 -- 172.21.15.98:0/577098357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 msgr2=0x5612c85982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:39.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.503+0000 7f4222a65740 1 --2- 172.21.15.98:0/577098357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 0x5612c85982c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f42181098d0 tx=0x7f420c003e00 comp rx=0 tx=0).stop 2024-04-25T17:42:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.503+0000 7f4222a65740 1 -- 172.21.15.98:0/577098357 shutdown_connections 2024-04-25T17:42:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.503+0000 7f4222a65740 1 --2- 172.21.15.98:0/577098357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c8598800 0x5612c859abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.503+0000 7f4222a65740 1 --2- 172.21.15.98:0/577098357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 0x5612c85982c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.503+0000 7f4222a65740 1 --2- 172.21.15.98:0/577098357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 0x5612c858da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.503+0000 7f4222a65740 1 -- 172.21.15.98:0/577098357 >> 172.21.15.98:0/577098357 conn(0x5612c8589b30 msgr2=0x5612c8589f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.503+0000 7f4222a65740 1 -- 172.21.15.98:0/577098357 shutdown_connections 2024-04-25T17:42:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.503+0000 7f4222a65740 1 -- 172.21.15.98:0/577098357 wait complete. 2024-04-25T17:42:39.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.504+0000 7f4222a65740 1 -- start start 2024-04-25T17:42:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.504+0000 7f4222a65740 1 Processor -- start 2024-04-25T17:42:39.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.505+0000 7f4222a65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 0x5612c8598150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.505+0000 7f4222a65740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858df40 0x5612c8595440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.505+0000 7f4222a65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c8598800 0x5612c85959a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.505+0000 7f4222a65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5612c859b430 con 0x5612c858d620 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.505+0000 7f4222a65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5612c848d0b0 con 0x5612c8598800 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.505+0000 7f4222a65740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5612c859b5b0 con 0x5612c858df40 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.505+0000 7f4222a65740 10 client.0 _ll_register_callbacks cb 0x5612c847ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.505+0000 7f421e2e1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c8598800 0x5612c85959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.505+0000 7f421e2e1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c8598800 0x5612c85959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:39134/0 (socket says 172.21.15.98:39134) 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.505+0000 7f421e2e1640 1 -- 172.21.15.98:0/99470187 learned_addr learned my addr 172.21.15.98:0/99470187 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.506+0000 7f421dae0640 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 0x5612c8598150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.506+0000 7f421e2e1640 1 -- 172.21.15.98:0/99470187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858df40 msgr2=0x5612c8595440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T17:42:39.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.506+0000 7f421e2e1640 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858df40 0x5612c8595440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.506+0000 7f421e2e1640 1 -- 172.21.15.98:0/99470187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 msgr2=0x5612c8598150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:39.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.506+0000 7f421e2e1640 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 0x5612c8598150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.506+0000 7f421e2e1640 1 -- 172.21.15.98:0/99470187 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612c82d1620 con 0x5612c8598800 2024-04-25T17:42:39.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.506+0000 7f421e2e1640 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c8598800 0x5612c85959a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4210009250 tx=0x7f4210004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52497a1e62bc4352 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:39.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.506+0000 7f42017fa640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:42:39.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f42017fa640 1 -- 172.21.15.98:0/99470187 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4210025070 con 0x5612c8598800 2024-04-25T17:42:39.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f4222a65740 1 -- 172.21.15.98:0/99470187 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f420c004cc0 con 0x5612c8598800 2024-04-25T17:42:39.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f42017fa640 1 -- 172.21.15.98:0/99470187 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f4210009030 con 0x5612c8598800 2024-04-25T17:42:39.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f42017fa640 1 -- 172.21.15.98:0/99470187 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4210030050 con 0x5612c8598800 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f4222a65740 10 client.25541 fetch_fsmap learned FSMap version 36 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f4222a65740 1 -- 172.21.15.98:0/99470187 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5612c85a73f0 con 0x5612c8598800 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f42017fa640 1 -- 172.21.15.98:0/99470187 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f420c004cc0 con 0x5612c8598800 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f42017fa640 1 -- 172.21.15.98:0/99470187 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f421001f040 con 0x5612c8598800 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f4222a65740 10 client.25541 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f4222a65740 10 client.25541 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f4222a65740 20 client.25541 populate_metadata read hostname 'smithi098' 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f4222a65740 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5612c85a8920 0x5612c85aade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f4222a65740 4 client.25541 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.507+0000 7f4222a65740 1 -- 172.21.15.98:0/99470187 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612c8500390 con 0x5612c85a8920 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.508+0000 7f421dae0640 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5612c85a8920 0x5612c85aade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.508+0000 7f42017fa640 10 client.25541 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:39.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.508+0000 7f421dae0640 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5612c85a8920 0x5612c85aade0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c55194664888725f server_cookie=4df1fabcdb42882f in_seq=0 out_seq=0 2024-04-25T17:42:39.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.510+0000 7f42017fa640 1 -- 172.21.15.98:0/99470187 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612c8500390 con 0x5612c85a8920 2024-04-25T17:42:39.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.510+0000 7f42017fa640 10 client.25541 handle_command_reply: tid=0 2024-04-25T17:42:39.510 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.510+0000 7f4222a65740 1 client.25541 shutdown 2024-04-25T17:42:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 1 -- 172.21.15.98:0/99470187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c8598800 msgr2=0x5612c85959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c8598800 0x5612c85959a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4210009250 tx=0x7f4210004730 comp rx=0 tx=0).stop 2024-04-25T17:42:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 1 -- 172.21.15.98:0/99470187 shutdown_connections 2024-04-25T17:42:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5612c85a8920 0x5612c85aade0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c8598800 0x5612c85959a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.511 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f42017fa640 0 client.25541 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858df40 0x5612c8595440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 1 --2- 172.21.15.98:0/99470187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c858d620 0x5612c8598150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 1 -- 172.21.15.98:0/99470187 >> 172.21.15.98:0/99470187 conn(0x5612c8589b30 msgr2=0x5612c858f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 1 -- 172.21.15.98:0/99470187 shutdown_connections 2024-04-25T17:42:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 1 -- 172.21.15.98:0/99470187 wait complete. 2024-04-25T17:42:39.512 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.511+0000 7f4222a65740 20 client.25541 trim_cache size 0 max 16384 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.532+0000 7f4222a65740 1 Processor -- start 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.532+0000 7f4222a65740 1 -- start start 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f4222a65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858d620 0x5612c85a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f4222a65740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858df40 0x5612c8588b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f4222a65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c8598800 0x5612c85b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f4222a65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42181b6af0 con 0x5612c8598800 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f4222a65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f42181b4360 con 0x5612c858d620 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f4222a65740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f42181b21d0 con 0x5612c858df40 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f421dae0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858d620 0x5612c85a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f421dae0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858d620 0x5612c85a42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:39140/0 (socket says 172.21.15.98:39140) 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f421dae0640 1 -- 172.21.15.98:0/2021490546 learned_addr learned my addr 172.21.15.98:0/2021490546 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f421d2df640 1 --2- 172.21.15.98:0/2021490546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858df40 0x5612c8588b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f421dae0640 1 -- 172.21.15.98:0/2021490546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858df40 msgr2=0x5612c8588b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f421dae0640 1 --2- 172.21.15.98:0/2021490546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858df40 0x5612c8588b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f421dae0640 1 -- 172.21.15.98:0/2021490546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c8598800 msgr2=0x5612c85b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f421dae0640 1 --2- 172.21.15.98:0/2021490546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c8598800 0x5612c85b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.533+0000 7f421dae0640 1 -- 172.21.15.98:0/2021490546 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612c8595d10 con 0x5612c858d620 2024-04-25T17:42:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.534+0000 7f421dae0640 1 --2- 172.21.15.98:0/2021490546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858d620 0x5612c85a42f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4204006010 tx=0x7f420402a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d57b2475ba04ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:39.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.534+0000 7f42037fe640 1 -- 172.21.15.98:0/2021490546 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42040d7070 con 0x5612c858d620 2024-04-25T17:42:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.534+0000 7f42037fe640 1 -- 172.21.15.98:0/2021490546 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42040c0da0 con 0x5612c858d620 2024-04-25T17:42:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.534+0000 7f42037fe640 1 -- 172.21.15.98:0/2021490546 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4204049040 con 0x5612c858d620 2024-04-25T17:42:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.535+0000 7f4222a65740 1 -- 172.21.15.98:0/2021490546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858d620 msgr2=0x5612c85a42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.535+0000 7f4222a65740 1 --2- 172.21.15.98:0/2021490546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858d620 0x5612c85a42f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4204006010 tx=0x7f420402a480 comp rx=0 tx=0).stop 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.535+0000 7f4222a65740 1 -- 172.21.15.98:0/2021490546 shutdown_connections 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.535+0000 7f4222a65740 1 --2- 172.21.15.98:0/2021490546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c8598800 0x5612c85b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.535+0000 7f4222a65740 1 --2- 172.21.15.98:0/2021490546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858df40 0x5612c8588b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.535+0000 7f4222a65740 1 --2- 172.21.15.98:0/2021490546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858d620 0x5612c85a42f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.535+0000 7f4222a65740 1 -- 172.21.15.98:0/2021490546 >> 172.21.15.98:0/2021490546 conn(0x5612c8589b30 msgr2=0x5612c8588420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.535+0000 7f4222a65740 1 -- 172.21.15.98:0/2021490546 shutdown_connections 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.536+0000 7f4222a65740 1 -- 172.21.15.98:0/2021490546 wait complete. 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.536+0000 7f4222a65740 1 -- start start 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.536+0000 7f4222a65740 1 Processor -- start 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.536+0000 7f4222a65740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858d620 0x5612c85a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.536+0000 7f4222a65740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 0x5612c859df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.536+0000 7f4222a65740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c8598800 0x5612c859e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.536+0000 7f4222a65740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f42181b2820 con 0x5612c8598800 2024-04-25T17:42:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.536+0000 7f4222a65740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f42181b3c90 con 0x5612c858df40 2024-04-25T17:42:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.536+0000 7f4222a65740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f42181b2ef0 con 0x5612c858d620 2024-04-25T17:42:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.536+0000 7f4222a65740 10 client.0 _ll_register_callbacks cb 0x7f42100134f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:42:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421d2df640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 0x5612c859df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421dae0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858d620 0x5612c85a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421d2df640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 0x5612c859df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:39142/0 (socket says 172.21.15.98:39142) 2024-04-25T17:42:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421dae0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858d620 0x5612c85a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:52572/0 (socket says 172.21.15.98:52572) 2024-04-25T17:42:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421d2df640 1 -- 172.21.15.98:0/2127004824 learned_addr learned my addr 172.21.15.98:0/2127004824 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421e2e1640 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c8598800 0x5612c859e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421dae0640 1 -- 172.21.15.98:0/2127004824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 msgr2=0x5612c859df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421dae0640 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 0x5612c859df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421dae0640 1 -- 172.21.15.98:0/2127004824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c8598800 msgr2=0x5612c859e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421dae0640 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c8598800 0x5612c859e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421dae0640 1 -- 172.21.15.98:0/2127004824 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612c858dd70 con 0x5612c858d620 2024-04-25T17:42:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f421dae0640 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858d620 0x5612c85a0b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f42040c5820 tx=0x7f4204005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8330f0137701b240 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.537+0000 7f42017fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:42:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.538+0000 7f42017fa640 1 -- 172.21.15.98:0/2127004824 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4204049020 con 0x5612c858d620 2024-04-25T17:42:39.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.538+0000 7f42017fa640 1 -- 172.21.15.98:0/2127004824 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f42040b9110 con 0x5612c858d620 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.538+0000 7f4222a65740 1 -- 172.21.15.98:0/2127004824 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42040c0da0 con 0x5612c858d620 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.538+0000 7f42017fa640 1 -- 172.21.15.98:0/2127004824 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f42040d7070 con 0x5612c858d620 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.538+0000 7f4222a65740 10 client.25531 fetch_fsmap learned FSMap version 36 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.538+0000 7f4222a65740 1 -- 172.21.15.98:0/2127004824 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5612c860ed70 con 0x5612c858d620 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.539+0000 7f42017fa640 1 -- 172.21.15.98:0/2127004824 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42040c0da0 con 0x5612c858d620 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.539+0000 7f42017fa640 1 -- 172.21.15.98:0/2127004824 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f420404c060 con 0x5612c858d620 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.539+0000 7f4222a65740 10 client.25531 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.539+0000 7f4222a65740 10 client.25531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.539+0000 7f4222a65740 20 client.25531 populate_metadata read hostname 'smithi098' 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.539+0000 7f4222a65740 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5612c860f3e0 0x5612c8611800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.539+0000 7f4222a65740 4 client.25531 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.539+0000 7f4222a65740 1 -- 172.21.15.98:0/2127004824 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5612c84fa200 con 0x5612c860f3e0 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.539+0000 7f421d2df640 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5612c860f3e0 0x5612c8611800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.540+0000 7f421d2df640 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5612c860f3e0 0x5612c8611800 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e0a0a1ef2157c7c server_cookie=4352c286d2463 in_seq=0 out_seq=0 2024-04-25T17:42:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.540+0000 7f42017fa640 10 client.25531 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f42017fa640 1 -- 172.21.15.98:0/2127004824 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5612c84fa200 con 0x5612c860f3e0 2024-04-25T17:42:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f42017fa640 10 client.25531 handle_command_reply: tid=0 2024-04-25T17:42:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 client.25531 shutdown 2024-04-25T17:42:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 -- 172.21.15.98:0/2127004824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858d620 msgr2=0x5612c85a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858d620 0x5612c85a0b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f42040c5820 tx=0x7f4204005c00 comp rx=0 tx=0).stop 2024-04-25T17:42:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 -- 172.21.15.98:0/2127004824 shutdown_connections 2024-04-25T17:42:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5612c860f3e0 0x5612c8611800 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612c8598800 0x5612c859e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612c858df40 0x5612c859df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f42017fa640 0 client.25531 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 --2- 172.21.15.98:0/2127004824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612c858d620 0x5612c85a0b70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 -- 172.21.15.98:0/2127004824 >> 172.21.15.98:0/2127004824 conn(0x5612c8589b30 msgr2=0x5612c8588cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 -- 172.21.15.98:0/2127004824 shutdown_connections 2024-04-25T17:42:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 1 -- 172.21.15.98:0/2127004824 wait complete. 2024-04-25T17:42:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.555+0000 7f4222a65740 20 client.25531 trim_cache size 0 max 16384 2024-04-25T17:42:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:42:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:42:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:42:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 -- 172.21.15.98:0/1659934926 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f42080a3dc0 msgr2=0x7f42080a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f42080a3dc0 0x7f42080a6280 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4210009fd0 tx=0x7f4210009ec0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 -- 172.21.15.98:0/1659934926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4218103220 msgr2=0x7f42181080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4218103220 0x7f42181080d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4204005940 tx=0x7f4204004270 comp rx=0 tx=0).stop 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 -- 172.21.15.98:0/1659934926 shutdown_connections 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f42080a3dc0 0x7f42080a6280 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f421810c560 0x7f4218108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4218103b40 0x7f4218108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 --2- 172.21.15.98:0/1659934926 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4218103220 0x7f42181080d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 -- 172.21.15.98:0/1659934926 >> 172.21.15.98:0/1659934926 conn(0x7f4218077fd0 msgr2=0x7f4218104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 -- 172.21.15.98:0/1659934926 shutdown_connections 2024-04-25T17:42:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:39.556+0000 7f421fd44640 1 -- 172.21.15.98:0/1659934926 wait complete. 2024-04-25T17:42:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:40 smithi098 ceph-mon[29192]: pgmap v208: 97 pgs: 97 active+clean; 3.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 738 op/s 2024-04-25T17:42:40.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:40 smithi203 ceph-mon[32834]: pgmap v208: 97 pgs: 97 active+clean; 3.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 738 op/s 2024-04-25T17:42:40.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:40 smithi173 ceph-mon[32856]: pgmap v208: 97 pgs: 97 active+clean; 3.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 738 op/s 2024-04-25T17:42:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 370.0 s, 3111.8 tps, lat 32.080 ms stddev 36.614 2024-04-25T17:42:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:42 smithi098 ceph-mon[29192]: pgmap v209: 97 pgs: 97 active+clean; 3.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 12 MiB/s wr, 797 op/s 2024-04-25T17:42:42.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:42 smithi203 ceph-mon[32834]: pgmap v209: 97 pgs: 97 active+clean; 3.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 12 MiB/s wr, 797 op/s 2024-04-25T17:42:42.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:42 smithi173 ceph-mon[32856]: pgmap v209: 97 pgs: 97 active+clean; 3.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 12 MiB/s wr, 797 op/s 2024-04-25T17:42:43.817 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 13.1 2024-04-25T17:42:43.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:42:44.184 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:42:44.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:44 smithi098 ceph-mon[29192]: pgmap v210: 97 pgs: 97 active+clean; 3.7 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 960 op/s 2024-04-25T17:42:44.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:44 smithi203 ceph-mon[32834]: pgmap v210: 97 pgs: 97 active+clean; 3.7 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 960 op/s 2024-04-25T17:42:44.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:44 smithi173 ceph-mon[32856]: pgmap v210: 97 pgs: 97 active+clean; 3.7 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 13 MiB/s wr, 960 op/s 2024-04-25T17:42:46.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.129+0000 7f6056bcf700 1 -- 172.21.15.98:0/887334494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6050101930 msgr2=0x7f6050101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:46.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.129+0000 7f6056bcf700 1 --2- 172.21.15.98:0/887334494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6050101930 0x7f6050101db0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f603c009990 tx=0x7f603c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.129+0000 7f6056bcf700 1 -- 172.21.15.98:0/887334494 shutdown_connections 2024-04-25T17:42:46.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.129+0000 7f6056bcf700 1 --2- 172.21.15.98:0/887334494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6050107980 0x7f6050107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.129+0000 7f6056bcf700 1 --2- 172.21.15.98:0/887334494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60501022f0 0x7f605010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.129+0000 7f6056bcf700 1 --2- 172.21.15.98:0/887334494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6050101930 0x7f6050101db0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.129+0000 7f6056bcf700 1 -- 172.21.15.98:0/887334494 >> 172.21.15.98:0/887334494 conn(0x7f60500788b0 msgr2=0x7f6050078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:46.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.129+0000 7f6056bcf700 1 -- 172.21.15.98:0/887334494 shutdown_connections 2024-04-25T17:42:46.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.129+0000 7f6056bcf700 1 -- 172.21.15.98:0/887334494 wait complete. 2024-04-25T17:42:46.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.130+0000 7f6056bcf700 1 Processor -- start 2024-04-25T17:42:46.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.130+0000 7f6056bcf700 1 -- start start 2024-04-25T17:42:46.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.131+0000 7f6056bcf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6050101930 0x7f6050197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:46.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.131+0000 7f6056bcf700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60501022f0 0x7f6050198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:46.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.131+0000 7f605496b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6050101930 0x7f6050197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:46.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f605496b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6050101930 0x7f6050197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33340/0 (socket says 172.21.15.98:33340) 2024-04-25T17:42:46.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f605496b700 1 -- 172.21.15.98:0/3650732606 learned_addr learned my addr 172.21.15.98:0/3650732606 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:46.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f6056bcf700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6050107980 0x7f605019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:46.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6050111af0 con 0x7f6050101930 2024-04-25T17:42:46.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6050111970 con 0x7f6050107980 2024-04-25T17:42:46.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6050111c70 con 0x7f60501022f0 2024-04-25T17:42:46.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f604ffff700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60501022f0 0x7f6050198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:46.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f605496b700 1 -- 172.21.15.98:0/3650732606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60501022f0 msgr2=0x7f6050198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:46.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f605496b700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60501022f0 0x7f6050198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f605496b700 1 -- 172.21.15.98:0/3650732606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6050107980 msgr2=0x7f605019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:46.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f605516c700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6050107980 0x7f605019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:46.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f605496b700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6050107980 0x7f605019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.132+0000 7f605496b700 1 -- 172.21.15.98:0/3650732606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605019cf80 con 0x7f6050101930 2024-04-25T17:42:46.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.133+0000 7f605496b700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6050101930 0x7f6050197fc0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f604400e9c0 tx=0x7f604400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:46.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.133+0000 7f604dffb700 1 -- 172.21.15.98:0/3650732606 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60440159e0 con 0x7f6050101930 2024-04-25T17:42:46.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.133+0000 7f604dffb700 1 -- 172.21.15.98:0/3650732606 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6044004ce0 con 0x7f6050101930 2024-04-25T17:42:46.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.133+0000 7f604dffb700 1 -- 172.21.15.98:0/3650732606 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6044005390 con 0x7f6050101930 2024-04-25T17:42:46.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.133+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605019d270 con 0x7f6050101930 2024-04-25T17:42:46.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.133+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605006afb0 con 0x7f6050101930 2024-04-25T17:42:46.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.134+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6050103050 con 0x7f6050101930 2024-04-25T17:42:46.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.136+0000 7f604dffb700 1 -- 172.21.15.98:0/3650732606 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6044005530 con 0x7f6050101930 2024-04-25T17:42:46.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.137+0000 7f604dffb700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f60400a3eb0 0x7f60400a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:46.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.138+0000 7f604ffff700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f60400a3eb0 0x7f60400a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:46.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.138+0000 7f604dffb700 1 -- 172.21.15.98:0/3650732606 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f60440d4e80 con 0x7f6050101930 2024-04-25T17:42:46.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.138+0000 7f604ffff700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f60400a3eb0 0x7f60400a6370 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f603c00b2e0 tx=0x7f603c005960 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:46.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.142+0000 7f604dffb700 1 -- 172.21.15.98:0/3650732606 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f604401a030 con 0x7f6050101930 2024-04-25T17:42:46.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:46 smithi098 ceph-mon[29192]: pgmap v211: 97 pgs: 97 active+clean; 3.7 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 8.6 MiB/s wr, 643 op/s 2024-04-25T17:42:46.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:46 smithi203 ceph-mon[32834]: pgmap v211: 97 pgs: 97 active+clean; 3.7 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 8.6 MiB/s wr, 643 op/s 2024-04-25T17:42:46.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:46 smithi173 ceph-mon[32856]: pgmap v211: 97 pgs: 97 active+clean; 3.7 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 8.6 MiB/s wr, 643 op/s 2024-04-25T17:42:46.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.537+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6050069730 con 0x7f6050101930 2024-04-25T17:42:46.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.537+0000 7f604dffb700 1 -- 172.21.15.98:0/3650732606 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11880 (secure 0 0 0) 0x7f60440a1540 con 0x7f6050101930 2024-04-25T17:42:46.538 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:42:46.538 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:42:24.357579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:42:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f60400a3eb0 msgr2=0x7f60400a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f60400a3eb0 0x7f60400a6370 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f603c00b2e0 tx=0x7f603c005960 comp rx=0 tx=0).stop 2024-04-25T17:42:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6050101930 msgr2=0x7f6050197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:46.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6050101930 0x7f6050197fc0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f604400e9c0 tx=0x7f604400ecd0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 shutdown_connections 2024-04-25T17:42:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f60400a3eb0 0x7f60400a6370 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6050107980 0x7f605019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6050101930 0x7f6050197fc0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 --2- 172.21.15.98:0/3650732606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60501022f0 0x7f6050198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 >> 172.21.15.98:0/3650732606 conn(0x7f60500788b0 msgr2=0x7f60500ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 shutdown_connections 2024-04-25T17:42:46.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:46.540+0000 7f6056bcf700 1 -- 172.21.15.98:0/3650732606 wait complete. 2024-04-25T17:42:46.543 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 36 2024-04-25T17:42:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 375.0 s, 3037.0 tps, lat 32.898 ms stddev 35.586 2024-04-25T17:42:47.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:47 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3650732606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:42:47.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:47 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3650732606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:42:47.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:47 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3650732606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:42:48.110 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 73.1 --await-for 42.0' 2024-04-25T17:42:48.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 73.1 --await-for 42.0 2024-04-25T17:42:48.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:48 smithi098 ceph-mon[29192]: pgmap v212: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-25T17:42:48.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:48 smithi203 ceph-mon[32834]: pgmap v212: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-25T17:42:48.477 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:42:48.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:48 smithi173 ceph-mon[32856]: pgmap v212: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-25T17:42:49.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:49 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:49.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:49 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:49.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:49 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:50.378 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:50 smithi098 ceph-mon[29192]: pgmap v213: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.9 MiB/s wr, 678 op/s 2024-04-25T17:42:50.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:50 smithi203 ceph-mon[32834]: pgmap v213: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.9 MiB/s wr, 678 op/s 2024-04-25T17:42:50.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:50 smithi173 ceph-mon[32856]: pgmap v213: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.9 MiB/s wr, 678 op/s 2024-04-25T17:42:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.543+0000 7f8094b38700 1 -- 172.21.15.98:0/3607450867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8090108bc0 msgr2=0x7f8090108fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.543+0000 7f8094b38700 1 --2- 172.21.15.98:0/3607450867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8090108bc0 0x7f8090108fa0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8080009920 tx=0x7f8080009c30 comp rx=0 tx=0).stop 2024-04-25T17:42:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.544+0000 7f8094b38700 1 -- 172.21.15.98:0/3607450867 shutdown_connections 2024-04-25T17:42:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.544+0000 7f8094b38700 1 --2- 172.21.15.98:0/3607450867 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80901019f0 0x7f809010f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.544+0000 7f8094b38700 1 --2- 172.21.15.98:0/3607450867 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8090108bc0 0x7f8090108fa0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.544+0000 7f8094b38700 1 --2- 172.21.15.98:0/3607450867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8090101030 0x7f80901014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.544+0000 7f8094b38700 1 -- 172.21.15.98:0/3607450867 >> 172.21.15.98:0/3607450867 conn(0x7f8090077fe0 msgr2=0x7f80900ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.544+0000 7f8094b38700 1 -- 172.21.15.98:0/3607450867 shutdown_connections 2024-04-25T17:42:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.544+0000 7f8094b38700 1 -- 172.21.15.98:0/3607450867 wait complete. 2024-04-25T17:42:50.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.545+0000 7f8094b38700 1 Processor -- start 2024-04-25T17:42:50.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.545+0000 7f8094b38700 1 -- start start 2024-04-25T17:42:50.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.545+0000 7f8094b38700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8090101030 0x7f80901a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.545+0000 7f8094b38700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80901019f0 0x7f80901a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.545+0000 7f8094b38700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8090108bc0 0x7f809019fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.545+0000 7f8094b38700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80901024d0 con 0x7f80901019f0 2024-04-25T17:42:50.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.545+0000 7f8094b38700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8090102350 con 0x7f8090108bc0 2024-04-25T17:42:50.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.545+0000 7f8094b38700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8090102650 con 0x7f8090101030 2024-04-25T17:42:50.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80901019f0 0x7f80901a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80901019f0 0x7f80901a15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33358/0 (socket says 172.21.15.98:33358) 2024-04-25T17:42:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8090101030 0x7f80901a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808dd9b700 1 -- 172.21.15.98:0/2327469522 learned_addr learned my addr 172.21.15.98:0/2327469522 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8090101030 0x7f80901a1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53604/0 (socket says 172.21.15.98:53604) 2024-04-25T17:42:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808ed9d700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8090108bc0 0x7f809019fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808e59c700 1 -- 172.21.15.98:0/2327469522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8090108bc0 msgr2=0x7f809019fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808e59c700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8090108bc0 0x7f809019fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808e59c700 1 -- 172.21.15.98:0/2327469522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80901019f0 msgr2=0x7f80901a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808e59c700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80901019f0 0x7f80901a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.546+0000 7f808e59c700 1 -- 172.21.15.98:0/2327469522 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80901a0520 con 0x7f8090101030 2024-04-25T17:42:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.547+0000 7f808e59c700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8090101030 0x7f80901a1080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8080000c00 tx=0x7f808000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.547+0000 7f807f7fe700 1 -- 172.21.15.98:0/2327469522 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080027070 con 0x7f8090101030 2024-04-25T17:42:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.547+0000 7f8094b38700 1 -- 172.21.15.98:0/2327469522 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80901a07b0 con 0x7f8090101030 2024-04-25T17:42:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.547+0000 7f807f7fe700 1 -- 172.21.15.98:0/2327469522 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8080005570 con 0x7f8090101030 2024-04-25T17:42:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.547+0000 7f807f7fe700 1 -- 172.21.15.98:0/2327469522 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080004930 con 0x7f8090101030 2024-04-25T17:42:50.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.547+0000 7f8094b38700 1 -- 172.21.15.98:0/2327469522 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80901a80c0 con 0x7f8090101030 2024-04-25T17:42:50.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.549+0000 7f807f7fe700 1 -- 172.21.15.98:0/2327469522 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f808002a070 con 0x7f8090101030 2024-04-25T17:42:50.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.550+0000 7f807f7fe700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80780a3ed0 0x7f80780a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.550+0000 7f807f7fe700 1 -- 172.21.15.98:0/2327469522 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f80800d1080 con 0x7f8090101030 2024-04-25T17:42:50.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.550+0000 7f808dd9b700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80780a3ed0 0x7f80780a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.551+0000 7f808dd9b700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80780a3ed0 0x7f80780a6390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8090105060 tx=0x7f8084009310 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:50.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.552+0000 7f809f3c1b80 1 Processor -- start 2024-04-25T17:42:50.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.552+0000 7f809f3c1b80 1 -- start start 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121da80 0x55e72121de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721228d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e72122b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e721170ec0 con 0x55e721229270 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e721117af0 con 0x55e72121da80 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e72116ce80 con 0x55e72121e3a0 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121da80 0x55e72121de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121da80 0x55e72121de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:44158/0 (socket says 172.21.15.98:44158) 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808dd9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721228d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808e59c700 1 -- 172.21.15.98:0/4041426841 learned_addr learned my addr 172.21.15.98:0/4041426841 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808ed9d700 1 --2- 172.21.15.98:0/4041426841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e72122b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808e59c700 1 -- 172.21.15.98:0/4041426841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 msgr2=0x55e721228d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:50.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808e59c700 1 --2- 172.21.15.98:0/4041426841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721228d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808e59c700 1 -- 172.21.15.98:0/4041426841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 msgr2=0x55e72122b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:50.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808e59c700 1 --2- 172.21.15.98:0/4041426841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e72122b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808e59c700 1 -- 172.21.15.98:0/4041426841 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7210bea60 con 0x55e72121da80 2024-04-25T17:42:50.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f808e59c700 1 --2- 172.21.15.98:0/4041426841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121da80 0x55e72121de60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f80800994b0 tx=0x7f8080009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=677aa6b5acc4c35a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:50.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.553+0000 7f807cff9700 1 -- 172.21.15.98:0/4041426841 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80800b6070 con 0x55e72121da80 2024-04-25T17:42:50.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.554+0000 7f807cff9700 1 -- 172.21.15.98:0/4041426841 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80800bd040 con 0x55e72121da80 2024-04-25T17:42:50.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.554+0000 7f807cff9700 1 -- 172.21.15.98:0/4041426841 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808000b680 con 0x55e72121da80 2024-04-25T17:42:50.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.554+0000 7f809f3c1b80 1 -- 172.21.15.98:0/4041426841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121da80 msgr2=0x55e72121de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:50.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.554+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/4041426841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121da80 0x55e72121de60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f80800994b0 tx=0x7f8080009520 comp rx=0 tx=0).stop 2024-04-25T17:42:50.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.554+0000 7f809f3c1b80 1 -- 172.21.15.98:0/4041426841 shutdown_connections 2024-04-25T17:42:50.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.554+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/4041426841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e72122b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.554+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/4041426841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121da80 0x55e72121de60 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f80800994b0 tx=0x7f8080009520 comp rx=0 tx=0).stop 2024-04-25T17:42:50.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.554+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/4041426841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721228d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.555+0000 7f809f3c1b80 1 -- 172.21.15.98:0/4041426841 >> 172.21.15.98:0/4041426841 conn(0x55e72114b820 msgr2=0x55e721219f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.555+0000 7f809f3c1b80 1 -- 172.21.15.98:0/4041426841 shutdown_connections 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.555+0000 7f809f3c1b80 1 -- 172.21.15.98:0/4041426841 wait complete. 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.555+0000 7f809f3c1b80 1 -- start start 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f809f3c1b80 1 Processor -- start 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721228c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e721229270 0x55e721225f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7212264f0 0x55e721226950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e721117af0 con 0x55e7212264f0 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e72110fd60 con 0x55e721229270 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e72122bba0 con 0x55e72121e3a0 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f809f3c1b80 10 client.0 _ll_register_callbacks cb 0x55e721109ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f808e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721228c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f808ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7212264f0 0x55e721226950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f808e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721228c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53620/0 (socket says 172.21.15.98:53620) 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f808ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7212264f0 0x55e721226950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33386/0 (socket says 172.21.15.98:33386) 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f808e59c700 1 -- 172.21.15.98:0/21049162 learned_addr learned my addr 172.21.15.98:0/21049162 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.556+0000 7f808dd9b700 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e721229270 0x55e721225f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f808ed9d700 1 -- 172.21.15.98:0/21049162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 msgr2=0x55e721228c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f808ed9d700 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721228c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f808ed9d700 1 -- 172.21.15.98:0/21049162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e721229270 msgr2=0x55e721225f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f808ed9d700 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e721229270 0x55e721225f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f808ed9d700 1 -- 172.21.15.98:0/21049162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e721128400 con 0x55e7212264f0 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f808e59c700 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721228c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f808ed9d700 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7212264f0 0x55e721226950 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8088009fd0 tx=0x7f8088004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4057831d616505fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f8075ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f8075ffb700 1 -- 172.21.15.98:0/21049162 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808801a070 con 0x55e7212264f0 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f8075ffb700 1 -- 172.21.15.98:0/21049162 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f8088004bb0 con 0x55e7212264f0 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.557+0000 7f809f3c1b80 1 -- 172.21.15.98:0/21049162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80800bd040 con 0x55e7212264f0 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f8075ffb700 1 -- 172.21.15.98:0/21049162 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8088025050 con 0x55e7212264f0 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f809f3c1b80 10 client.16029 fetch_fsmap learned FSMap version 37 2024-04-25T17:42:50.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f809f3c1b80 1 -- 172.21.15.98:0/21049162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e721236910 con 0x55e7212264f0 2024-04-25T17:42:50.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f8075ffb700 1 -- 172.21.15.98:0/21049162 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8088004db0 con 0x55e7212264f0 2024-04-25T17:42:50.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f8075ffb700 1 -- 172.21.15.98:0/21049162 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f80880166d0 con 0x55e7212264f0 2024-04-25T17:42:50.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f809f3c1b80 10 client.16029 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T17:42:50.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f809f3c1b80 10 client.16029 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f809f3c1b80 20 client.16029 populate_metadata read hostname 'smithi098' 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e721238f40 0x55e72123b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f809f3c1b80 4 client.16029 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.558+0000 7f809f3c1b80 1 -- 172.21.15.98:0/21049162 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f80800bd040 con 0x55e721238f40 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.559+0000 7f808e59c700 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e721238f40 0x55e72123b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.560+0000 7f808e59c700 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e721238f40 0x55e72123b360 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f465740f0fbb0629 server_cookie=83ac8df6dc887f5c in_seq=0 out_seq=0 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.560+0000 7f8075ffb700 10 client.16029 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f8075ffb700 1 -- 172.21.15.98:0/21049162 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f80800bd040 con 0x55e721238f40 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f8075ffb700 10 client.16029 handle_command_reply: tid=0 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 client.16029 shutdown 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 -- 172.21.15.98:0/21049162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7212264f0 msgr2=0x55e721226950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7212264f0 0x55e721226950 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8088009fd0 tx=0x7f8088004040 comp rx=0 tx=0).stop 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 -- 172.21.15.98:0/21049162 shutdown_connections 2024-04-25T17:42:50.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e721238f40 0x55e72123b360 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f8075ffb700 0 client.16029 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:50.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721228c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e7212264f0 0x55e721226950 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/21049162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e721229270 0x55e721225f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 -- 172.21.15.98:0/21049162 >> 172.21.15.98:0/21049162 conn(0x55e72114b820 msgr2=0x55e721219b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:50.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 -- 172.21.15.98:0/21049162 shutdown_connections 2024-04-25T17:42:50.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 1 -- 172.21.15.98:0/21049162 wait complete. 2024-04-25T17:42:50.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.561+0000 7f809f3c1b80 20 client.16029 trim_cache size 0 max 16384 2024-04-25T17:42:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f809f3c1b80 1 Processor -- start 2024-04-25T17:42:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f809f3c1b80 1 -- start start 2024-04-25T17:42:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721224ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7212264f0 0x55e7212347c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e721240bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80901b1d90 con 0x55e721229270 2024-04-25T17:42:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f80901af590 con 0x55e7212264f0 2024-04-25T17:42:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80901ad400 con 0x55e72121e3a0 2024-04-25T17:42:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f808ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e721240bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f808ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e721240bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33394/0 (socket says 172.21.15.98:33394) 2024-04-25T17:42:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f808e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721224ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f808ed9d700 1 -- 172.21.15.98:0/647065864 learned_addr learned my addr 172.21.15.98:0/647065864 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.587+0000 7f808dd9b700 1 --2- 172.21.15.98:0/647065864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7212264f0 0x55e7212347c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.588+0000 7f808ed9d700 1 -- 172.21.15.98:0/647065864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 msgr2=0x55e721224ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.588+0000 7f808ed9d700 1 --2- 172.21.15.98:0/647065864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721224ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.588+0000 7f808ed9d700 1 -- 172.21.15.98:0/647065864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7212264f0 msgr2=0x55e7212347c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.588+0000 7f808ed9d700 1 --2- 172.21.15.98:0/647065864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7212264f0 0x55e7212347c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.588+0000 7f808ed9d700 1 -- 172.21.15.98:0/647065864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e72123b9c0 con 0x55e721229270 2024-04-25T17:42:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.588+0000 7f808dd9b700 1 --2- 172.21.15.98:0/647065864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7212264f0 0x55e7212347c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:42:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.588+0000 7f808ed9d700 1 --2- 172.21.15.98:0/647065864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e721240bf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8088017ee0 tx=0x7f8088017f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad91d6d3cd3a279 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.588+0000 7f8077fff700 1 -- 172.21.15.98:0/647065864 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808801a040 con 0x55e721229270 2024-04-25T17:42:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.588+0000 7f8077fff700 1 -- 172.21.15.98:0/647065864 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8088004870 con 0x55e721229270 2024-04-25T17:42:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.588+0000 7f8077fff700 1 -- 172.21.15.98:0/647065864 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088007740 con 0x55e721229270 2024-04-25T17:42:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.589+0000 7f809f3c1b80 1 -- 172.21.15.98:0/647065864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 msgr2=0x55e721240bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.589+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/647065864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e721240bf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8088017ee0 tx=0x7f8088017f10 comp rx=0 tx=0).stop 2024-04-25T17:42:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.590+0000 7f809f3c1b80 1 -- 172.21.15.98:0/647065864 shutdown_connections 2024-04-25T17:42:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.590+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/647065864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e72121e3a0 0x55e721224ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.590+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/647065864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e721240bf0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.590+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/647065864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e7212264f0 0x55e7212347c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.590+0000 7f809f3c1b80 1 -- 172.21.15.98:0/647065864 >> 172.21.15.98:0/647065864 conn(0x55e72114b820 msgr2=0x55e721228020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.590+0000 7f809f3c1b80 1 -- 172.21.15.98:0/647065864 shutdown_connections 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.590+0000 7f809f3c1b80 1 -- 172.21.15.98:0/647065864 wait complete. 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.590+0000 7f809f3c1b80 1 -- start start 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.590+0000 7f809f3c1b80 1 Processor -- start 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121e3a0 0x55e72122dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7212264f0 0x55e72122dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f809f3c1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e72129f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80901ada50 con 0x55e721229270 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f80901aeec0 con 0x55e72121e3a0 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f809f3c1b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80901ae120 con 0x55e7212264f0 2024-04-25T17:42:50.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f808dd9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7212264f0 0x55e72122dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f808e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121e3a0 0x55e72122dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f808dd9b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7212264f0 0x55e72122dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53636/0 (socket says 172.21.15.98:53636) 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f808dd9b700 1 -- 172.21.15.98:0/2299209912 learned_addr learned my addr 172.21.15.98:0/2299209912 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f809f3c1b80 10 client.0 _ll_register_callbacks cb 0x55e721109ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f808ed9d700 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e72129f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f808dd9b700 1 -- 172.21.15.98:0/2299209912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121e3a0 msgr2=0x55e72122dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f808dd9b700 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121e3a0 0x55e72122dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f808dd9b700 1 -- 172.21.15.98:0/2299209912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 msgr2=0x55e72129f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f808dd9b700 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e72129f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.591+0000 7f808dd9b700 1 -- 172.21.15.98:0/2299209912 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e72129f9b0 con 0x55e7212264f0 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f808dd9b700 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7212264f0 0x55e72122dff0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8084009b00 tx=0x7f80840047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee47bf9317c9600b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f8075ffb700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:42:50.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f8075ffb700 1 -- 172.21.15.98:0/2299209912 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808402d020 con 0x55e7212264f0 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f8075ffb700 1 -- 172.21.15.98:0/2299209912 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f8084029440 con 0x55e7212264f0 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f809f3c1b80 1 -- 172.21.15.98:0/2299209912 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8088004870 con 0x55e7212264f0 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f8075ffb700 1 -- 172.21.15.98:0/2299209912 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8084038050 con 0x55e7212264f0 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f809f3c1b80 10 client.25555 fetch_fsmap learned FSMap version 37 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f809f3c1b80 1 -- 172.21.15.98:0/2299209912 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e7212a0b40 con 0x55e7212264f0 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f8075ffb700 1 -- 172.21.15.98:0/2299209912 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8088004870 con 0x55e7212264f0 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f8075ffb700 1 -- 172.21.15.98:0/2299209912 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f808401c7d0 con 0x55e7212264f0 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.592+0000 7f809f3c1b80 10 client.25555 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.593+0000 7f809f3c1b80 10 client.25555 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.593+0000 7f809f3c1b80 20 client.25555 populate_metadata read hostname 'smithi098' 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.593+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e7212a13d0 0x55e7212a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.593+0000 7f809f3c1b80 4 client.25555 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 73.1, "await_for": 42.0}] 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.593+0000 7f809f3c1b80 1 -- 172.21.15.98:0/2299209912 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 73.1, "await_for": 42.0}) -- 0x55e72117e6b0 con 0x55e7212a13d0 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.593+0000 7f808e59c700 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e7212a13d0 0x55e7212a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:50.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.593+0000 7f8075ffb700 10 client.25555 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:50.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:42:50.593+0000 7f808e59c700 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e7212a13d0 0x55e7212a37f0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf5721ad856ee9d2 server_cookie=eddcade12ec82c29 in_seq=0 out_seq=0 2024-04-25T17:42:51.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 380.0 s, 1945.0 tps, lat 42.131 ms stddev 47.482 2024-04-25T17:42:52.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:52 smithi098 ceph-mon[29192]: pgmap v214: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.1 MiB/s wr, 754 op/s 2024-04-25T17:42:52.524 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:52 smithi173 ceph-mon[32856]: pgmap v214: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.1 MiB/s wr, 754 op/s 2024-04-25T17:42:52.525 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:52 smithi203 ceph-mon[32834]: pgmap v214: 97 pgs: 97 active+clean; 3.8 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.1 MiB/s wr, 754 op/s 2024-04-25T17:42:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.651+0000 7fbc0a74a640 1 -- 172.21.15.98:0/3460704048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc041021d0 msgr2=0x7fbc041025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.651+0000 7fbc0a74a640 1 --2- 172.21.15.98:0/3460704048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc041021d0 0x7fbc041025b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009900 tx=0x7fbbf402a410 comp rx=0 tx=0).stop 2024-04-25T17:42:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.651+0000 7fbc0a74a640 1 -- 172.21.15.98:0/3460704048 shutdown_connections 2024-04-25T17:42:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.651+0000 7fbc0a74a640 1 --2- 172.21.15.98:0/3460704048 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc041072a0 0x7fbc0410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.651+0000 7fbc0a74a640 1 --2- 172.21.15.98:0/3460704048 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc04102b80 0x7fbc04106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.651+0000 7fbc0a74a640 1 --2- 172.21.15.98:0/3460704048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc041021d0 0x7fbc041025b0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.651+0000 7fbc0a74a640 1 -- 172.21.15.98:0/3460704048 >> 172.21.15.98:0/3460704048 conn(0x7fbc040793e0 msgr2=0x7fbc040797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.651+0000 7fbc0a74a640 1 -- 172.21.15.98:0/3460704048 shutdown_connections 2024-04-25T17:42:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.651+0000 7fbc0a74a640 1 -- 172.21.15.98:0/3460704048 wait complete. 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.652+0000 7fbc0a74a640 1 Processor -- start 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.652+0000 7fbc0a74a640 1 -- start start 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.652+0000 7fbc0a74a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc041021d0 0x7fbc04074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.652+0000 7fbc0a74a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc041072a0 0x7fbc04074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.652+0000 7fbc0a74a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc04074cd0 0x7fbc041a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.652+0000 7fbc0a74a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc041121e0 con 0x7fbc041072a0 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.652+0000 7fbc0a74a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc04112060 con 0x7fbc04074cd0 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.652+0000 7fbc0a74a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc04112360 con 0x7fbc041021d0 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc041072a0 0x7fbc04074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc041072a0 0x7fbc04074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33432/0 (socket says 172.21.15.98:33432) 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc037fe640 1 -- 172.21.15.98:0/1978817018 learned_addr learned my addr 172.21.15.98:0/1978817018 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc03fff640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc041021d0 0x7fbc04074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc08ce7640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc04074cd0 0x7fbc041a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc037fe640 1 -- 172.21.15.98:0/1978817018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc041021d0 msgr2=0x7fbc04074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc037fe640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc041021d0 0x7fbc04074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc037fe640 1 -- 172.21.15.98:0/1978817018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc04074cd0 msgr2=0x7fbc041a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc037fe640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc04074cd0 0x7fbc041a6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc037fe640 1 -- 172.21.15.98:0/1978817018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc041a6f50 con 0x7fbc041072a0 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc037fe640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc041072a0 0x7fbc04074790 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800d770 tx=0x7fbbf800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc017fa640 1 -- 172.21.15.98:0/1978817018 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8017070 con 0x7fbc041072a0 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc017fa640 1 -- 172.21.15.98:0/1978817018 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf8009c30 con 0x7fbc041072a0 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc0a74a640 1 -- 172.21.15.98:0/1978817018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc041a7240 con 0x7fbc041072a0 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc017fa640 1 -- 172.21.15.98:0/1978817018 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf800f620 con 0x7fbc041072a0 2024-04-25T17:42:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.653+0000 7fbc0a74a640 1 -- 172.21.15.98:0/1978817018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc041a7790 con 0x7fbc041072a0 2024-04-25T17:42:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.655+0000 7fbc017fa640 1 -- 172.21.15.98:0/1978817018 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbbf801a070 con 0x7fbc041072a0 2024-04-25T17:42:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc0d45c740 1 Processor -- start 2024-04-25T17:42:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc017fa640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbbec0a3df0 0x7fbbec0a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc0d45c740 1 -- start start 2024-04-25T17:42:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55d620 0x55f93a55da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a55df40 0x55f93a5682c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a568800 0x55f93a56abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f93a4c2970 con 0x55f93a55d620 2024-04-25T17:42:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f93a45f4a0 con 0x55f93a568800 2024-04-25T17:42:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f93a4be930 con 0x55f93a55df40 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc08ce7640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbbec0a3df0 0x7fbbec0a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc017fa640 1 -- 172.21.15.98:0/1978817018 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fbbf80d0250 con 0x7fbc041072a0 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc03fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55d620 0x55f93a55da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc03fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55d620 0x55f93a55da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33444/0 (socket says 172.21.15.98:33444) 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc03fff640 1 -- 172.21.15.98:0/3649244932 learned_addr learned my addr 172.21.15.98:0/3649244932 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc037fe640 1 --2- 172.21.15.98:0/3649244932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a55df40 0x55f93a5682c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.656+0000 7fbc08ce7640 1 --2- 172.21.15.98:0/3649244932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a568800 0x55f93a56abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbc03fff640 1 -- 172.21.15.98:0/3649244932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a55df40 msgr2=0x55f93a5682c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbc03fff640 1 --2- 172.21.15.98:0/3649244932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a55df40 0x55f93a5682c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbc03fff640 1 -- 172.21.15.98:0/3649244932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a568800 msgr2=0x55f93a56abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbc03fff640 1 --2- 172.21.15.98:0/3649244932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a568800 0x55f93a56abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbc03fff640 1 -- 172.21.15.98:0/3649244932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f93a1d5210 con 0x55f93a55d620 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbc08ce7640 1 --2- 172.21.15.98:0/3649244932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a568800 0x55f93a56abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:42:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbc08ce7640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbbec0a3df0 0x7fbbec0a62b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc005fd0 tx=0x7fbbfc014b70 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbc03fff640 1 --2- 172.21.15.98:0/3649244932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55d620 0x55f93a55da00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009900 tx=0x7fbbf4032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192a48c2e9752cd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbbea7fc640 1 -- 172.21.15.98:0/3649244932 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4040070 con 0x55f93a55d620 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbbea7fc640 1 -- 172.21.15.98:0/3649244932 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf4004030 con 0x55f93a55d620 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.657+0000 7fbbea7fc640 1 -- 172.21.15.98:0/3649244932 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4035040 con 0x55f93a55d620 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 -- 172.21.15.98:0/3649244932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55d620 msgr2=0x55f93a55da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/3649244932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55d620 0x55f93a55da00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009900 tx=0x7fbbf4032bd0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 -- 172.21.15.98:0/3649244932 shutdown_connections 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/3649244932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a568800 0x55f93a56abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/3649244932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a55df40 0x55f93a5682c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/3649244932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55d620 0x55f93a55da00 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 -- 172.21.15.98:0/3649244932 >> 172.21.15.98:0/3649244932 conn(0x55f93a559b30 msgr2=0x55f93a559f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 -- 172.21.15.98:0/3649244932 shutdown_connections 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 -- 172.21.15.98:0/3649244932 wait complete. 2024-04-25T17:42:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 -- start start 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 Processor -- start 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a568100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a5653f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.658+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a568800 0x55f93a565930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f93a56b3d0 con 0x55f93a55df40 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f93a45d0b0 con 0x55f93a55d620 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f93a56b550 con 0x55f93a568800 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc0d45c740 10 client.0 _ll_register_callbacks cb 0x55f93a44ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a5653f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a5653f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33454/0 (socket says 172.21.15.98:33454) 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc037fe640 1 -- 172.21.15.98:0/432846187 learned_addr learned my addr 172.21.15.98:0/432846187 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc03fff640 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a568100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.667 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc08ce7640 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a568800 0x55f93a565930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc037fe640 1 -- 172.21.15.98:0/432846187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a568800 msgr2=0x55f93a565930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc037fe640 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a568800 0x55f93a565930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc037fe640 1 -- 172.21.15.98:0/432846187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 msgr2=0x55f93a568100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc037fe640 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a568100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc037fe640 1 -- 172.21.15.98:0/432846187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f93a2a1620 con 0x55f93a55df40 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc037fe640 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a5653f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009610 tx=0x7fbbf809c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e3d0124f576f4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbbe37fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbbe37fe640 1 -- 172.21.15.98:0/432846187 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8021040 con 0x55f93a55df40 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbbe37fe640 1 -- 172.21.15.98:0/432846187 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fbbf8098910 con 0x55f93a55df40 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc0d45c740 1 -- 172.21.15.98:0/432846187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbf4004030 con 0x55f93a55df40 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbbe37fe640 1 -- 172.21.15.98:0/432846187 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fbbf805b140 con 0x55f93a55df40 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc0d45c740 10 client.16047 fetch_fsmap learned FSMap version 37 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.659+0000 7fbc0d45c740 1 -- 172.21.15.98:0/432846187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f93a577300 con 0x55f93a55df40 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.660+0000 7fbbe37fe640 1 -- 172.21.15.98:0/432846187 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf4004030 con 0x55f93a55df40 2024-04-25T17:42:52.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.660+0000 7fbbe37fe640 1 -- 172.21.15.98:0/432846187 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7fbbf8097ff0 con 0x55f93a55df40 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.660+0000 7fbc0d45c740 10 client.16047 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.660+0000 7fbc0d45c740 10 client.16047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.660+0000 7fbc0d45c740 20 client.16047 populate_metadata read hostname 'smithi098' 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.660+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f93a5787c0 0x55f93a57ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.660+0000 7fbc0d45c740 4 client.16047 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.660+0000 7fbc0d45c740 1 -- 172.21.15.98:0/432846187 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f93a4d0390 con 0x55f93a5787c0 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.660+0000 7fbc03fff640 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f93a5787c0 0x55f93a57ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbc03fff640 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f93a5787c0 0x55f93a57ac50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c716ec9f47b91faa server_cookie=11b915b9d59ec999 in_seq=0 out_seq=0 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbbe37fe640 10 client.16047 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbbe37fe640 1 -- 172.21.15.98:0/432846187 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f93a4d0390 con 0x55f93a5787c0 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbbe37fe640 10 client.16047 handle_command_reply: tid=0 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbc0d45c740 1 client.16047 shutdown 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbc0d45c740 1 -- 172.21.15.98:0/432846187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 msgr2=0x55f93a5653f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:52.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a5653f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009610 tx=0x7fbbf809c2a0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbc0d45c740 1 -- 172.21.15.98:0/432846187 shutdown_connections 2024-04-25T17:42:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f93a5787c0 0x55f93a57ac50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a568800 0x55f93a565930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbbe37fe640 0 client.16047 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a5653f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/432846187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a568100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.661+0000 7fbc0d45c740 1 -- 172.21.15.98:0/432846187 >> 172.21.15.98:0/432846187 conn(0x55f93a559b30 msgr2=0x55f93a55f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.662+0000 7fbc0d45c740 1 -- 172.21.15.98:0/432846187 shutdown_connections 2024-04-25T17:42:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.662+0000 7fbc0d45c740 1 -- 172.21.15.98:0/432846187 wait complete. 2024-04-25T17:42:52.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.662+0000 7fbc0d45c740 20 client.16047 trim_cache size 0 max 16384 2024-04-25T17:42:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.676+0000 7fbc0d45c740 1 Processor -- start 2024-04-25T17:42:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.676+0000 7fbc0d45c740 1 -- start start 2024-04-25T17:42:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.676+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a559b30 0x55f93a574290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.676+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a560970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.676+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a567c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.676+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc041b2650 con 0x55f93a55df40 2024-04-25T17:42:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.676+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc041afe90 con 0x55f93a55d620 2024-04-25T17:42:52.682 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.676+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc041add00 con 0x55f93a559b30 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc08ce7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a567c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc08ce7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a567c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33456/0 (socket says 172.21.15.98:33456) 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc08ce7640 1 -- 172.21.15.98:0/167108503 learned_addr learned my addr 172.21.15.98:0/167108503 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc037fe640 1 --2- 172.21.15.98:0/167108503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a560970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc03fff640 1 --2- 172.21.15.98:0/167108503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a559b30 0x55f93a574290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc08ce7640 1 -- 172.21.15.98:0/167108503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a559b30 msgr2=0x55f93a574290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc08ce7640 1 --2- 172.21.15.98:0/167108503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a559b30 0x55f93a574290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc08ce7640 1 -- 172.21.15.98:0/167108503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 msgr2=0x55f93a560970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc08ce7640 1 --2- 172.21.15.98:0/167108503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a560970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc08ce7640 1 -- 172.21.15.98:0/167108503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f93a563940 con 0x55f93a55df40 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc03fff640 1 --2- 172.21.15.98:0/167108503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a559b30 0x55f93a574290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbc08ce7640 1 --2- 172.21.15.98:0/167108503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a567c30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc012ca0 tx=0x7fbbfc01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b51a0e52aedf63f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbbe9ffb640 1 -- 172.21.15.98:0/167108503 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc012cd0 con 0x55f93a55df40 2024-04-25T17:42:52.683 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbbe9ffb640 1 -- 172.21.15.98:0/167108503 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbfc012e70 con 0x55f93a55df40 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.677+0000 7fbbe9ffb640 1 -- 172.21.15.98:0/167108503 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc02c430 con 0x55f93a55df40 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 -- 172.21.15.98:0/167108503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 msgr2=0x55f93a567c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/167108503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a567c30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc012ca0 tx=0x7fbbfc01d000 comp rx=0 tx=0).stop 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 -- 172.21.15.98:0/167108503 shutdown_connections 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/167108503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a55df40 0x55f93a567c30 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/167108503 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a560970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/167108503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a559b30 0x55f93a574290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 -- 172.21.15.98:0/167108503 >> 172.21.15.98:0/167108503 conn(0x55f93a565a90 msgr2=0x55f93a560260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 -- 172.21.15.98:0/167108503 shutdown_connections 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 -- 172.21.15.98:0/167108503 wait complete. 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 -- start start 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.678+0000 7fbc0d45c740 1 Processor -- start 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a559b30 0x55f93a5707c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.684 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a570d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc0d45c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a55df40 0x55f93a56bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbc041ae350 con 0x55f93a559b30 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbc041af7c0 con 0x55f93a55d620 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc0d45c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc041aea20 con 0x55f93a55df40 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc0d45c740 10 client.0 _ll_register_callbacks cb 0x7fbbf809bc00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc03fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a559b30 0x55f93a5707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc03fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a559b30 0x55f93a5707c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33472/0 (socket says 172.21.15.98:33472) 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc03fff640 1 -- 172.21.15.98:0/1015369066 learned_addr learned my addr 172.21.15.98:0/1015369066 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc08ce7640 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a55df40 0x55f93a56bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc037fe640 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a570d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc03fff640 1 -- 172.21.15.98:0/1015369066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a55df40 msgr2=0x55f93a56bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc03fff640 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a55df40 0x55f93a56bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc03fff640 1 -- 172.21.15.98:0/1015369066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 msgr2=0x55f93a570d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:42:52.685 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc03fff640 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a570d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc03fff640 1 -- 172.21.15.98:0/1015369066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f93a56c500 con 0x55f93a559b30 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc03fff640 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a559b30 0x55f93a5707c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fbbf402a930 tx=0x7fbbf402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36dc35400ae7e012 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbbe37fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbbe37fe640 1 -- 172.21.15.98:0/1015369066 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4040040 con 0x55f93a559b30 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbbe37fe640 1 -- 172.21.15.98:0/1015369066 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fbbf40318b0 con 0x55f93a559b30 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc0d45c740 1 -- 172.21.15.98:0/1015369066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbfc012e70 con 0x55f93a559b30 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbbe37fe640 1 -- 172.21.15.98:0/1015369066 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fbbf405f050 con 0x55f93a559b30 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc0d45c740 10 client.16053 fetch_fsmap learned FSMap version 37 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.679+0000 7fbc0d45c740 1 -- 172.21.15.98:0/1015369066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f93a5deaa0 con 0x55f93a559b30 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbbe37fe640 1 -- 172.21.15.98:0/1015369066 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbfc012e70 con 0x55f93a559b30 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbbe37fe640 1 -- 172.21.15.98:0/1015369066 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7fbbf4004750 con 0x55f93a559b30 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbc0d45c740 10 client.16053 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbc0d45c740 10 client.16053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:42:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbc0d45c740 20 client.16053 populate_metadata read hostname 'smithi098' 2024-04-25T17:42:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f93a568800 0x55f93a5e0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:42:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbc0d45c740 4 client.16053 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:42:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbc0d45c740 1 -- 172.21.15.98:0/1015369066 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f93a4ca200 con 0x55f93a568800 2024-04-25T17:42:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbc037fe640 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f93a568800 0x55f93a5e0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:42:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbbe37fe640 10 client.16053 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.680+0000 7fbc037fe640 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f93a568800 0x55f93a5e0f50 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcb720f342ceefa7 server_cookie=e63277e75df59c08 in_seq=0 out_seq=0 2024-04-25T17:42:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbbe37fe640 1 -- 172.21.15.98:0/1015369066 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f93a4ca200 con 0x55f93a568800 2024-04-25T17:42:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbbe37fe640 10 client.16053 handle_command_reply: tid=0 2024-04-25T17:42:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 client.16053 shutdown 2024-04-25T17:42:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 -- 172.21.15.98:0/1015369066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a559b30 msgr2=0x55f93a5707c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a559b30 0x55f93a5707c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fbbf402a930 tx=0x7fbbf402a410 comp rx=0 tx=0).stop 2024-04-25T17:42:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 -- 172.21.15.98:0/1015369066 shutdown_connections 2024-04-25T17:42:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f93a568800 0x55f93a5e0f50 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f93a55df40 0x55f93a56bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbbe37fe640 0 client.16053 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:42:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f93a55d620 0x55f93a570d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 --2- 172.21.15.98:0/1015369066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f93a559b30 0x55f93a5707c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 -- 172.21.15.98:0/1015369066 >> 172.21.15.98:0/1015369066 conn(0x55f93a565a90 msgr2=0x55f93a558cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 -- 172.21.15.98:0/1015369066 shutdown_connections 2024-04-25T17:42:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 1 -- 172.21.15.98:0/1015369066 wait complete. 2024-04-25T17:42:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0d45c740 20 client.16053 trim_cache size 0 max 16384 2024-04-25T17:42:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:42:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:42:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:42:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:42:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 -- 172.21.15.98:0/1978817018 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbbec0a3df0 msgr2=0x7fbbec0a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbbec0a3df0 0x7fbbec0a62b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc005fd0 tx=0x7fbbfc014b70 comp rx=0 tx=0).stop 2024-04-25T17:42:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 -- 172.21.15.98:0/1978817018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc041072a0 msgr2=0x7fbc04074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:42:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc041072a0 0x7fbc04074790 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800d770 tx=0x7fbbf800dc30 comp rx=0 tx=0).stop 2024-04-25T17:42:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 -- 172.21.15.98:0/1978817018 shutdown_connections 2024-04-25T17:42:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fbbec0a3df0 0x7fbbec0a62b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc04074cd0 0x7fbc041a6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc041072a0 0x7fbc04074790 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 --2- 172.21.15.98:0/1978817018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc041021d0 0x7fbc04074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:42:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 -- 172.21.15.98:0/1978817018 >> 172.21.15.98:0/1978817018 conn(0x7fbc040793e0 msgr2=0x7fbc040ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:42:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 -- 172.21.15.98:0/1978817018 shutdown_connections 2024-04-25T17:42:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:42:52.696+0000 7fbc0a74a640 1 -- 172.21.15.98:0/1978817018 wait complete. 2024-04-25T17:42:54.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:54 smithi098 ceph-mon[29192]: pgmap v215: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 908 op/s 2024-04-25T17:42:54.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:54 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:54.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:54 smithi203 ceph-mon[32834]: pgmap v215: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 908 op/s 2024-04-25T17:42:54.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:54 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:54.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:54 smithi173 ceph-mon[32856]: pgmap v215: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 908 op/s 2024-04-25T17:42:54.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:54 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:42:56.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:56 smithi098 ceph-mon[29192]: pgmap v216: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.1 MiB/s wr, 655 op/s 2024-04-25T17:42:56.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:56 smithi203 ceph-mon[32834]: pgmap v216: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.1 MiB/s wr, 655 op/s 2024-04-25T17:42:56.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:56 smithi173 ceph-mon[32856]: pgmap v216: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.1 MiB/s wr, 655 op/s 2024-04-25T17:42:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 385.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:42:58.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:42:58 smithi098 ceph-mon[29192]: pgmap v217: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 8.3 MiB/s wr, 677 op/s 2024-04-25T17:42:58.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:42:58 smithi203 ceph-mon[32834]: pgmap v217: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 8.3 MiB/s wr, 677 op/s 2024-04-25T17:42:58.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:42:58 smithi173 ceph-mon[32856]: pgmap v217: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 8.3 MiB/s wr, 677 op/s 2024-04-25T17:43:00.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:00 smithi098 ceph-mon[29192]: pgmap v218: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 4.1 MiB/s wr, 311 op/s 2024-04-25T17:43:00.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:00 smithi203 ceph-mon[32834]: pgmap v218: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 4.1 MiB/s wr, 311 op/s 2024-04-25T17:43:00.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:00 smithi173 ceph-mon[32856]: pgmap v218: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 4.1 MiB/s wr, 311 op/s 2024-04-25T17:43:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 390.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:43:02.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:02 smithi098 ceph-mon[29192]: pgmap v219: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 6.1 MiB/s wr, 398 op/s 2024-04-25T17:43:02.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:02 smithi203 ceph-mon[32834]: pgmap v219: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 6.1 MiB/s wr, 398 op/s 2024-04-25T17:43:02.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:02 smithi173 ceph-mon[32856]: pgmap v219: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 6.1 MiB/s wr, 398 op/s 2024-04-25T17:43:02.748 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:43:02.748 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 38, 2024-04-25T17:43:02.748 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:43:02.748 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 18, 2024-04-25T17:43:02.748 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "8b98b092": { 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 18, 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 73.1, 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:43:02.749 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:43:02.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:02.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:02.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:02.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:02.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:02.750 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f8075ffb700 1 -- 172.21.15.98:0/2299209912 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55e72117e6b0 con 0x55e7212a13d0 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f8075ffb700 10 client.25555 handle_command_reply: tid=0 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f809f3c1b80 1 client.25555 shutdown 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f809f3c1b80 1 -- 172.21.15.98:0/2299209912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7212264f0 msgr2=0x55e72122dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7212264f0 0x55e72122dff0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8084009b00 tx=0x7f80840047c0 comp rx=0 tx=0).stop 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f809f3c1b80 1 -- 172.21.15.98:0/2299209912 shutdown_connections 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e7212a13d0 0x55e7212a37f0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e721229270 0x55e72129f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f8075ffb700 0 client.25555 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e72121e3a0 0x55e72122dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:02.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f809f3c1b80 1 --2- 172.21.15.98:0/2299209912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e7212264f0 0x55e72122dff0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f809f3c1b80 1 -- 172.21.15.98:0/2299209912 >> 172.21.15.98:0/2299209912 conn(0x55e72114b820 msgr2=0x55e721227790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.747+0000 7f809f3c1b80 1 -- 172.21.15.98:0/2299209912 shutdown_connections 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f809f3c1b80 1 -- 172.21.15.98:0/2299209912 wait complete. 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f809f3c1b80 20 client.25555 trim_cache size 0 max 16384 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f8094b38700 1 -- 172.21.15.98:0/2327469522 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80780a3ed0 msgr2=0x7f80780a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f8094b38700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80780a3ed0 0x7f80780a6390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8090105060 tx=0x7f8084009310 comp rx=0 tx=0).stop 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f8094b38700 1 -- 172.21.15.98:0/2327469522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8090101030 msgr2=0x7f80901a1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f8094b38700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8090101030 0x7f80901a1080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8080000c00 tx=0x7f808000fa00 comp rx=0 tx=0).stop 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f8094b38700 1 -- 172.21.15.98:0/2327469522 shutdown_connections 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f8094b38700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80780a3ed0 0x7f80780a6390 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f8094b38700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8090101030 0x7f80901a1080 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f8094b38700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8090108bc0 0x7f809019fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f8094b38700 1 --2- 172.21.15.98:0/2327469522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80901019f0 0x7f80901a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.748+0000 7f8094b38700 1 -- 172.21.15.98:0/2327469522 >> 172.21.15.98:0/2327469522 conn(0x7f8090077fe0 msgr2=0x7f8090103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.749+0000 7f8094b38700 1 -- 172.21.15.98:0/2327469522 shutdown_connections 2024-04-25T17:43:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:02.749+0000 7f8094b38700 1 -- 172.21.15.98:0/2327469522 wait complete. 2024-04-25T17:43:03.519 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '8b98b092', quiesce took 19.7 seconds. Will release after: 13.1 2024-04-25T17:43:04.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:04 smithi098 ceph-mon[29192]: pgmap v220: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 11 MiB/s wr, 562 op/s 2024-04-25T17:43:04.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:04 smithi203 ceph-mon[32834]: pgmap v220: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 11 MiB/s wr, 562 op/s 2024-04-25T17:43:04.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:04 smithi173 ceph-mon[32856]: pgmap v220: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 11 MiB/s wr, 562 op/s 2024-04-25T17:43:05.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:05 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:43:05.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:05 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:43:05.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:05 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:43:05.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:05 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:43:05.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:05 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:43:05.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:05 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:43:05.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:05 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:43:05.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:05 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:43:05.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:05 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:43:05.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:05 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:43:05.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:05 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:43:05.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:05 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:43:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.787+0000 7fc39afcf640 1 -- 172.21.15.98:0/2268714043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3941006c0 msgr2=0x7fc394100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.787+0000 7fc39afcf640 1 --2- 172.21.15.98:0/2268714043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3941006c0 0x7fc394100aa0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc388009900 tx=0x7fc38802a410 comp rx=0 tx=0).stop 2024-04-25T17:43:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.787+0000 7fc39afcf640 1 -- 172.21.15.98:0/2268714043 shutdown_connections 2024-04-25T17:43:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.787+0000 7fc39afcf640 1 --2- 172.21.15.98:0/2268714043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc394107330 0x7fc39410eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.787+0000 7fc39afcf640 1 --2- 172.21.15.98:0/2268714043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc394101070 0x7fc394106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.787+0000 7fc39afcf640 1 --2- 172.21.15.98:0/2268714043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3941006c0 0x7fc394100aa0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.787+0000 7fc39afcf640 1 -- 172.21.15.98:0/2268714043 >> 172.21.15.98:0/2268714043 conn(0x7fc394077390 msgr2=0x7fc3940fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.787+0000 7fc39afcf640 1 -- 172.21.15.98:0/2268714043 shutdown_connections 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.787+0000 7fc39afcf640 1 -- 172.21.15.98:0/2268714043 wait complete. 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.787+0000 7fc39afcf640 1 Processor -- start 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc39afcf640 1 -- start start 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc39afcf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3941006c0 0x7fc394198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc39afcf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc394107330 0x7fc394198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc39afcf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc39419cf30 0x7fc39419d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc39afcf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc394112250 con 0x7fc3941006c0 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc39afcf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc3941120d0 con 0x7fc394107330 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc39afcf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3941123d0 con 0x7fc39419cf30 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc398d6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3941006c0 0x7fc394198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc398d6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3941006c0 0x7fc394198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51544/0 (socket says 172.21.15.98:51544) 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc398d6b640 1 -- 172.21.15.98:0/1923671908 learned_addr learned my addr 172.21.15.98:0/1923671908 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc39956c640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc39419cf30 0x7fc39419d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc393fff640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc394107330 0x7fc394198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc398d6b640 1 -- 172.21.15.98:0/1923671908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc39419cf30 msgr2=0x7fc39419d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc398d6b640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc39419cf30 0x7fc39419d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc398d6b640 1 -- 172.21.15.98:0/1923671908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc394107330 msgr2=0x7fc394198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc398d6b640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc394107330 0x7fc394198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc398d6b640 1 -- 172.21.15.98:0/1923671908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39419da20 con 0x7fc3941006c0 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc39956c640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc39419cf30 0x7fc39419d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc393fff640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc394107330 0x7fc394198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc398d6b640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3941006c0 0x7fc394198670 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc388006010 tx=0x7fc3880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.788+0000 7fc391ffb640 1 -- 172.21.15.98:0/1923671908 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38803f030 con 0x7fc3941006c0 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.789+0000 7fc391ffb640 1 -- 172.21.15.98:0/1923671908 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc388003800 con 0x7fc3941006c0 2024-04-25T17:43:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.789+0000 7fc39afcf640 1 -- 172.21.15.98:0/1923671908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39419dcb0 con 0x7fc3941006c0 2024-04-25T17:43:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.789+0000 7fc391ffb640 1 -- 172.21.15.98:0/1923671908 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc388035070 con 0x7fc3941006c0 2024-04-25T17:43:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.789+0000 7fc39afcf640 1 -- 172.21.15.98:0/1923671908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3941a5470 con 0x7fc3941006c0 2024-04-25T17:43:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.791+0000 7fc391ffb640 1 -- 172.21.15.98:0/1923671908 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc388003cb0 con 0x7fc3941006c0 2024-04-25T17:43:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.791+0000 7fc39ddbd740 1 Processor -- start 2024-04-25T17:43:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.791+0000 7fc39ddbd740 1 -- start start 2024-04-25T17:43:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04500 0x563d36f048e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04e20 0x563d36f136d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc391ffb640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc3840a3dc0 0x7fc3840a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f16000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563d36e69970 con 0x563d36f13c10 2024-04-25T17:43:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563d36e064a0 con 0x563d36f04e20 2024-04-25T17:43:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563d36e65930 con 0x563d36f04500 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc393fff640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc3840a3dc0 0x7fc3840a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39956c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f16000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39956c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f16000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51546/0 (socket says 172.21.15.98:51546) 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39956c640 1 -- 172.21.15.98:0/670442333 learned_addr learned my addr 172.21.15.98:0/670442333 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc398d6b640 1 --2- 172.21.15.98:0/670442333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04e20 0x563d36f136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc391ffb640 1 -- 172.21.15.98:0/1923671908 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fc3880f1250 con 0x7fc3941006c0 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39956c640 1 --2- 172.21.15.98:0/670442333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04500 0x563d36f048e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39956c640 1 -- 172.21.15.98:0/670442333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04500 msgr2=0x563d36f048e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39956c640 1 --2- 172.21.15.98:0/670442333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04500 0x563d36f048e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39956c640 1 -- 172.21.15.98:0/670442333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04e20 msgr2=0x563d36f136d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39956c640 1 --2- 172.21.15.98:0/670442333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04e20 0x563d36f136d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39956c640 1 -- 172.21.15.98:0/670442333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d36b7c210 con 0x563d36f13c10 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc393fff640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc3840a3dc0 0x7fc3840a6280 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc394104aa0 tx=0x7fc38000a5c0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc39956c640 1 --2- 172.21.15.98:0/670442333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f16000 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00af90 tx=0x7fc38c00a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0365dcea1be952 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc37effd640 1 -- 172.21.15.98:0/670442333 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c018070 con 0x563d36f13c10 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc37effd640 1 -- 172.21.15.98:0/670442333 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc38c010e00 con 0x563d36f13c10 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.792+0000 7fc37effd640 1 -- 172.21.15.98:0/670442333 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c0056a0 con 0x563d36f13c10 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 -- 172.21.15.98:0/670442333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 msgr2=0x563d36f16000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/670442333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f16000 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00af90 tx=0x7fc38c00a840 comp rx=0 tx=0).stop 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 -- 172.21.15.98:0/670442333 shutdown_connections 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/670442333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f16000 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/670442333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04e20 0x563d36f136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/670442333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04500 0x563d36f048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 -- 172.21.15.98:0/670442333 >> 172.21.15.98:0/670442333 conn(0x563d36f00b30 msgr2=0x563d36f00f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 -- 172.21.15.98:0/670442333 shutdown_connections 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 -- 172.21.15.98:0/670442333 wait complete. 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 -- start start 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.793+0000 7fc39ddbd740 1 Processor -- start 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04500 0x563d36f13550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04e20 0x563d36f10840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f10d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563d36f16840 con 0x563d36f13c10 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563d36e040b0 con 0x563d36f04e20 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563d36f169c0 con 0x563d36f04500 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39ddbd740 10 client.0 _ll_register_callbacks cb 0x563d36df1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39956c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f10d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc393fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04e20 0x563d36f10840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39956c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f10d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51548/0 (socket says 172.21.15.98:51548) 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39956c640 1 -- 172.21.15.98:0/1086675754 learned_addr learned my addr 172.21.15.98:0/1086675754 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc398d6b640 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04500 0x563d36f13550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39956c640 1 -- 172.21.15.98:0/1086675754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04500 msgr2=0x563d36f13550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39956c640 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04500 0x563d36f13550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39956c640 1 -- 172.21.15.98:0/1086675754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04e20 msgr2=0x563d36f10840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39956c640 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04e20 0x563d36f10840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39956c640 1 -- 172.21.15.98:0/1086675754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d36c48620 con 0x563d36f13c10 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc39956c640 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f10d80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc38c005360 tx=0x7fc38c00aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b298dc65d915575 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.794+0000 7fc373fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc373fff640 1 -- 172.21.15.98:0/1086675754 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c018070 con 0x563d36f13c10 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc373fff640 1 -- 172.21.15.98:0/1086675754 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fc38c004a40 con 0x563d36f13c10 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc39ddbd740 1 -- 172.21.15.98:0/1086675754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc38c010e00 con 0x563d36f13c10 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc373fff640 1 -- 172.21.15.98:0/1086675754 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc38c004c40 con 0x563d36f13c10 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc39ddbd740 10 client.16065 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc39ddbd740 1 -- 172.21.15.98:0/1086675754 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x563d36f22830 con 0x563d36f13c10 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc373fff640 1 -- 172.21.15.98:0/1086675754 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc38c010e00 con 0x563d36f13c10 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc373fff640 1 -- 172.21.15.98:0/1086675754 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fc38c023030 con 0x563d36f13c10 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc39ddbd740 10 client.16065 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc39ddbd740 10 client.16065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:43:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc39ddbd740 20 client.16065 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563d36f23c30 0x563d36f26020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc39ddbd740 4 client.16065 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc39ddbd740 1 -- 172.21.15.98:0/1086675754 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d36e77390 con 0x563d36f23c30 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.795+0000 7fc398d6b640 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563d36f23c30 0x563d36f26020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.796+0000 7fc373fff640 10 client.16065 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.796+0000 7fc398d6b640 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563d36f23c30 0x563d36f26020 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c168232032393b1 server_cookie=58669b828f3e1c7f in_seq=0 out_seq=0 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.796+0000 7fc373fff640 1 -- 172.21.15.98:0/1086675754 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563d36e77390 con 0x563d36f23c30 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.796+0000 7fc373fff640 10 client.16065 handle_command_reply: tid=0 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.796+0000 7fc39ddbd740 1 client.16065 shutdown 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 1 -- 172.21.15.98:0/1086675754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 msgr2=0x563d36f10d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f10d80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc38c005360 tx=0x7fc38c00aeb0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 1 -- 172.21.15.98:0/1086675754 shutdown_connections 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563d36f23c30 0x563d36f26020 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f10d80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04e20 0x563d36f10840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc373fff640 0 client.16065 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/1086675754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04500 0x563d36f13550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 1 -- 172.21.15.98:0/1086675754 >> 172.21.15.98:0/1086675754 conn(0x563d36f00b30 msgr2=0x563d36f069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 1 -- 172.21.15.98:0/1086675754 shutdown_connections 2024-04-25T17:43:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 1 -- 172.21.15.98:0/1086675754 wait complete. 2024-04-25T17:43:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.797+0000 7fc39ddbd740 20 client.16065 trim_cache size 0 max 16384 2024-04-25T17:43:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.811+0000 7fc39ddbd740 1 Processor -- start 2024-04-25T17:43:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.811+0000 7fc39ddbd740 1 -- start start 2024-04-25T17:43:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.811+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f04500 0x563d36f1f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.811+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 0x563d36effb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.811+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f13c10 0x563d36f13080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.811+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3941ae210 con 0x563d36f04500 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.811+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc3941aba20 con 0x563d36f13c10 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.811+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3941a9890 con 0x563d36f04e20 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc398d6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f04500 0x563d36f1f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc398d6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f04500 0x563d36f1f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51564/0 (socket says 172.21.15.98:51564) 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc398d6b640 1 -- 172.21.15.98:0/858631564 learned_addr learned my addr 172.21.15.98:0/858631564 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc393fff640 1 --2- 172.21.15.98:0/858631564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 0x563d36effb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc398d6b640 1 -- 172.21.15.98:0/858631564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 msgr2=0x563d36effb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc398d6b640 1 --2- 172.21.15.98:0/858631564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 0x563d36effb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc398d6b640 1 -- 172.21.15.98:0/858631564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f13c10 msgr2=0x563d36f13080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc398d6b640 1 --2- 172.21.15.98:0/858631564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f13c10 0x563d36f13080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc398d6b640 1 -- 172.21.15.98:0/858631564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d36f265e0 con 0x563d36f04500 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc393fff640 1 --2- 172.21.15.98:0/858631564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 0x563d36effb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc398d6b640 1 --2- 172.21.15.98:0/858631564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f04500 0x563d36f1f7a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc3880b9650 tx=0x7fc3880cd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efab58b5f5f08e96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc37e7fc640 1 -- 172.21.15.98:0/858631564 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc388049040 con 0x563d36f04500 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc37e7fc640 1 -- 172.21.15.98:0/858631564 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3880d1030 con 0x563d36f04500 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.812+0000 7fc37e7fc640 1 -- 172.21.15.98:0/858631564 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3880b9860 con 0x563d36f04500 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 -- 172.21.15.98:0/858631564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f04500 msgr2=0x563d36f1f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/858631564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f04500 0x563d36f1f7a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc3880b9650 tx=0x7fc3880cd040 comp rx=0 tx=0).stop 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 -- 172.21.15.98:0/858631564 shutdown_connections 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/858631564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f13c10 0x563d36f13080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/858631564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 0x563d36effb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/858631564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f04500 0x563d36f1f7a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 -- 172.21.15.98:0/858631564 >> 172.21.15.98:0/858631564 conn(0x563d36f00b30 msgr2=0x563d36eff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 -- 172.21.15.98:0/858631564 shutdown_connections 2024-04-25T17:43:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 -- 172.21.15.98:0/858631564 wait complete. 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 -- start start 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 Processor -- start 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04500 0x563d36f17790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 0x563d36f17cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f18210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3941a9ee0 con 0x563d36f13c10 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc3941ab350 con 0x563d36f04500 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3941aa5b0 con 0x563d36f04e20 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39ddbd740 10 client.0 _ll_register_callbacks cb 0x7fc38c005bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.813+0000 7fc39956c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f18210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39956c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f18210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51576/0 (socket says 172.21.15.98:51576) 2024-04-25T17:43:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39956c640 1 -- 172.21.15.98:0/2666392362 learned_addr learned my addr 172.21.15.98:0/2666392362 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc393fff640 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 0x563d36f17cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc398d6b640 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04500 0x563d36f17790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39956c640 1 -- 172.21.15.98:0/2666392362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 msgr2=0x563d36f17cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39956c640 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 0x563d36f17cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39956c640 1 -- 172.21.15.98:0/2666392362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04500 msgr2=0x563d36f17790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39956c640 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04500 0x563d36f17790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39956c640 1 -- 172.21.15.98:0/2666392362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d36f05b70 con 0x563d36f13c10 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc398d6b640 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04500 0x563d36f17790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39956c640 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f18210 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc38c005360 tx=0x7fc38c011d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e19e81dd545c97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc373fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc373fff640 1 -- 172.21.15.98:0/2666392362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c018070 con 0x563d36f13c10 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc373fff640 1 -- 172.21.15.98:0/2666392362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fc38c012390 con 0x563d36f13c10 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39ddbd740 1 -- 172.21.15.98:0/2666392362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3880d1030 con 0x563d36f13c10 2024-04-25T17:43:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc373fff640 1 -- 172.21.15.98:0/2666392362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fc38c036080 con 0x563d36f13c10 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39ddbd740 10 client.16071 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc39ddbd740 1 -- 172.21.15.98:0/2666392362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x563d36f8a1e0 con 0x563d36f13c10 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc373fff640 1 -- 172.21.15.98:0/2666392362 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3880d1030 con 0x563d36f13c10 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.814+0000 7fc373fff640 1 -- 172.21.15.98:0/2666392362 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fc38c033030 con 0x563d36f13c10 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.815+0000 7fc39ddbd740 10 client.16071 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.815+0000 7fc39ddbd740 10 client.16071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.815+0000 7fc39ddbd740 20 client.16071 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.815+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563d36f8a8a0 0x563d36f8ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.815+0000 7fc39ddbd740 4 client.16071 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.815+0000 7fc39ddbd740 1 -- 172.21.15.98:0/2666392362 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563d36e71200 con 0x563d36f8a8a0 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.815+0000 7fc398d6b640 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563d36f8a8a0 0x563d36f8ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.815+0000 7fc398d6b640 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563d36f8a8a0 0x563d36f8ccc0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efbacf7d6430fad1 server_cookie=3dfa2804eccae351 in_seq=0 out_seq=0 2024-04-25T17:43:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.815+0000 7fc373fff640 10 client.16071 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.830+0000 7fc373fff640 1 -- 172.21.15.98:0/2666392362 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563d36e71200 con 0x563d36f8a8a0 2024-04-25T17:43:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.830+0000 7fc373fff640 10 client.16071 handle_command_reply: tid=0 2024-04-25T17:43:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.830+0000 7fc39ddbd740 1 client.16071 shutdown 2024-04-25T17:43:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.830+0000 7fc39ddbd740 1 -- 172.21.15.98:0/2666392362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 msgr2=0x563d36f18210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f18210 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc38c005360 tx=0x7fc38c011d50 comp rx=0 tx=0).stop 2024-04-25T17:43:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39ddbd740 1 -- 172.21.15.98:0/2666392362 shutdown_connections 2024-04-25T17:43:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563d36f8a8a0 0x563d36f8ccc0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563d36f13c10 0x563d36f18210 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563d36f04e20 0x563d36f17cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc373fff640 0 client.16071 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39ddbd740 1 --2- 172.21.15.98:0/2666392362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563d36f04500 0x563d36f17790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39ddbd740 1 -- 172.21.15.98:0/2666392362 >> 172.21.15.98:0/2666392362 conn(0x563d36f00b30 msgr2=0x563d36effcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39ddbd740 1 -- 172.21.15.98:0/2666392362 shutdown_connections 2024-04-25T17:43:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39ddbd740 1 -- 172.21.15.98:0/2666392362 wait complete. 2024-04-25T17:43:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39ddbd740 20 client.16071 trim_cache size 0 max 16384 2024-04-25T17:43:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:43:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:43:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:43:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:43:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39afcf640 1 -- 172.21.15.98:0/1923671908 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc3840a3dc0 msgr2=0x7fc3840a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39afcf640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc3840a3dc0 0x7fc3840a6280 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc394104aa0 tx=0x7fc38000a5c0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39afcf640 1 -- 172.21.15.98:0/1923671908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3941006c0 msgr2=0x7fc394198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39afcf640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3941006c0 0x7fc394198670 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc388006010 tx=0x7fc3880042f0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39afcf640 1 -- 172.21.15.98:0/1923671908 shutdown_connections 2024-04-25T17:43:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39afcf640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc3840a3dc0 0x7fc3840a6280 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39afcf640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc39419cf30 0x7fc39419d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39afcf640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc394107330 0x7fc394198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39afcf640 1 --2- 172.21.15.98:0/1923671908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3941006c0 0x7fc394198670 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.831+0000 7fc39afcf640 1 -- 172.21.15.98:0/1923671908 >> 172.21.15.98:0/1923671908 conn(0x7fc394077390 msgr2=0x7fc394105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.832+0000 7fc39afcf640 1 -- 172.21.15.98:0/1923671908 shutdown_connections 2024-04-25T17:43:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:05.832+0000 7fc39afcf640 1 -- 172.21.15.98:0/1923671908 wait complete. 2024-04-25T17:43:06.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:06 smithi098 ceph-mon[29192]: pgmap v221: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.9 MiB/s wr, 348 op/s 2024-04-25T17:43:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:06 smithi203 ceph-mon[32834]: pgmap v221: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.9 MiB/s wr, 348 op/s 2024-04-25T17:43:06.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:06 smithi173 ceph-mon[32856]: pgmap v221: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.9 MiB/s wr, 348 op/s 2024-04-25T17:43:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 395.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:43:08.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:08 smithi098 ceph-mon[29192]: pgmap v222: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.9 MiB/s wr, 357 op/s 2024-04-25T17:43:08.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:08 smithi203 ceph-mon[32834]: pgmap v222: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.9 MiB/s wr, 357 op/s 2024-04-25T17:43:08.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:08 smithi173 ceph-mon[32856]: pgmap v222: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.9 MiB/s wr, 357 op/s 2024-04-25T17:43:10.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:10 smithi098 ceph-mon[29192]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.8 MiB/s wr, 335 op/s 2024-04-25T17:43:10.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:10 smithi203 ceph-mon[32834]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.8 MiB/s wr, 335 op/s 2024-04-25T17:43:10.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:10 smithi173 ceph-mon[32856]: pgmap v223: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 7.8 MiB/s wr, 335 op/s 2024-04-25T17:43:11.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:11 smithi098 ceph-mon[29192]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T17:43:11.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:11 smithi203 ceph-mon[32834]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T17:43:11.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:11 smithi173 ceph-mon[32856]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T17:43:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 400.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:43:12.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:12 smithi098 ceph-mon[29192]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.9 MiB/s wr, 302 op/s 2024-04-25T17:43:12.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:12 smithi203 ceph-mon[32834]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.9 MiB/s wr, 302 op/s 2024-04-25T17:43:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:12 smithi173 ceph-mon[32856]: pgmap v225: 97 pgs: 97 active+clean; 3.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.9 MiB/s wr, 302 op/s 2024-04-25T17:43:14.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:14 smithi098 ceph-mon[29192]: pgmap v226: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 60 KiB/s wr, 18 op/s 2024-04-25T17:43:14.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:14 smithi203 ceph-mon[32834]: pgmap v226: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 60 KiB/s wr, 18 op/s 2024-04-25T17:43:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:14 smithi173 ceph-mon[32856]: pgmap v226: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 60 KiB/s wr, 18 op/s 2024-04-25T17:43:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:16 smithi098 ceph-mon[29192]: pgmap v227: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 60 KiB/s wr, 18 op/s 2024-04-25T17:43:16.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:16 smithi098 ceph-mon[29192]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T17:43:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:16 smithi203 ceph-mon[32834]: pgmap v227: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 60 KiB/s wr, 18 op/s 2024-04-25T17:43:16.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:16 smithi203 ceph-mon[32834]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T17:43:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:16 smithi173 ceph-mon[32856]: pgmap v227: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 60 KiB/s wr, 18 op/s 2024-04-25T17:43:16.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:16 smithi173 ceph-mon[32856]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T17:43:16.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 405.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:43:16.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:43:16.870 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:43:18.409 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:18 smithi098 ceph-mon[29192]: pgmap v229: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 23 MiB/s wr, 658 op/s 2024-04-25T17:43:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:18 smithi203 ceph-mon[32834]: pgmap v229: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 23 MiB/s wr, 658 op/s 2024-04-25T17:43:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:18 smithi173 ceph-mon[32856]: pgmap v229: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 23 MiB/s wr, 658 op/s 2024-04-25T17:43:18.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.793+0000 7f5deaf7a700 1 -- 172.21.15.98:0/986629261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de4108a70 msgr2=0x7f5de4108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.793+0000 7f5deaf7a700 1 --2- 172.21.15.98:0/986629261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de4108a70 0x7f5de4108e50 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00b0a0 tx=0x7f5ddc00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.793+0000 7f5deaf7a700 1 -- 172.21.15.98:0/986629261 shutdown_connections 2024-04-25T17:43:18.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.793+0000 7f5deaf7a700 1 --2- 172.21.15.98:0/986629261 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de4103460 0x7f5de410f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.793+0000 7f5deaf7a700 1 --2- 172.21.15.98:0/986629261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de4108a70 0x7f5de4108e50 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.793+0000 7f5deaf7a700 1 --2- 172.21.15.98:0/986629261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de4102ac0 0x7f5de4102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.793+0000 7f5deaf7a700 1 -- 172.21.15.98:0/986629261 >> 172.21.15.98:0/986629261 conn(0x7f5de4079bf0 msgr2=0x7f5de407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:18.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.793+0000 7f5deaf7a700 1 -- 172.21.15.98:0/986629261 shutdown_connections 2024-04-25T17:43:18.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.793+0000 7f5deaf7a700 1 -- 172.21.15.98:0/986629261 wait complete. 2024-04-25T17:43:18.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5deaf7a700 1 Processor -- start 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5deaf7a700 1 -- start start 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5deaf7a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de4102ac0 0x7f5de4199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5deaf7a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de4103460 0x7f5de41996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5deaf7a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de4108a70 0x7f5de419da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5deaf7a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5de4112c70 con 0x7f5de4102ac0 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5deaf7a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5de4112af0 con 0x7f5de4103460 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5deaf7a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5de4112df0 con 0x7f5de4108a70 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de8d16700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de4102ac0 0x7f5de4199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de8d16700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de4102ac0 0x7f5de4199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47244/0 (socket says 172.21.15.98:47244) 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de8d16700 1 -- 172.21.15.98:0/2959525743 learned_addr learned my addr 172.21.15.98:0/2959525743 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de3fff700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de4103460 0x7f5de41996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de8d16700 1 -- 172.21.15.98:0/2959525743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de4108a70 msgr2=0x7f5de419da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de8d16700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de4108a70 0x7f5de419da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de8d16700 1 -- 172.21.15.98:0/2959525743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de4103460 msgr2=0x7f5de41996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de8d16700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de4103460 0x7f5de41996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de8d16700 1 -- 172.21.15.98:0/2959525743 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de419e170 con 0x7f5de4102ac0 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de3fff700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de4103460 0x7f5de41996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.794+0000 7f5de8d16700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de4102ac0 0x7f5de4199160 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00b070 tx=0x7f5ddc004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.795+0000 7f5de1ffb700 1 -- 172.21.15.98:0/2959525743 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc021070 con 0x7f5de4102ac0 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.795+0000 7f5deaf7a700 1 -- 172.21.15.98:0/2959525743 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de419e400 con 0x7f5de4102ac0 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.795+0000 7f5de1ffb700 1 -- 172.21.15.98:0/2959525743 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ddc007c80 con 0x7f5de4102ac0 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.795+0000 7f5de1ffb700 1 -- 172.21.15.98:0/2959525743 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc01d450 con 0x7f5de4102ac0 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.795+0000 7f5deaf7a700 1 -- 172.21.15.98:0/2959525743 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de41a5ce0 con 0x7f5de4102ac0 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.796+0000 7f5deaf7a700 1 -- 172.21.15.98:0/2959525743 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de4069730 con 0x7f5de4102ac0 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.797+0000 7f5de1ffb700 1 -- 172.21.15.98:0/2959525743 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5ddc004000 con 0x7f5de4102ac0 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.797+0000 7f5de1ffb700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5dcc0a3db0 0x7f5dcc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.798+0000 7f5de3fff700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5dcc0a3db0 0x7f5dcc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.798+0000 7f5de1ffb700 1 -- 172.21.15.98:0/2959525743 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f5ddc00f020 con 0x7f5de4102ac0 2024-04-25T17:43:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.798+0000 7f5de3fff700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5dcc0a3db0 0x7f5dcc0a6270 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5de407a1d0 tx=0x7f5dd8008040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:18.801+0000 7f5de1ffb700 1 -- 172.21.15.98:0/2959525743 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ddc02a020 con 0x7f5de4102ac0 2024-04-25T17:43:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 -- 172.21.15.98:0/2730934357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75241031e0 msgr2=0x7f75241035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 --2- 172.21.15.98:0/2730934357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75241031e0 0x7f75241035c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7514009900 tx=0x7f751402a510 comp rx=0 tx=0).stop 2024-04-25T17:43:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 -- 172.21.15.98:0/2730934357 shutdown_connections 2024-04-25T17:43:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 --2- 172.21.15.98:0/2730934357 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752410c520 0x7f752410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 --2- 172.21.15.98:0/2730934357 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7524103b00 0x7f752410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 --2- 172.21.15.98:0/2730934357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f75241031e0 0x7f75241035c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 -- 172.21.15.98:0/2730934357 >> 172.21.15.98:0/2730934357 conn(0x7f7524077fd0 msgr2=0x7f75240783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:18.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 -- 172.21.15.98:0/2730934357 shutdown_connections 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 -- 172.21.15.98:0/2730934357 wait complete. 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 Processor -- start 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 -- start start 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75241031e0 0x7f75241986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.920+0000 7f7529536640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7524103b00 0x7f7524198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f7529536640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752410c520 0x7f752419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f7529536640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75241121b0 con 0x7f7524103b00 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f7529536640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7524112030 con 0x7f75241031e0 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f7529536640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7524112330 con 0x7f752410c520 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f751bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7524103b00 0x7f7524198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f751bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7524103b00 0x7f7524198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47266/0 (socket says 172.21.15.98:47266) 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f751bfff640 1 -- 172.21.15.98:0/1438015872 learned_addr learned my addr 172.21.15.98:0/1438015872 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f7522ffd640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75241031e0 0x7f75241986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f75237fe640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752410c520 0x7f752419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f751bfff640 1 -- 172.21.15.98:0/1438015872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752410c520 msgr2=0x7f752419cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f751bfff640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752410c520 0x7f752419cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f751bfff640 1 -- 172.21.15.98:0/1438015872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75241031e0 msgr2=0x7f75241986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f751bfff640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75241031e0 0x7f75241986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f751bfff640 1 -- 172.21.15.98:0/1438015872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752419d6a0 con 0x7f7524103b00 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f751bfff640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7524103b00 0x7f7524198bf0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f750c00d950 tx=0x7f750c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f7520ff9640 1 -- 172.21.15.98:0/1438015872 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750c017070 con 0x7f7524103b00 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f7529536640 1 -- 172.21.15.98:0/1438015872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f752419d990 con 0x7f7524103b00 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f7520ff9640 1 -- 172.21.15.98:0/1438015872 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f750c009c30 con 0x7f7524103b00 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f7520ff9640 1 -- 172.21.15.98:0/1438015872 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750c00f620 con 0x7f7524103b00 2024-04-25T17:43:18.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.921+0000 7f7529536640 1 -- 172.21.15.98:0/1438015872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75241a5220 con 0x7f7524103b00 2024-04-25T17:43:18.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.923+0000 7f7520ff9640 1 -- 172.21.15.98:0/1438015872 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f750c00f7c0 con 0x7f7524103b00 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f752c251740 1 Processor -- start 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f752c251740 1 -- start start 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f752c251740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849ee620 0x5601849eea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f752c251740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849eef40 0x5601849f92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f752c251740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849f9800 0x5601849fbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f752c251740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560184953970 con 0x5601849eef40 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f752c251740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5601848f04a0 con 0x5601849ee620 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f752c251740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56018494f930 con 0x5601849f9800 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f7520ff9640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75100a3dc0 0x7f75100a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f7522ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849eef40 0x5601849f92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f7522ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849eef40 0x5601849f92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47274/0 (socket says 172.21.15.98:47274) 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f7522ffd640 1 -- 172.21.15.98:0/3517126885 learned_addr learned my addr 172.21.15.98:0/3517126885 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f75237fe640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75100a3dc0 0x7f75100a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f75237fe640 1 --2- 172.21.15.98:0/3517126885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849ee620 0x5601849eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f7520ff9640 1 -- 172.21.15.98:0/1438015872 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f750c0d0440 con 0x7f7524103b00 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.924+0000 7f751bfff640 1 --2- 172.21.15.98:0/3517126885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849f9800 0x5601849fbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.925+0000 7f7522ffd640 1 -- 172.21.15.98:0/3517126885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849f9800 msgr2=0x5601849fbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.925+0000 7f7522ffd640 1 --2- 172.21.15.98:0/3517126885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849f9800 0x5601849fbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.925+0000 7f7522ffd640 1 -- 172.21.15.98:0/3517126885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849ee620 msgr2=0x5601849eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.925+0000 7f7522ffd640 1 --2- 172.21.15.98:0/3517126885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849ee620 0x5601849eea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.925+0000 7f7522ffd640 1 -- 172.21.15.98:0/3517126885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560184666210 con 0x5601849eef40 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.925+0000 7f75237fe640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75100a3dc0 0x7f75100a6280 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f751c005c40 tx=0x7f751c009c90 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.925+0000 7f7522ffd640 1 --2- 172.21.15.98:0/3517126885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849eef40 0x5601849f92c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7514009900 tx=0x7f7514032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c5a5c8cef8d39f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.925+0000 7f75197fa640 1 -- 172.21.15.98:0/3517126885 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7514040070 con 0x5601849eef40 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.925+0000 7f75197fa640 1 -- 172.21.15.98:0/3517126885 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7514004030 con 0x5601849eef40 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.925+0000 7f75197fa640 1 -- 172.21.15.98:0/3517126885 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7514035040 con 0x5601849eef40 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 -- 172.21.15.98:0/3517126885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849eef40 msgr2=0x5601849f92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 --2- 172.21.15.98:0/3517126885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849eef40 0x5601849f92c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f7514009900 tx=0x7f7514032bd0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 -- 172.21.15.98:0/3517126885 shutdown_connections 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 --2- 172.21.15.98:0/3517126885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849f9800 0x5601849fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 --2- 172.21.15.98:0/3517126885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849eef40 0x5601849f92c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 --2- 172.21.15.98:0/3517126885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849ee620 0x5601849eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 -- 172.21.15.98:0/3517126885 >> 172.21.15.98:0/3517126885 conn(0x5601849eab30 msgr2=0x5601849eaf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 -- 172.21.15.98:0/3517126885 shutdown_connections 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 -- 172.21.15.98:0/3517126885 wait complete. 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 -- start start 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 Processor -- start 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849ee620 0x5601849f90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849eef40 0x5601849f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849f9800 0x5601849f6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5601849fc3d0 con 0x5601849ee620 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5601848ee0b0 con 0x5601849eef40 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.926+0000 7f752c251740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5601849fc550 con 0x5601849f9800 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f752c251740 10 client.0 _ll_register_callbacks cb 0x5601848dbd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f7522ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849ee620 0x5601849f90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f7522ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849ee620 0x5601849f90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47280/0 (socket says 172.21.15.98:47280) 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f7522ffd640 1 -- 172.21.15.98:0/2306343143 learned_addr learned my addr 172.21.15.98:0/2306343143 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f75237fe640 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849f9800 0x5601849f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f7522ffd640 1 -- 172.21.15.98:0/2306343143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849f9800 msgr2=0x5601849f6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f7522ffd640 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849f9800 0x5601849f6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f7522ffd640 1 -- 172.21.15.98:0/2306343143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849eef40 msgr2=0x5601849f63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f7522ffd640 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849eef40 0x5601849f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f7522ffd640 1 -- 172.21.15.98:0/2306343143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560184732620 con 0x5601849ee620 2024-04-25T17:43:18.937 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f7522ffd640 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849ee620 0x5601849f90d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f751402aa30 tx=0x7f7514009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7b2f9a9afa8f3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f750a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f750a7fc640 1 -- 172.21.15.98:0/2306343143 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751404c020 con 0x5601849ee620 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f752c251740 1 -- 172.21.15.98:0/2306343143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7514004030 con 0x5601849ee620 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f750a7fc640 1 -- 172.21.15.98:0/2306343143 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f7514005430 con 0x5601849ee620 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f750a7fc640 1 -- 172.21.15.98:0/2306343143 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7514057050 con 0x5601849ee620 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f752c251740 10 client.16089 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f752c251740 1 -- 172.21.15.98:0/2306343143 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560184a0c680 con 0x5601849ee620 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.927+0000 7f750a7fc640 1 -- 172.21.15.98:0/2306343143 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7514004030 con 0x5601849ee620 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.928+0000 7f750a7fc640 1 -- 172.21.15.98:0/2306343143 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f7514004bd0 con 0x5601849ee620 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.928+0000 7f752c251740 10 client.16089 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.928+0000 7f752c251740 10 client.16089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.928+0000 7f752c251740 20 client.16089 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.928+0000 7f752c251740 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560184a0d9d0 0x560184a0fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.928+0000 7f752c251740 4 client.16089 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.928+0000 7f752c251740 1 -- 172.21.15.98:0/2306343143 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560184961390 con 0x560184a0d9d0 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.928+0000 7f751bfff640 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560184a0d9d0 0x560184a0fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.928+0000 7f750a7fc640 10 client.16089 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.928+0000 7f751bfff640 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560184a0d9d0 0x560184a0fdc0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3417348eee9b172 server_cookie=a70e826c36f1afc4 in_seq=0 out_seq=0 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f750a7fc640 1 -- 172.21.15.98:0/2306343143 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560184961390 con 0x560184a0d9d0 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f750a7fc640 10 client.16089 handle_command_reply: tid=0 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 client.16089 shutdown 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 -- 172.21.15.98:0/2306343143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849ee620 msgr2=0x5601849f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849ee620 0x5601849f90d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f751402aa30 tx=0x7f7514009620 comp rx=0 tx=0).stop 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 -- 172.21.15.98:0/2306343143 shutdown_connections 2024-04-25T17:43:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560184a0d9d0 0x560184a0fdc0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849f9800 0x5601849f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849eef40 0x5601849f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 --2- 172.21.15.98:0/2306343143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849ee620 0x5601849f90d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 -- 172.21.15.98:0/2306343143 >> 172.21.15.98:0/2306343143 conn(0x5601849eab30 msgr2=0x5601849f08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f750a7fc640 0 client.16089 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 -- 172.21.15.98:0/2306343143 shutdown_connections 2024-04-25T17:43:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 1 -- 172.21.15.98:0/2306343143 wait complete. 2024-04-25T17:43:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.929+0000 7f752c251740 20 client.16089 trim_cache size 0 max 16384 2024-04-25T17:43:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f752c251740 1 Processor -- start 2024-04-25T17:43:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f752c251740 1 -- start start 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f752c251740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849ee620 0x5601849f8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f752c251740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849eef40 0x560184a154e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f752c251740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a15a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f752c251740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75241ae340 con 0x5601849f9800 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f752c251740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f75241abb50 con 0x5601849eef40 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f752c251740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75241a99c0 con 0x5601849ee620 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f75237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a15a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f75237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a15a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47290/0 (socket says 172.21.15.98:47290) 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f75237fe640 1 -- 172.21.15.98:0/2844859123 learned_addr learned my addr 172.21.15.98:0/2844859123 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f7522ffd640 1 --2- 172.21.15.98:0/2844859123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849ee620 0x5601849f8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f751bfff640 1 --2- 172.21.15.98:0/2844859123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849eef40 0x560184a154e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f75237fe640 1 -- 172.21.15.98:0/2844859123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849ee620 msgr2=0x5601849f8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f75237fe640 1 --2- 172.21.15.98:0/2844859123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849ee620 0x5601849f8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.944+0000 7f75237fe640 1 -- 172.21.15.98:0/2844859123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849eef40 msgr2=0x560184a154e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.945+0000 7f75237fe640 1 --2- 172.21.15.98:0/2844859123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849eef40 0x560184a154e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.945+0000 7f75237fe640 1 -- 172.21.15.98:0/2844859123 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601849ebb30 con 0x5601849f9800 2024-04-25T17:43:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.945+0000 7f751bfff640 1 --2- 172.21.15.98:0/2844859123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849eef40 0x560184a154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:43:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.945+0000 7f75237fe640 1 --2- 172.21.15.98:0/2844859123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a15a20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f751c009b60 tx=0x7f751c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e385943a1f7a4248 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.945+0000 7f7518ff9640 1 -- 172.21.15.98:0/2844859123 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c02c480 con 0x5601849f9800 2024-04-25T17:43:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.945+0000 7f7518ff9640 1 -- 172.21.15.98:0/2844859123 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f751c02cad0 con 0x5601849f9800 2024-04-25T17:43:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.945+0000 7f7518ff9640 1 -- 172.21.15.98:0/2844859123 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c021b60 con 0x5601849f9800 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 -- 172.21.15.98:0/2844859123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 msgr2=0x560184a15a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 --2- 172.21.15.98:0/2844859123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a15a20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f751c009b60 tx=0x7f751c01f000 comp rx=0 tx=0).stop 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 -- 172.21.15.98:0/2844859123 shutdown_connections 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 --2- 172.21.15.98:0/2844859123 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a15a20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 --2- 172.21.15.98:0/2844859123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849eef40 0x560184a154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 --2- 172.21.15.98:0/2844859123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849ee620 0x5601849f8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 -- 172.21.15.98:0/2844859123 >> 172.21.15.98:0/2844859123 conn(0x5601849eab30 msgr2=0x5601849f6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 -- 172.21.15.98:0/2844859123 shutdown_connections 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 -- 172.21.15.98:0/2844859123 wait complete. 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 -- start start 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 Processor -- start 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849ee620 0x560184a05900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849eef40 0x560184a05e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a00d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f75241aa010 con 0x5601849f9800 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f75241ab480 con 0x5601849ee620 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75241aa6e0 con 0x5601849eef40 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f752c251740 10 client.0 _ll_register_callbacks cb 0x7f7514032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f75237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a00d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f75237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a00d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47294/0 (socket says 172.21.15.98:47294) 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f75237fe640 1 -- 172.21.15.98:0/3416459869 learned_addr learned my addr 172.21.15.98:0/3416459869 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f7522ffd640 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849ee620 0x560184a05900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.946+0000 7f751bfff640 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849eef40 0x560184a05e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f75237fe640 1 -- 172.21.15.98:0/3416459869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849eef40 msgr2=0x560184a05e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f75237fe640 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849eef40 0x560184a05e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f75237fe640 1 -- 172.21.15.98:0/3416459869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849ee620 msgr2=0x560184a05900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f75237fe640 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849ee620 0x560184a05900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f75237fe640 1 -- 172.21.15.98:0/3416459869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560184a01500 con 0x5601849f9800 2024-04-25T17:43:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f75237fe640 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a00d90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f751c021740 tx=0x7f751c02c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be08d026f112f064 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f750a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:43:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f750a7fc640 1 -- 172.21.15.98:0/3416459869 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c02c900 con 0x5601849f9800 2024-04-25T17:43:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f750a7fc640 1 -- 172.21.15.98:0/3416459869 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f751c004b20 con 0x5601849f9800 2024-04-25T17:43:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f752c251740 1 -- 172.21.15.98:0/3416459869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f751c02cad0 con 0x5601849f9800 2024-04-25T17:43:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f750a7fc640 1 -- 172.21.15.98:0/3416459869 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f751c004d40 con 0x5601849f9800 2024-04-25T17:43:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f752c251740 10 client.16095 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f752c251740 1 -- 172.21.15.98:0/3416459869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560184a73730 con 0x5601849f9800 2024-04-25T17:43:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f750a7fc640 1 -- 172.21.15.98:0/3416459869 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f751c02cad0 con 0x5601849f9800 2024-04-25T17:43:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f750a7fc640 1 -- 172.21.15.98:0/3416459869 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f751c01f070 con 0x5601849f9800 2024-04-25T17:43:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f752c251740 10 client.16095 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f752c251740 10 client.16095 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:43:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f752c251740 20 client.16095 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f752c251740 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560184a73e50 0x560184a76270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f752c251740 4 client.16095 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:43:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.947+0000 7f752c251740 1 -- 172.21.15.98:0/3416459869 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56018495b200 con 0x560184a73e50 2024-04-25T17:43:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.948+0000 7f7522ffd640 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560184a73e50 0x560184a76270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.948+0000 7f750a7fc640 10 client.16095 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:18.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.948+0000 7f7522ffd640 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560184a73e50 0x560184a76270 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=654ccfee169c94f3 server_cookie=d73a6b139bb15b65 in_seq=0 out_seq=0 2024-04-25T17:43:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f750a7fc640 1 -- 172.21.15.98:0/3416459869 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56018495b200 con 0x560184a73e50 2024-04-25T17:43:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f750a7fc640 10 client.16095 handle_command_reply: tid=0 2024-04-25T17:43:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 client.16095 shutdown 2024-04-25T17:43:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 -- 172.21.15.98:0/3416459869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 msgr2=0x560184a00d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a00d90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f751c021740 tx=0x7f751c02c480 comp rx=0 tx=0).stop 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 -- 172.21.15.98:0/3416459869 shutdown_connections 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560184a73e50 0x560184a76270 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5601849f9800 0x560184a00d90 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f750a7fc640 0 client.16095 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5601849eef40 0x560184a05e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 --2- 172.21.15.98:0/3416459869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5601849ee620 0x560184a05900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 -- 172.21.15.98:0/3416459869 >> 172.21.15.98:0/3416459869 conn(0x5601849eab30 msgr2=0x5601849f1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 -- 172.21.15.98:0/3416459869 shutdown_connections 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 1 -- 172.21.15.98:0/3416459869 wait complete. 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.961+0000 7f752c251740 20 client.16095 trim_cache size 0 max 16384 2024-04-25T17:43:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:43:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:43:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:43:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 -- 172.21.15.98:0/1438015872 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75100a3dc0 msgr2=0x7f75100a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75100a3dc0 0x7f75100a6280 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f751c005c40 tx=0x7f751c009c90 comp rx=0 tx=0).stop 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 -- 172.21.15.98:0/1438015872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7524103b00 msgr2=0x7f7524198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7524103b00 0x7f7524198bf0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f750c00d950 tx=0x7f750c00de10 comp rx=0 tx=0).stop 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 -- 172.21.15.98:0/1438015872 shutdown_connections 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75100a3dc0 0x7f75100a6280 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f752410c520 0x7f752419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7524103b00 0x7f7524198bf0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 --2- 172.21.15.98:0/1438015872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75241031e0 0x7f75241986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 -- 172.21.15.98:0/1438015872 >> 172.21.15.98:0/1438015872 conn(0x7f7524077fd0 msgr2=0x7f75240fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 -- 172.21.15.98:0/1438015872 shutdown_connections 2024-04-25T17:43:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:18.962+0000 7f7529536640 1 -- 172.21.15.98:0/1438015872 wait complete. 2024-04-25T17:43:19.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.058+0000 7f5deaf7a700 1 -- 172.21.15.98:0/2959525743 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5de402df80 con 0x7f5de4102ac0 2024-04-25T17:43:19.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.059+0000 7f5de1ffb700 1 -- 172.21.15.98:0/2959525743 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7f5ddc09e910 con 0x7f5de4102ac0 2024-04-25T17:43:19.059 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:43:19.059 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:42:52.734463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:43:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 -- 172.21.15.98:0/2959525743 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5dcc0a3db0 msgr2=0x7f5dcc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5dcc0a3db0 0x7f5dcc0a6270 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5de407a1d0 tx=0x7f5dd8008040 comp rx=0 tx=0).stop 2024-04-25T17:43:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 -- 172.21.15.98:0/2959525743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de4102ac0 msgr2=0x7f5de4199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de4102ac0 0x7f5de4199160 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00b070 tx=0x7f5ddc004db0 comp rx=0 tx=0).stop 2024-04-25T17:43:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 -- 172.21.15.98:0/2959525743 shutdown_connections 2024-04-25T17:43:19.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5dcc0a3db0 0x7f5dcc0a6270 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de4108a70 0x7f5de419da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de4102ac0 0x7f5de4199160 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 --2- 172.21.15.98:0/2959525743 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de4103460 0x7f5de41996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 -- 172.21.15.98:0/2959525743 >> 172.21.15.98:0/2959525743 conn(0x7f5de4079bf0 msgr2=0x7f5de4100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 -- 172.21.15.98:0/2959525743 shutdown_connections 2024-04-25T17:43:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:19.061+0000 7f5deaf7a700 1 -- 172.21.15.98:0/2959525743 wait complete. 2024-04-25T17:43:19.062 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 38 2024-04-25T17:43:19.353 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:19 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2959525743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:43:19.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:19 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2959525743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:43:19.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:19 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2959525743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:43:19.691 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id 8b98b092 --release --await' 2024-04-25T17:43:19.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id 8b98b092 --release --await 2024-04-25T17:43:19.938 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:43:20.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:20 smithi098 ceph-mon[29192]: pgmap v230: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 MiB/s wr, 637 op/s 2024-04-25T17:43:20.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:20 smithi203 ceph-mon[32834]: pgmap v230: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 MiB/s wr, 637 op/s 2024-04-25T17:43:20.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:20 smithi173 ceph-mon[32856]: pgmap v230: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 MiB/s wr, 637 op/s 2024-04-25T17:43:21.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 -- 172.21.15.98:0/231270450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b44101970 msgr2=0x7f9b44101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:21.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 --2- 172.21.15.98:0/231270450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b44101970 0x7f9b44101dd0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009990 tx=0x7f9b38009ca0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 -- 172.21.15.98:0/231270450 shutdown_connections 2024-04-25T17:43:21.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 --2- 172.21.15.98:0/231270450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b44107920 0x7f9b44107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 --2- 172.21.15.98:0/231270450 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44102310 0x7f9b4410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 --2- 172.21.15.98:0/231270450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b44101970 0x7f9b44101dd0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 -- 172.21.15.98:0/231270450 >> 172.21.15.98:0/231270450 conn(0x7f9b440788e0 msgr2=0x7f9b44078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:21.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 -- 172.21.15.98:0/231270450 shutdown_connections 2024-04-25T17:43:21.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 -- 172.21.15.98:0/231270450 wait complete. 2024-04-25T17:43:21.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 Processor -- start 2024-04-25T17:43:21.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.584+0000 7f9b4a717700 1 -- start start 2024-04-25T17:43:21.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b4a717700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101970 0x7f9b44197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b4a717700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b44102310 0x7f9b44198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b4a717700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b44107920 0x7f9b4419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b4a717700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b44111b60 con 0x7f9b44102310 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b4a717700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441119e0 con 0x7f9b44101970 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b4a717700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b44111ce0 con 0x7f9b44107920 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b44102310 0x7f9b44198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b44102310 0x7f9b44198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47302/0 (socket says 172.21.15.98:47302) 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b437fe700 1 -- 172.21.15.98:0/709500904 learned_addr learned my addr 172.21.15.98:0/709500904 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b43fff700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101970 0x7f9b44197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b48cb4700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b44107920 0x7f9b4419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b437fe700 1 -- 172.21.15.98:0/709500904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b44107920 msgr2=0x7f9b4419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b437fe700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b44107920 0x7f9b4419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b437fe700 1 -- 172.21.15.98:0/709500904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101970 msgr2=0x7f9b44197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:21.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b437fe700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101970 0x7f9b44197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b437fe700 1 -- 172.21.15.98:0/709500904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4419cfe0 con 0x7f9b44102310 2024-04-25T17:43:21.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.585+0000 7f9b437fe700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b44102310 0x7f9b44198510 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9b380057f0 tx=0x7f9b38003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:21.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.586+0000 7f9b417fa700 1 -- 172.21.15.98:0/709500904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38027070 con 0x7f9b44102310 2024-04-25T17:43:21.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.586+0000 7f9b417fa700 1 -- 172.21.15.98:0/709500904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b380040a0 con 0x7f9b44102310 2024-04-25T17:43:21.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.586+0000 7f9b4a717700 1 -- 172.21.15.98:0/709500904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4419d270 con 0x7f9b44102310 2024-04-25T17:43:21.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.586+0000 7f9b417fa700 1 -- 172.21.15.98:0/709500904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38022dc0 con 0x7f9b44102310 2024-04-25T17:43:21.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.586+0000 7f9b4a717700 1 -- 172.21.15.98:0/709500904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b441a4b50 con 0x7f9b44102310 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.588+0000 7f9b417fa700 1 -- 172.21.15.98:0/709500904 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9b3802a070 con 0x7f9b44102310 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.588+0000 7f9b417fa700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9b2c0a3db0 0x7f9b2c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.589+0000 7f9b43fff700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9b2c0a3db0 0x7f9b2c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.589+0000 7f9b417fa700 1 -- 172.21.15.98:0/709500904 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f9b380d16b0 con 0x7f9b44102310 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.589+0000 7f9b43fff700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9b2c0a3db0 0x7f9b2c0a6270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400ac50 tx=0x7f9b3400a380 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.589+0000 7f9b54fa0b80 1 Processor -- start 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.589+0000 7f9b54fa0b80 1 -- start start 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.589+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 0x55ed19315dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.589+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed19316310 0x55ed19322eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed193257e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ed19268e30 con 0x55ed193233f0 2024-04-25T17:43:21.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ed1920fa60 con 0x55ed193159f0 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ed19264df0 con 0x55ed19316310 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b48cb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed193257e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b48cb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed193257e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47310/0 (socket says 172.21.15.98:47310) 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b48cb4700 1 -- 172.21.15.98:0/1990490328 learned_addr learned my addr 172.21.15.98:0/1990490328 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b437fe700 1 --2- 172.21.15.98:0/1990490328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed19316310 0x55ed19322eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b43fff700 1 --2- 172.21.15.98:0/1990490328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 0x55ed19315dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b48cb4700 1 -- 172.21.15.98:0/1990490328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed19316310 msgr2=0x55ed19322eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b48cb4700 1 --2- 172.21.15.98:0/1990490328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed19316310 0x55ed19322eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b48cb4700 1 -- 172.21.15.98:0/1990490328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 msgr2=0x55ed19315dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b48cb4700 1 --2- 172.21.15.98:0/1990490328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 0x55ed19315dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b48cb4700 1 -- 172.21.15.98:0/1990490328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed191b69d0 con 0x55ed193233f0 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b48cb4700 1 --2- 172.21.15.98:0/1990490328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed193257e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009f20 tx=0x7f9b3c009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10952ed4221657f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b2a7fc700 1 -- 172.21.15.98:0/1990490328 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c01a070 con 0x55ed193233f0 2024-04-25T17:43:21.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b2a7fc700 1 -- 172.21.15.98:0/1990490328 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c015410 con 0x55ed193233f0 2024-04-25T17:43:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.590+0000 7f9b2a7fc700 1 -- 172.21.15.98:0/1990490328 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c00a660 con 0x55ed193233f0 2024-04-25T17:43:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.591+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1990490328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 msgr2=0x55ed193257e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.591+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/1990490328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed193257e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009f20 tx=0x7f9b3c009f50 comp rx=0 tx=0).stop 2024-04-25T17:43:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.591+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1990490328 shutdown_connections 2024-04-25T17:43:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.591+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/1990490328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed193257e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.591+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/1990490328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 0x55ed19315dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.591+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/1990490328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed19316310 0x55ed19322eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.591+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1990490328 >> 172.21.15.98:0/1990490328 conn(0x55ed19243790 msgr2=0x55ed19311ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.591+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1990490328 shutdown_connections 2024-04-25T17:43:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.591+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1990490328 wait complete. 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.591+0000 7f9b54fa0b80 1 -- start start 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b54fa0b80 1 Processor -- start 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed193159f0 0x55ed19322b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed1931fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193233f0 0x55ed193203b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ed19325de0 con 0x55ed19316310 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ed19207cd0 con 0x55ed193233f0 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ed19325f60 con 0x55ed193159f0 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b54fa0b80 10 client.0 _ll_register_callbacks cb 0x55ed19201a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed1931fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed1931fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47314/0 (socket says 172.21.15.98:47314) 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b437fe700 1 -- 172.21.15.98:0/2797910980 learned_addr learned my addr 172.21.15.98:0/2797910980 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b48cb4700 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193233f0 0x55ed193203b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b43fff700 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed193159f0 0x55ed19322b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b437fe700 1 -- 172.21.15.98:0/2797910980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed193159f0 msgr2=0x55ed19322b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b437fe700 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed193159f0 0x55ed19322b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b437fe700 1 -- 172.21.15.98:0/2797910980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193233f0 msgr2=0x55ed193203b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b437fe700 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193233f0 0x55ed193203b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b437fe700 1 -- 172.21.15.98:0/2797910980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed19220370 con 0x55ed19316310 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b437fe700 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed1931fe70 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9b38099b40 tx=0x7f9b3809d550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8f4aa5b49ef558 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.592+0000 7f9b237fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b237fe700 1 -- 172.21.15.98:0/2797910980 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3809db40 con 0x55ed19316310 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b237fe700 1 -- 172.21.15.98:0/2797910980 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f9b380a5e30 con 0x55ed19316310 2024-04-25T17:43:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/2797910980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3c015410 con 0x55ed19316310 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b237fe700 1 -- 172.21.15.98:0/2797910980 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9b380880d0 con 0x55ed19316310 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b54fa0b80 10 client.16107 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/2797910980 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ed19330650 con 0x55ed19316310 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b237fe700 1 -- 172.21.15.98:0/2797910980 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c015410 con 0x55ed19316310 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b237fe700 1 -- 172.21.15.98:0/2797910980 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f9b3809cb30 con 0x55ed19316310 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b54fa0b80 10 client.16107 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b54fa0b80 10 client.16107 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b54fa0b80 20 client.16107 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ed19332c70 0x55ed19335090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b54fa0b80 4 client.16107 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/2797910980 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed19276620 con 0x55ed19332c70 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.593+0000 7f9b43fff700 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ed19332c70 0x55ed19335090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.594+0000 7f9b237fe700 10 client.16107 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.594+0000 7f9b43fff700 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ed19332c70 0x55ed19335090 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a591d18c7a642f23 server_cookie=119d664022d2a238 in_seq=0 out_seq=0 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.594+0000 7f9b237fe700 1 -- 172.21.15.98:0/2797910980 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ed19276620 con 0x55ed19332c70 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.594+0000 7f9b237fe700 10 client.16107 handle_command_reply: tid=0 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 client.16107 shutdown 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/2797910980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 msgr2=0x55ed1931fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed1931fe70 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9b38099b40 tx=0x7f9b3809d550 comp rx=0 tx=0).stop 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/2797910980 shutdown_connections 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ed19332c70 0x55ed19335090 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed193159f0 0x55ed19322b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193233f0 0x55ed193203b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/2797910980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed1931fe70 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/2797910980 >> 172.21.15.98:0/2797910980 conn(0x55ed19243790 msgr2=0x55ed19311b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b237fe700 0 client.16107 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/2797910980 shutdown_connections 2024-04-25T17:43:21.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/2797910980 wait complete. 2024-04-25T17:43:21.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.595+0000 7f9b54fa0b80 20 client.16107 trim_cache size 0 max 16384 2024-04-25T17:43:21.601 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 410.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:43:21.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b54fa0b80 1 Processor -- start 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b54fa0b80 1 -- start start 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 0x55ed19311fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed19316310 0x55ed1931b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed1932eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441ad950 con 0x55ed193233f0 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441ab120 con 0x55ed193159f0 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441a8f90 con 0x55ed19316310 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b48cb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed1932eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b48cb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed1932eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47322/0 (socket says 172.21.15.98:47322) 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b48cb4700 1 -- 172.21.15.98:0/438650645 learned_addr learned my addr 172.21.15.98:0/438650645 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b43fff700 1 --2- 172.21.15.98:0/438650645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 0x55ed19311fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b437fe700 1 --2- 172.21.15.98:0/438650645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed19316310 0x55ed1931b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b48cb4700 1 -- 172.21.15.98:0/438650645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed19316310 msgr2=0x55ed1931b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b48cb4700 1 --2- 172.21.15.98:0/438650645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed19316310 0x55ed1931b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b48cb4700 1 -- 172.21.15.98:0/438650645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 msgr2=0x55ed19311fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b48cb4700 1 --2- 172.21.15.98:0/438650645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 0x55ed19311fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b48cb4700 1 -- 172.21.15.98:0/438650645 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed19322ed0 con 0x55ed193233f0 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.614+0000 7f9b43fff700 1 --2- 172.21.15.98:0/438650645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 0x55ed19311fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b48cb4700 1 --2- 172.21.15.98:0/438650645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed1932eca0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009f20 tx=0x7f9b3c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9d81e9df02aaa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b29ffb700 1 -- 172.21.15.98:0/438650645 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c01a040 con 0x55ed193233f0 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b29ffb700 1 -- 172.21.15.98:0/438650645 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c0158c0 con 0x55ed193233f0 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b29ffb700 1 -- 172.21.15.98:0/438650645 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c01f790 con 0x55ed193233f0 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/438650645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 msgr2=0x55ed1932eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/438650645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed1932eca0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009f20 tx=0x7f9b3c00cc20 comp rx=0 tx=0).stop 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/438650645 shutdown_connections 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/438650645 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed193233f0 0x55ed1932eca0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/438650645 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193159f0 0x55ed19311fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/438650645 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed19316310 0x55ed1931b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/438650645 >> 172.21.15.98:0/438650645 conn(0x55ed19243790 msgr2=0x55ed193216a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/438650645 shutdown_connections 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.615+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/438650645 wait complete. 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b54fa0b80 1 -- start start 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b54fa0b80 1 Processor -- start 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed193159f0 0x55ed1932b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed19328180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b54fa0b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193233f0 0x55ed193286c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441a95e0 con 0x55ed19316310 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441aaa50 con 0x55ed193233f0 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b54fa0b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b441a9cb0 con 0x55ed193159f0 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b54fa0b80 10 client.0 _ll_register_callbacks cb 0x55ed19201a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:21.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed19328180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed19328180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47338/0 (socket says 172.21.15.98:47338) 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.616+0000 7f9b437fe700 1 -- 172.21.15.98:0/1702844849 learned_addr learned my addr 172.21.15.98:0/1702844849 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b43fff700 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed193159f0 0x55ed1932b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b437fe700 1 -- 172.21.15.98:0/1702844849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed193159f0 msgr2=0x55ed1932b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b437fe700 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed193159f0 0x55ed1932b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b437fe700 1 -- 172.21.15.98:0/1702844849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193233f0 msgr2=0x55ed193286c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b437fe700 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193233f0 0x55ed193286c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b437fe700 1 -- 172.21.15.98:0/1702844849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed19328c00 con 0x55ed19316310 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b437fe700 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed19328180 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9b380cf400 tx=0x7f9b3809dce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3fc6daa73195cb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b237fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b237fe700 1 -- 172.21.15.98:0/1702844849 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38066030 con 0x55ed19316310 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b237fe700 1 -- 172.21.15.98:0/1702844849 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f9b3809d940 con 0x55ed19316310 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1702844849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3c0158c0 con 0x55ed19316310 2024-04-25T17:43:21.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b237fe700 1 -- 172.21.15.98:0/1702844849 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f9b380880d0 con 0x55ed19316310 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b54fa0b80 10 client.16113 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1702844849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ed19399f90 con 0x55ed19316310 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b237fe700 1 -- 172.21.15.98:0/1702844849 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c0158c0 con 0x55ed19316310 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b237fe700 1 -- 172.21.15.98:0/1702844849 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f9b38099430 con 0x55ed19316310 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b54fa0b80 10 client.16113 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b54fa0b80 10 client.16113 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b54fa0b80 20 client.16113 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ed1939a4b0 0x55ed1939c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b54fa0b80 4 client.16113 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "8b98b092", "release": true, "await": true}] 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.617+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1702844849 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "8b98b092", "release": true, "await": true}) -- 0x55ed192706c0 con 0x55ed1939a4b0 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.618+0000 7f9b43fff700 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ed1939a4b0 0x55ed1939c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.618+0000 7f9b43fff700 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ed1939a4b0 0x55ed1939c8d0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97e37f3151db8027 server_cookie=489d62e5871b99a in_seq=0 out_seq=0 2024-04-25T17:43:21.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.618+0000 7f9b237fe700 10 client.16113 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:21.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b237fe700 1 -- 172.21.15.98:0/1702844849 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55ed192706c0 con 0x55ed1939a4b0 2024-04-25T17:43:21.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b237fe700 10 client.16113 handle_command_reply: tid=0 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 client.16113 shutdown 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1702844849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 msgr2=0x55ed19328180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed19328180 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9b380cf400 tx=0x7f9b3809dce0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1702844849 shutdown_connections 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ed1939a4b0 0x55ed1939c8d0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ed193159f0 0x55ed1932b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ed193233f0 0x55ed193286c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 --2- 172.21.15.98:0/1702844849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ed19316310 0x55ed19328180 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1702844849 >> 172.21.15.98:0/1702844849 conn(0x55ed19243790 msgr2=0x55ed1931b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b237fe700 0 client.16113 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1702844849 shutdown_connections 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.623+0000 7f9b54fa0b80 1 -- 172.21.15.98:0/1702844849 wait complete. 2024-04-25T17:43:21.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b54fa0b80 20 client.16113 trim_cache size 0 max 16384 2024-04-25T17:43:21.625 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:43:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 38, 2024-04-25T17:43:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:43:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 20, 2024-04-25T17:43:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:43:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "8b98b092": { 2024-04-25T17:43:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 20, 2024-04-25T17:43:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:43:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:43:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 73.1, 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:21.627 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:43:21.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 -- 172.21.15.98:0/709500904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9b2c0a3db0 msgr2=0x7f9b2c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:21.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9b2c0a3db0 0x7f9b2c0a6270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400ac50 tx=0x7f9b3400a380 comp rx=0 tx=0).stop 2024-04-25T17:43:21.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 -- 172.21.15.98:0/709500904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b44102310 msgr2=0x7f9b44198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:21.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b44102310 0x7f9b44198510 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9b380057f0 tx=0x7f9b38003680 comp rx=0 tx=0).stop 2024-04-25T17:43:21.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 -- 172.21.15.98:0/709500904 shutdown_connections 2024-04-25T17:43:21.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9b2c0a3db0 0x7f9b2c0a6270 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b44107920 0x7f9b4419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101970 0x7f9b44197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 --2- 172.21.15.98:0/709500904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b44102310 0x7f9b44198510 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:21.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 -- 172.21.15.98:0/709500904 >> 172.21.15.98:0/709500904 conn(0x7f9b440788e0 msgr2=0x7f9b440ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:21.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 -- 172.21.15.98:0/709500904 shutdown_connections 2024-04-25T17:43:21.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:21.624+0000 7f9b4a717700 1 -- 172.21.15.98:0/709500904 wait complete. 2024-04-25T17:43:22.383 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:22 smithi098 ceph-mon[29192]: pgmap v231: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 560 op/s 2024-04-25T17:43:22.420 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '8b98b092', seconds elapsed: 38.6 2024-04-25T17:43:22.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:22 smithi203 ceph-mon[32834]: pgmap v231: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 560 op/s 2024-04-25T17:43:22.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:22 smithi173 ceph-mon[32856]: pgmap v231: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 560 op/s 2024-04-25T17:43:24.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:24 smithi098 ceph-mon[29192]: pgmap v232: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 744 op/s 2024-04-25T17:43:24.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:24 smithi203 ceph-mon[32834]: pgmap v232: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 744 op/s 2024-04-25T17:43:24.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:24 smithi173 ceph-mon[32856]: pgmap v232: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 744 op/s 2024-04-25T17:43:26.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:26 smithi098 ceph-mon[29192]: pgmap v233: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 744 op/s 2024-04-25T17:43:26.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:26 smithi203 ceph-mon[32834]: pgmap v233: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 744 op/s 2024-04-25T17:43:26.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:26 smithi173 ceph-mon[32856]: pgmap v233: 97 pgs: 97 active+clean; 3.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 744 op/s 2024-04-25T17:43:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 415.0 s, 2682.7 tps, lat 267.795 ms stddev 2679.358 2024-04-25T17:43:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:28 smithi098 ceph-mon[29192]: pgmap v234: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 MiB/s wr, 895 op/s 2024-04-25T17:43:28.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:28 smithi203 ceph-mon[32834]: pgmap v234: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 MiB/s wr, 895 op/s 2024-04-25T17:43:28.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:28 smithi173 ceph-mon[32856]: pgmap v234: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 MiB/s wr, 895 op/s 2024-04-25T17:43:30.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:30 smithi098 ceph-mon[29192]: pgmap v235: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.3 MiB/s wr, 415 op/s 2024-04-25T17:43:30.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:30 smithi203 ceph-mon[32834]: pgmap v235: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.3 MiB/s wr, 415 op/s 2024-04-25T17:43:30.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:30 smithi173 ceph-mon[32856]: pgmap v235: 97 pgs: 97 active+clean; 3.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.3 MiB/s wr, 415 op/s 2024-04-25T17:43:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 420.0 s, 2835.2 tps, lat 35.466 ms stddev 37.656 2024-04-25T17:43:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.086+0000 7f62d96cb640 1 -- 172.21.15.98:0/236466295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62d411b730 msgr2=0x7f62d411db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.086+0000 7f62d96cb640 1 --2- 172.21.15.98:0/236466295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62d411b730 0x7f62d411db20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009930 tx=0x7f62c802a510 comp rx=0 tx=0).stop 2024-04-25T17:43:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.086+0000 7f62d96cb640 1 -- 172.21.15.98:0/236466295 shutdown_connections 2024-04-25T17:43:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.086+0000 7f62d96cb640 1 --2- 172.21.15.98:0/236466295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62d411b730 0x7f62d411db20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.086+0000 7f62d96cb640 1 --2- 172.21.15.98:0/236466295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62d4110b20 0x7f62d4109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.086+0000 7f62d96cb640 1 --2- 172.21.15.98:0/236466295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62d4110740 0x7f62d4108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.086+0000 7f62d96cb640 1 -- 172.21.15.98:0/236466295 >> 172.21.15.98:0/236466295 conn(0x7f62d406d020 msgr2=0x7f62d406d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.087+0000 7f62d96cb640 1 -- 172.21.15.98:0/236466295 shutdown_connections 2024-04-25T17:43:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.087+0000 7f62d96cb640 1 -- 172.21.15.98:0/236466295 wait complete. 2024-04-25T17:43:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d96cb640 1 Processor -- start 2024-04-25T17:43:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d96cb640 1 -- start start 2024-04-25T17:43:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d96cb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62d4110740 0x7f62d4118d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d96cb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62d411b730 0x7f62d4111da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d96cb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62d41122e0 0x7f62d4112790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d96cb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62d4120ff0 con 0x7f62d411b730 2024-04-25T17:43:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d96cb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f62d4120e70 con 0x7f62d4110740 2024-04-25T17:43:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d96cb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62d4121170 con 0x7f62d41122e0 2024-04-25T17:43:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d2ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62d4110740 0x7f62d4118d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d2ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62d4110740 0x7f62d4118d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:39370/0 (socket says 172.21.15.98:39370) 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62d411b730 0x7f62d4111da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.088+0000 7f62d2ffd640 1 -- 172.21.15.98:0/3099390704 learned_addr learned my addr 172.21.15.98:0/3099390704 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.089+0000 7f62d37fe640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62d41122e0 0x7f62d4112790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.089+0000 7f62d2ffd640 1 -- 172.21.15.98:0/3099390704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62d41122e0 msgr2=0x7f62d4112790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.089+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62d41122e0 0x7f62d4112790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.089+0000 7f62d2ffd640 1 -- 172.21.15.98:0/3099390704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62d411b730 msgr2=0x7f62d4111da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.089+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62d411b730 0x7f62d4111da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.089+0000 7f62d2ffd640 1 -- 172.21.15.98:0/3099390704 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d4116840 con 0x7f62d4110740 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.089+0000 7f62d27fc640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62d411b730 0x7f62d4111da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.089+0000 7f62d37fe640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62d41122e0 0x7f62d4112790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.089+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62d4110740 0x7f62d4118d20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62c400e9f0 tx=0x7f62c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.089+0000 7f62bbfff640 1 -- 172.21.15.98:0/3099390704 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c400cd80 con 0x7f62d4110740 2024-04-25T17:43:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.090+0000 7f62bbfff640 1 -- 172.21.15.98:0/3099390704 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c40108e0 con 0x7f62d4110740 2024-04-25T17:43:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.090+0000 7f62bbfff640 1 -- 172.21.15.98:0/3099390704 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4018980 con 0x7f62d4110740 2024-04-25T17:43:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.090+0000 7f62d96cb640 1 -- 172.21.15.98:0/3099390704 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d4116b30 con 0x7f62d4110740 2024-04-25T17:43:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.090+0000 7f62d96cb640 1 -- 172.21.15.98:0/3099390704 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d4116f70 con 0x7f62d4110740 2024-04-25T17:43:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.093+0000 7f62bbfff640 1 -- 172.21.15.98:0/3099390704 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f62c4018b20 con 0x7f62d4110740 2024-04-25T17:43:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62dc3db740 1 Processor -- start 2024-04-25T17:43:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62dc3db740 1 -- start start 2024-04-25T17:43:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564620 0x55ab7e564a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 0x55ab7e571bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62bbfff640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f62bc0a3e90 0x7f62bc0a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62dc3db740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab7e4c9970 con 0x55ab7e564f40 2024-04-25T17:43:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62dc3db740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ab7e4664a0 con 0x55ab7e564620 2024-04-25T17:43:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62dc3db740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ab7e4c5930 con 0x55ab7e56f800 2024-04-25T17:43:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62d37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 0x55ab7e571bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62d37fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 0x55ab7e571bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:55592/0 (socket says 172.21.15.98:55592) 2024-04-25T17:43:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62d27fc640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f62bc0a3e90 0x7f62bc0a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53544/0 (socket says 172.21.15.98:53544) 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62d37fe640 1 -- 172.21.15.98:0/4105917870 learned_addr learned my addr 172.21.15.98:0/4105917870 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.094+0000 7f62bbfff640 1 -- 172.21.15.98:0/3099390704 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f62c40d0bd0 con 0x7f62d4110740 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62d37fe640 1 --2- 172.21.15.98:0/4105917870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564620 0x55ab7e564a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62d2ffd640 1 -- 172.21.15.98:0/4105917870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 msgr2=0x55ab7e571bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/4105917870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 0x55ab7e571bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62d2ffd640 1 -- 172.21.15.98:0/4105917870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564620 msgr2=0x55ab7e564a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/4105917870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564620 0x55ab7e564a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62d2ffd640 1 -- 172.21.15.98:0/4105917870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab7e1dc210 con 0x55ab7e564f40 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/4105917870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56f2c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f62c40ce2c0 tx=0x7f62c406f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567b1d5bff9defe6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62d27fc640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f62bc0a3e90 0x7f62bc0a6350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009fd0 tx=0x7f62cc009680 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62b97fa640 1 -- 172.21.15.98:0/4105917870 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c409da80 con 0x55ab7e564f40 2024-04-25T17:43:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62b97fa640 1 -- 172.21.15.98:0/4105917870 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c40a1e30 con 0x55ab7e564f40 2024-04-25T17:43:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.095+0000 7f62b97fa640 1 -- 172.21.15.98:0/4105917870 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c40ce540 con 0x55ab7e564f40 2024-04-25T17:43:32.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.096+0000 7f62dc3db740 1 -- 172.21.15.98:0/4105917870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 msgr2=0x55ab7e56f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.096+0000 7f62dc3db740 1 --2- 172.21.15.98:0/4105917870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56f2c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f62c40ce2c0 tx=0x7f62c406f580 comp rx=0 tx=0).stop 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.096+0000 7f62dc3db740 1 -- 172.21.15.98:0/4105917870 shutdown_connections 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.096+0000 7f62dc3db740 1 --2- 172.21.15.98:0/4105917870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 0x55ab7e571bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.096+0000 7f62dc3db740 1 --2- 172.21.15.98:0/4105917870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56f2c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.096+0000 7f62dc3db740 1 --2- 172.21.15.98:0/4105917870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564620 0x55ab7e564a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.096+0000 7f62dc3db740 1 -- 172.21.15.98:0/4105917870 >> 172.21.15.98:0/4105917870 conn(0x55ab7e560b30 msgr2=0x55ab7e560f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.096+0000 7f62dc3db740 1 -- 172.21.15.98:0/4105917870 shutdown_connections 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.096+0000 7f62dc3db740 1 -- 172.21.15.98:0/4105917870 wait complete. 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62dc3db740 1 -- start start 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62dc3db740 1 Processor -- start 2024-04-25T17:43:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564620 0x55ab7e56f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e56f800 0x55ab7e56cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62dc3db740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab7e572370 con 0x55ab7e564f40 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62dc3db740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ab7e4640b0 con 0x55ab7e56f800 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62dc3db740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ab7e5724f0 con 0x55ab7e564620 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62dc3db740 10 client.0 _ll_register_callbacks cb 0x55ab7e451d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62d27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62d27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53548/0 (socket says 172.21.15.98:53548) 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62d27fc640 1 -- 172.21.15.98:0/2961423462 learned_addr learned my addr 172.21.15.98:0/2961423462 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564620 0x55ab7e56f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.097+0000 7f62d37fe640 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e56f800 0x55ab7e56cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62d2ffd640 1 -- 172.21.15.98:0/2961423462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e56f800 msgr2=0x55ab7e56cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e56f800 0x55ab7e56cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62d2ffd640 1 -- 172.21.15.98:0/2961423462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 msgr2=0x55ab7e56c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62d2ffd640 1 -- 172.21.15.98:0/2961423462 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab7e2a8620 con 0x55ab7e564620 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62d27fc640 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564620 0x55ab7e56f290 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f62c409bd30 tx=0x7f62c40ce2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2548d6b50d2ae48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62b6ff5640 1 -- 172.21.15.98:0/2961423462 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c40ce740 con 0x55ab7e564620 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62b6ff5640 1 -- 172.21.15.98:0/2961423462 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f62c40a6260 con 0x55ab7e564620 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62dc3db740 1 -- 172.21.15.98:0/2961423462 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c40a1e30 con 0x55ab7e564620 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62b6ff5640 1 -- 172.21.15.98:0/2961423462 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f62c409da80 con 0x55ab7e564620 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62dc3db740 10 client.25627 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62dc3db740 1 -- 172.21.15.98:0/2961423462 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ab7e572370 con 0x55ab7e564620 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.098+0000 7f62b6ff5640 1 -- 172.21.15.98:0/2961423462 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c40a1e30 con 0x55ab7e564620 2024-04-25T17:43:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.099+0000 7f62b6ff5640 1 -- 172.21.15.98:0/2961423462 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f62c4010430 con 0x55ab7e564620 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.099+0000 7f62dc3db740 10 client.25627 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.099+0000 7f62dc3db740 10 client.25627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.099+0000 7f62dc3db740 20 client.25627 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.099+0000 7f62dc3db740 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab7e57f540 0x55ab7e581930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.099+0000 7f62dc3db740 4 client.25627 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.099+0000 7f62dc3db740 1 -- 172.21.15.98:0/2961423462 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab7e4d7390 con 0x55ab7e57f540 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.099+0000 7f62d27fc640 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab7e57f540 0x55ab7e581930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.101+0000 7f62b6ff5640 10 client.25627 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.101+0000 7f62d27fc640 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab7e57f540 0x55ab7e581930 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=797bc2d5727b572 server_cookie=f16b7e23caa2ac2e in_seq=0 out_seq=0 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62b6ff5640 1 -- 172.21.15.98:0/2961423462 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab7e4d7390 con 0x55ab7e57f540 2024-04-25T17:43:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62b6ff5640 10 client.25627 handle_command_reply: tid=0 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 client.25627 shutdown 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 -- 172.21.15.98:0/2961423462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564620 msgr2=0x55ab7e56f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564620 0x55ab7e56f290 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f62c409bd30 tx=0x7f62c40ce2d0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 -- 172.21.15.98:0/2961423462 shutdown_connections 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab7e57f540 0x55ab7e581930 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e56f800 0x55ab7e56cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62b6ff5640 0 client.25627 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564f40 0x55ab7e56c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 --2- 172.21.15.98:0/2961423462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564620 0x55ab7e56f290 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 -- 172.21.15.98:0/2961423462 >> 172.21.15.98:0/2961423462 conn(0x55ab7e560b30 msgr2=0x55ab7e5668b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 -- 172.21.15.98:0/2961423462 shutdown_connections 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 1 -- 172.21.15.98:0/2961423462 wait complete. 2024-04-25T17:43:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.102+0000 7f62dc3db740 20 client.25627 trim_cache size 0 max 16384 2024-04-25T17:43:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62dc3db740 1 Processor -- start 2024-04-25T17:43:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62dc3db740 1 -- start start 2024-04-25T17:43:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564620 0x55ab7e57b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564f40 0x55ab7e567970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 0x55ab7e56edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62dc3db740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62d41c7b80 con 0x55ab7e564620 2024-04-25T17:43:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62dc3db740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f62d41c5390 con 0x55ab7e564f40 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62dc3db740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62d41c3200 con 0x55ab7e56f800 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564620 0x55ab7e57b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62d2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564620 0x55ab7e57b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53550/0 (socket says 172.21.15.98:53550) 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62d2ffd640 1 -- 172.21.15.98:0/4140244184 learned_addr learned my addr 172.21.15.98:0/4140244184 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62d37fe640 1 --2- 172.21.15.98:0/4140244184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 0x55ab7e56edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.124+0000 7f62d27fc640 1 --2- 172.21.15.98:0/4140244184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564f40 0x55ab7e567970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.125+0000 7f62d2ffd640 1 -- 172.21.15.98:0/4140244184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 msgr2=0x55ab7e56edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.125+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/4140244184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 0x55ab7e56edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.125+0000 7f62d2ffd640 1 -- 172.21.15.98:0/4140244184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564f40 msgr2=0x55ab7e567970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.125+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/4140244184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564f40 0x55ab7e567970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.125+0000 7f62d2ffd640 1 -- 172.21.15.98:0/4140244184 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab7e56aaf0 con 0x55ab7e564620 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.125+0000 7f62d27fc640 1 --2- 172.21.15.98:0/4140244184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564f40 0x55ab7e567970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.125+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/4140244184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564620 0x55ab7e57b230 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f62c4099ee0 tx=0x7f62c409bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4974bc54781effdf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.125+0000 7f62b8ff9640 1 -- 172.21.15.98:0/4140244184 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c40a66f0 con 0x55ab7e564620 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.125+0000 7f62b8ff9640 1 -- 172.21.15.98:0/4140244184 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c4010430 con 0x55ab7e564620 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.125+0000 7f62b8ff9640 1 -- 172.21.15.98:0/4140244184 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c40ce540 con 0x55ab7e564620 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 -- 172.21.15.98:0/4140244184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564620 msgr2=0x55ab7e57b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 --2- 172.21.15.98:0/4140244184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564620 0x55ab7e57b230 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f62c4099ee0 tx=0x7f62c409bf90 comp rx=0 tx=0).stop 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 -- 172.21.15.98:0/4140244184 shutdown_connections 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 --2- 172.21.15.98:0/4140244184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e56f800 0x55ab7e56edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 --2- 172.21.15.98:0/4140244184 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564f40 0x55ab7e567970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 --2- 172.21.15.98:0/4140244184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e564620 0x55ab7e57b230 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 -- 172.21.15.98:0/4140244184 >> 172.21.15.98:0/4140244184 conn(0x55ab7e560b30 msgr2=0x55ab7e567260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 -- 172.21.15.98:0/4140244184 shutdown_connections 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 -- 172.21.15.98:0/4140244184 wait complete. 2024-04-25T17:43:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 -- start start 2024-04-25T17:43:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.126+0000 7f62dc3db740 1 Processor -- start 2024-04-25T17:43:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.127+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564620 0x55ab7e5779a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.127+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564f40 0x55ab7e568e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.127+0000 7f62dc3db740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e56f800 0x55ab7e569350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.127+0000 7f62dc3db740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f62d41c3850 con 0x55ab7e56f800 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.127+0000 7f62dc3db740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f62d41c4cc0 con 0x55ab7e564620 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.127+0000 7f62dc3db740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62d41c3f20 con 0x55ab7e564f40 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.127+0000 7f62d27fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564f40 0x55ab7e568e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.127+0000 7f62dc3db740 10 client.0 _ll_register_callbacks cb 0x7f62c409d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.127+0000 7f62d27fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564f40 0x55ab7e568e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:55618/0 (socket says 172.21.15.98:55618) 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.127+0000 7f62d27fc640 1 -- 172.21.15.98:0/656644431 learned_addr learned my addr 172.21.15.98:0/656644431 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.128+0000 7f62d27fc640 1 -- 172.21.15.98:0/656644431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564620 msgr2=0x55ab7e5779a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.128+0000 7f62d27fc640 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564620 0x55ab7e5779a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.128+0000 7f62d27fc640 1 -- 172.21.15.98:0/656644431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e56f800 msgr2=0x55ab7e569350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.128+0000 7f62d27fc640 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e56f800 0x55ab7e569350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.128+0000 7f62d27fc640 1 -- 172.21.15.98:0/656644431 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab7e5699f0 con 0x55ab7e564f40 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.128+0000 7f62d27fc640 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564f40 0x55ab7e568e10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55ab7e56f7d0 tx=0x7f62cc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9845bc67b3c7a13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.128+0000 7f62b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.128+0000 7f62b6ff5640 1 -- 172.21.15.98:0/656644431 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc012d30 con 0x55ab7e564f40 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62b6ff5640 1 -- 172.21.15.98:0/656644431 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f62cc029330 con 0x55ab7e564f40 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62dc3db740 1 -- 172.21.15.98:0/656644431 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c4010430 con 0x55ab7e564f40 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62b6ff5640 1 -- 172.21.15.98:0/656644431 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f62cc039050 con 0x55ab7e564f40 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62dc3db740 10 client.25633 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62dc3db740 1 -- 172.21.15.98:0/656644431 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ab7e5e5780 con 0x55ab7e564f40 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62b6ff5640 1 -- 172.21.15.98:0/656644431 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c4010430 con 0x55ab7e564f40 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62b6ff5640 1 -- 172.21.15.98:0/656644431 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f62cc00f040 con 0x55ab7e564f40 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62dc3db740 10 client.25633 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62dc3db740 10 client.25633 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62dc3db740 20 client.25633 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62dc3db740 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab7e57f540 0x55ab7e5e7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62dc3db740 4 client.25633 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:43:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.129+0000 7f62dc3db740 1 -- 172.21.15.98:0/656644431 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab7e4d1200 con 0x55ab7e57f540 2024-04-25T17:43:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.130+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab7e57f540 0x55ab7e5e7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.130+0000 7f62b6ff5640 10 client.25633 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.130+0000 7f62d2ffd640 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab7e57f540 0x55ab7e5e7fa0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed9a8573c0f873be server_cookie=52f75f7b915f27e6 in_seq=0 out_seq=0 2024-04-25T17:43:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.140+0000 7f62b6ff5640 1 -- 172.21.15.98:0/656644431 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab7e4d1200 con 0x55ab7e57f540 2024-04-25T17:43:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.140+0000 7f62b6ff5640 10 client.25633 handle_command_reply: tid=0 2024-04-25T17:43:32.141 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 client.25633 shutdown 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 -- 172.21.15.98:0/656644431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564f40 msgr2=0x55ab7e568e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564f40 0x55ab7e568e10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55ab7e56f7d0 tx=0x7f62cc004ab0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 -- 172.21.15.98:0/656644431 shutdown_connections 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab7e57f540 0x55ab7e5e7fa0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab7e56f800 0x55ab7e569350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62b6ff5640 0 client.25633 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab7e564f40 0x55ab7e568e10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 --2- 172.21.15.98:0/656644431 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab7e564620 0x55ab7e5779a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 -- 172.21.15.98:0/656644431 >> 172.21.15.98:0/656644431 conn(0x55ab7e560b30 msgr2=0x55ab7e566820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 -- 172.21.15.98:0/656644431 shutdown_connections 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 1 -- 172.21.15.98:0/656644431 wait complete. 2024-04-25T17:43:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.141+0000 7f62dc3db740 20 client.25633 trim_cache size 0 max 16384 2024-04-25T17:43:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:43:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:43:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:43:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 -- 172.21.15.98:0/3099390704 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f62bc0a3e90 msgr2=0x7f62bc0a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f62bc0a3e90 0x7f62bc0a6350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009fd0 tx=0x7f62cc009680 comp rx=0 tx=0).stop 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 -- 172.21.15.98:0/3099390704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62d4110740 msgr2=0x7f62d4118d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62d4110740 0x7f62d4118d20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62c400e9f0 tx=0x7f62c400eeb0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 -- 172.21.15.98:0/3099390704 shutdown_connections 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f62bc0a3e90 0x7f62bc0a6350 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62d41122e0 0x7f62d4112790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62d411b730 0x7f62d4111da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 --2- 172.21.15.98:0/3099390704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62d4110740 0x7f62d4118d20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 -- 172.21.15.98:0/3099390704 >> 172.21.15.98:0/3099390704 conn(0x7f62d406d020 msgr2=0x7f62d410e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 -- 172.21.15.98:0/3099390704 shutdown_connections 2024-04-25T17:43:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:32.142+0000 7f62d96cb640 1 -- 172.21.15.98:0/3099390704 wait complete. 2024-04-25T17:43:32.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:32 smithi098 ceph-mon[29192]: pgmap v236: 97 pgs: 97 active+clean; 3.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.0 MiB/s wr, 481 op/s 2024-04-25T17:43:32.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:32 smithi203 ceph-mon[32834]: pgmap v236: 97 pgs: 97 active+clean; 3.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.0 MiB/s wr, 481 op/s 2024-04-25T17:43:32.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:32 smithi173 ceph-mon[32856]: pgmap v236: 97 pgs: 97 active+clean; 3.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.0 MiB/s wr, 481 op/s 2024-04-25T17:43:34.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:34 smithi098 ceph-mon[29192]: pgmap v237: 97 pgs: 97 active+clean; 3.8 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 10 MiB/s wr, 683 op/s 2024-04-25T17:43:34.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:34 smithi203 ceph-mon[32834]: pgmap v237: 97 pgs: 97 active+clean; 3.8 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 10 MiB/s wr, 683 op/s 2024-04-25T17:43:34.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:34 smithi173 ceph-mon[32856]: pgmap v237: 97 pgs: 97 active+clean; 3.8 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 10 MiB/s wr, 683 op/s 2024-04-25T17:43:36.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:36 smithi098 ceph-mon[29192]: pgmap v238: 97 pgs: 97 active+clean; 3.8 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-04-25T17:43:36.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:36 smithi203 ceph-mon[32834]: pgmap v238: 97 pgs: 97 active+clean; 3.8 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-04-25T17:43:36.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:36 smithi173 ceph-mon[32856]: pgmap v238: 97 pgs: 97 active+clean; 3.8 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-04-25T17:43:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 425.0 s, 2682.8 tps, lat 37.268 ms stddev 39.595 2024-04-25T17:43:38.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:38 smithi098 ceph-mon[29192]: pgmap v239: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 7.8 MiB/s wr, 808 op/s 2024-04-25T17:43:38.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:38 smithi203 ceph-mon[32834]: pgmap v239: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 7.8 MiB/s wr, 808 op/s 2024-04-25T17:43:38.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:38 smithi173 ceph-mon[32856]: pgmap v239: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 7.8 MiB/s wr, 808 op/s 2024-04-25T17:43:40.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:40 smithi098 ceph-mon[29192]: pgmap v240: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 5.6 MiB/s wr, 580 op/s 2024-04-25T17:43:40.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:40 smithi203 ceph-mon[32834]: pgmap v240: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 5.6 MiB/s wr, 580 op/s 2024-04-25T17:43:40.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:40 smithi173 ceph-mon[32856]: pgmap v240: 97 pgs: 97 active+clean; 3.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 5.6 MiB/s wr, 580 op/s 2024-04-25T17:43:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 430.0 s, 2919.6 tps, lat 34.063 ms stddev 35.108 2024-04-25T17:43:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:42 smithi098 ceph-mon[29192]: pgmap v241: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 6.0 MiB/s wr, 626 op/s 2024-04-25T17:43:42.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:42 smithi203 ceph-mon[32834]: pgmap v241: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 6.0 MiB/s wr, 626 op/s 2024-04-25T17:43:42.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:42 smithi173 ceph-mon[32856]: pgmap v241: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 6.0 MiB/s wr, 626 op/s 2024-04-25T17:43:44.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:44 smithi098 ceph-mon[29192]: pgmap v242: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 7.9 MiB/s wr, 812 op/s 2024-04-25T17:43:44.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:44 smithi203 ceph-mon[32834]: pgmap v242: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 7.9 MiB/s wr, 812 op/s 2024-04-25T17:43:44.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:44 smithi173 ceph-mon[32856]: pgmap v242: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 7.9 MiB/s wr, 812 op/s 2024-04-25T17:43:45.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.261+0000 7f651e7f7640 1 -- 172.21.15.98:0/1034586896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6518108f50 msgr2=0x7f6518109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.261+0000 7f651e7f7640 1 --2- 172.21.15.98:0/1034586896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6518108f50 0x7f6518109330 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6510009ca0 tx=0x7f651002a950 comp rx=0 tx=0).stop 2024-04-25T17:43:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.261+0000 7f651e7f7640 1 -- 172.21.15.98:0/1034586896 shutdown_connections 2024-04-25T17:43:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.261+0000 7f651e7f7640 1 --2- 172.21.15.98:0/1034586896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65181150a0 0x7f6518117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.261+0000 7f651e7f7640 1 --2- 172.21.15.98:0/1034586896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6518109870 0x7f6518114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.261+0000 7f651e7f7640 1 --2- 172.21.15.98:0/1034586896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6518108f50 0x7f6518109330 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.261+0000 7f651e7f7640 1 -- 172.21.15.98:0/1034586896 >> 172.21.15.98:0/1034586896 conn(0x7f651806d020 msgr2=0x7f651806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.261+0000 7f651e7f7640 1 -- 172.21.15.98:0/1034586896 shutdown_connections 2024-04-25T17:43:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.261+0000 7f651e7f7640 1 -- 172.21.15.98:0/1034586896 wait complete. 2024-04-25T17:43:45.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.262+0000 7f651e7f7640 1 Processor -- start 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.262+0000 7f651e7f7640 1 -- start start 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f651e7f7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6518108f50 0x7f65181a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f651e7f7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6518109870 0x7f65181a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f651e7f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65181150a0 0x7f65181a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f651e7f7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f651811a9d0 con 0x7f6518108f50 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f651e7f7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f651811a850 con 0x7f6518109870 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f651e7f7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f651811ab50 con 0x7f65181150a0 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f6517fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6518108f50 0x7f65181a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f6517fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6518108f50 0x7f65181a0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49060/0 (socket says 172.21.15.98:49060) 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f6517fff640 1 -- 172.21.15.98:0/3343115579 learned_addr learned my addr 172.21.15.98:0/3343115579 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:45.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f65177fe640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6518109870 0x7f65181a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f651cd94640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65181150a0 0x7f65181a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f6517fff640 1 -- 172.21.15.98:0/3343115579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65181150a0 msgr2=0x7f65181a5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f6517fff640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65181150a0 0x7f65181a5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f6517fff640 1 -- 172.21.15.98:0/3343115579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6518109870 msgr2=0x7f65181a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f6517fff640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6518109870 0x7f65181a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.263+0000 7f6517fff640 1 -- 172.21.15.98:0/3343115579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65181a5e50 con 0x7f6518108f50 2024-04-25T17:43:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.264+0000 7f6517fff640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6518108f50 0x7f65181a0eb0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6510005cc0 tx=0x7f651002f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:45.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.264+0000 7f65157fa640 1 -- 172.21.15.98:0/3343115579 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510034040 con 0x7f6518108f50 2024-04-25T17:43:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.264+0000 7f65157fa640 1 -- 172.21.15.98:0/3343115579 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65100308f0 con 0x7f6518108f50 2024-04-25T17:43:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.264+0000 7f65157fa640 1 -- 172.21.15.98:0/3343115579 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65100046a0 con 0x7f6518108f50 2024-04-25T17:43:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.264+0000 7f651e7f7640 1 -- 172.21.15.98:0/3343115579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65181a60e0 con 0x7f6518108f50 2024-04-25T17:43:45.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.264+0000 7f651e7f7640 1 -- 172.21.15.98:0/3343115579 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65181a6490 con 0x7f6518108f50 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.267+0000 7f65157fa640 1 -- 172.21.15.98:0/3343115579 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6510037020 con 0x7f6518108f50 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.267+0000 7f65215e4740 1 Processor -- start 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65215e4740 1 -- start start 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65157fa640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f65000a3f20 0x7f65000a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65215e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310620 0x55f839310a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65215e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 0x55f83931b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65215e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f83931b800 0x55f83931dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65215e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f839275970 con 0x55f83931b800 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65215e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f8392124a0 con 0x55f839310f40 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65215e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f839271930 con 0x55f839310620 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65157fa640 1 -- 172.21.15.98:0/3343115579 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f65100f13e0 con 0x7f6518108f50 2024-04-25T17:43:45.269 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f651cd94640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f65000a3f20 0x7f65000a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65177fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 0x55f83931b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65177fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 0x55f83931b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48062/0 (socket says 172.21.15.98:48062) 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65177fe640 1 -- 172.21.15.98:0/3284804674 learned_addr learned my addr 172.21.15.98:0/3284804674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f651cd94640 1 --2- 172.21.15.98:0/3284804674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f83931b800 0x55f83931dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f6517fff640 1 --2- 172.21.15.98:0/3284804674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310620 0x55f839310a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65177fe640 1 -- 172.21.15.98:0/3284804674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310620 msgr2=0x55f839310a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65177fe640 1 --2- 172.21.15.98:0/3284804674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310620 0x55f839310a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65177fe640 1 -- 172.21.15.98:0/3284804674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f83931b800 msgr2=0x55f83931dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65177fe640 1 --2- 172.21.15.98:0/3284804674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f83931b800 0x55f83931dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65177fe640 1 -- 172.21.15.98:0/3284804674 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f838f88210 con 0x55f839310f40 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f651cd94640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f65000a3f20 0x7f65000a63e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f650c005f60 tx=0x7f650c009ae0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:45.270 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.268+0000 7f65177fe640 1 --2- 172.21.15.98:0/3284804674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 0x55f83931b2c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6508009c90 tx=0x7f6508011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c17007c63fd8e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.269+0000 7f64fe7fc640 1 -- 172.21.15.98:0/3284804674 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650801e070 con 0x55f839310f40 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.269+0000 7f64fe7fc640 1 -- 172.21.15.98:0/3284804674 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6508019ae0 con 0x55f839310f40 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.269+0000 7f64fe7fc640 1 -- 172.21.15.98:0/3284804674 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650800e9d0 con 0x55f839310f40 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.270+0000 7f65215e4740 1 -- 172.21.15.98:0/3284804674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 msgr2=0x55f83931b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.270+0000 7f65215e4740 1 --2- 172.21.15.98:0/3284804674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 0x55f83931b2c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6508009c90 tx=0x7f6508011810 comp rx=0 tx=0).stop 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.270+0000 7f65215e4740 1 -- 172.21.15.98:0/3284804674 shutdown_connections 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.270+0000 7f65215e4740 1 --2- 172.21.15.98:0/3284804674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f83931b800 0x55f83931dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.270+0000 7f65215e4740 1 --2- 172.21.15.98:0/3284804674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 0x55f83931b2c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.270+0000 7f65215e4740 1 --2- 172.21.15.98:0/3284804674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310620 0x55f839310a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.270+0000 7f65215e4740 1 -- 172.21.15.98:0/3284804674 >> 172.21.15.98:0/3284804674 conn(0x55f83930cb30 msgr2=0x55f83930cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.271+0000 7f65215e4740 1 -- 172.21.15.98:0/3284804674 shutdown_connections 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.271+0000 7f65215e4740 1 -- 172.21.15.98:0/3284804674 wait complete. 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.271+0000 7f65215e4740 1 -- start start 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.271+0000 7f65215e4740 1 Processor -- start 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f65215e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310620 0x55f83931b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f65215e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 0x55f8393184c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f65215e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 0x55f839318ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f65215e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f83931e430 con 0x55f839310620 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f65215e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f8392100b0 con 0x55f839310f40 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f65215e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f83931e5b0 con 0x55f83931b800 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f6517fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310620 0x55f83931b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f65215e4740 10 client.0 _ll_register_callbacks cb 0x55f8391fdd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f6517fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310620 0x55f83931b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49084/0 (socket says 172.21.15.98:49084) 2024-04-25T17:43:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f6517fff640 1 -- 172.21.15.98:0/1624107161 learned_addr learned my addr 172.21.15.98:0/1624107161 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f65177fe640 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 0x55f8393184c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f651cd94640 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 0x55f839318ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f6517fff640 1 -- 172.21.15.98:0/1624107161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 msgr2=0x55f839318ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f6517fff640 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 0x55f839318ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f6517fff640 1 -- 172.21.15.98:0/1624107161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 msgr2=0x55f8393184c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f6517fff640 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 0x55f8393184c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f6517fff640 1 -- 172.21.15.98:0/1624107161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f839054620 con 0x55f839310620 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f6517fff640 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310620 0x55f83931b1d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f65100eeca0 tx=0x7f65100bcfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73edfd1f4b5408c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f64f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f64f77fe640 1 -- 172.21.15.98:0/1624107161 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651002c040 con 0x55f839310620 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f64f77fe640 1 -- 172.21.15.98:0/1624107161 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f651007ac40 con 0x55f839310620 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.272+0000 7f65215e4740 1 -- 172.21.15.98:0/1624107161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6508019ae0 con 0x55f839310620 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f64f77fe640 1 -- 172.21.15.98:0/1624107161 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f65100bdb90 con 0x55f839310620 2024-04-25T17:43:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f65215e4740 10 client.16140 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f65215e4740 1 -- 172.21.15.98:0/1624107161 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f83932a160 con 0x55f839310620 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f64f77fe640 1 -- 172.21.15.98:0/1624107161 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6508019ae0 con 0x55f839310620 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f64f77fe640 1 -- 172.21.15.98:0/1624107161 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f65100dc2a0 con 0x55f839310620 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f65215e4740 10 client.16140 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f65215e4740 10 client.16140 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f65215e4740 20 client.16140 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f65215e4740 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f83932b5c0 0x55f83932da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f65215e4740 4 client.16140 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.273+0000 7f65215e4740 1 -- 172.21.15.98:0/1624107161 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f839283390 con 0x55f83932b5c0 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.274+0000 7f65177fe640 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f83932b5c0 0x55f83932da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.274+0000 7f65177fe640 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f83932b5c0 0x55f83932da40 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=838af3f2a101ee12 server_cookie=993f58aeb6b39853 in_seq=0 out_seq=0 2024-04-25T17:43:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.274+0000 7f64f77fe640 10 client.16140 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f64f77fe640 1 -- 172.21.15.98:0/1624107161 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f839283390 con 0x55f83932b5c0 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f64f77fe640 10 client.16140 handle_command_reply: tid=0 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 client.16140 shutdown 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 -- 172.21.15.98:0/1624107161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310620 msgr2=0x55f83931b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310620 0x55f83931b1d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f65100eeca0 tx=0x7f65100bcfd0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 -- 172.21.15.98:0/1624107161 shutdown_connections 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f83932b5c0 0x55f83932da40 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f64f77fe640 0 client.16140 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 0x55f839318ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310f40 0x55f8393184c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 --2- 172.21.15.98:0/1624107161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310620 0x55f83931b1d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 -- 172.21.15.98:0/1624107161 >> 172.21.15.98:0/1624107161 conn(0x55f83930cb30 msgr2=0x55f839312860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 -- 172.21.15.98:0/1624107161 shutdown_connections 2024-04-25T17:43:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 1 -- 172.21.15.98:0/1624107161 wait complete. 2024-04-25T17:43:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.275+0000 7f65215e4740 20 client.16140 trim_cache size 0 max 16384 2024-04-25T17:43:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.298+0000 7f65215e4740 1 Processor -- start 2024-04-25T17:43:45.297 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.298+0000 7f65215e4740 1 -- start start 2024-04-25T17:43:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.298+0000 7f65215e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f8393272f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.298+0000 7f65215e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310f40 0x55f83931ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.298+0000 7f65215e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 0x55f839333350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.298+0000 7f65215e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65181bf1d0 con 0x55f839310f40 2024-04-25T17:43:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.298+0000 7f65215e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65181bc9e0 con 0x55f839310620 2024-04-25T17:43:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.298+0000 7f65215e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f65181ba850 con 0x55f83931b800 2024-04-25T17:43:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.298+0000 7f6517fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f8393272f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f6517fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f8393272f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48086/0 (socket says 172.21.15.98:48086) 2024-04-25T17:43:45.299 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f6517fff640 1 -- 172.21.15.98:0/285687848 learned_addr learned my addr 172.21.15.98:0/285687848 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f651cd94640 1 --2- 172.21.15.98:0/285687848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 0x55f839333350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f65177fe640 1 --2- 172.21.15.98:0/285687848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310f40 0x55f83931ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f651cd94640 1 -- 172.21.15.98:0/285687848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 msgr2=0x55f8393272f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f651cd94640 1 --2- 172.21.15.98:0/285687848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f8393272f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f651cd94640 1 -- 172.21.15.98:0/285687848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310f40 msgr2=0x55f83931ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f651cd94640 1 --2- 172.21.15.98:0/285687848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310f40 0x55f83931ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f651cd94640 1 -- 172.21.15.98:0/285687848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f83931df60 con 0x55f83931b800 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f651cd94640 1 --2- 172.21.15.98:0/285687848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 0x55f839333350 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f650c009d20 tx=0x7f650c021000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d356952c6575133 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f64fdffb640 1 -- 172.21.15.98:0/285687848 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c011dd0 con 0x55f83931b800 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f64fdffb640 1 -- 172.21.15.98:0/285687848 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f650c005020 con 0x55f83931b800 2024-04-25T17:43:45.300 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.299+0000 7f64fdffb640 1 -- 172.21.15.98:0/285687848 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c004a00 con 0x55f83931b800 2024-04-25T17:43:45.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.300+0000 7f65215e4740 1 -- 172.21.15.98:0/285687848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 msgr2=0x55f839333350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.300+0000 7f65215e4740 1 --2- 172.21.15.98:0/285687848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 0x55f839333350 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f650c009d20 tx=0x7f650c021000 comp rx=0 tx=0).stop 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.301+0000 7f65215e4740 1 -- 172.21.15.98:0/285687848 shutdown_connections 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.301+0000 7f65215e4740 1 --2- 172.21.15.98:0/285687848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f83931b800 0x55f839333350 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.301+0000 7f65215e4740 1 --2- 172.21.15.98:0/285687848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f839310f40 0x55f83931ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.301+0000 7f65215e4740 1 --2- 172.21.15.98:0/285687848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f8393272f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.301+0000 7f65215e4740 1 -- 172.21.15.98:0/285687848 >> 172.21.15.98:0/285687848 conn(0x55f83930cb30 msgr2=0x55f83930bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.301+0000 7f65215e4740 1 -- 172.21.15.98:0/285687848 shutdown_connections 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.301+0000 7f65215e4740 1 -- 172.21.15.98:0/285687848 wait complete. 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.301+0000 7f65215e4740 1 -- start start 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65215e4740 1 Processor -- start 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65215e4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f839323af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65215e4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310f40 0x55f839314e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65215e4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f83931b800 0x55f839315350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65215e4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f65181baea0 con 0x55f83931b800 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65215e4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f65181bc310 con 0x55f839310620 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65215e4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f65181bb570 con 0x55f839310f40 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65215e4740 10 client.0 _ll_register_callbacks cb 0x7f65100b9730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f6517fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f839323af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f651cd94640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f83931b800 0x55f839315350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65177fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310f40 0x55f839314e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f6517fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f839323af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48094/0 (socket says 172.21.15.98:48094) 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f6517fff640 1 -- 172.21.15.98:0/4172315924 learned_addr learned my addr 172.21.15.98:0/4172315924 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65177fe640 1 -- 172.21.15.98:0/4172315924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 msgr2=0x55f839323af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:45.307 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65177fe640 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f839323af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65177fe640 1 -- 172.21.15.98:0/4172315924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f83931b800 msgr2=0x55f839315350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65177fe640 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f83931b800 0x55f839315350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65177fe640 1 -- 172.21.15.98:0/4172315924 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f839315a90 con 0x55f839310f40 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f6517fff640 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f839323af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f65177fe640 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310f40 0x55f839314e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55f839319a90 tx=0x7f6508009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4df84e0e99ef6e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.302+0000 7f64f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f64f77fe640 1 -- 172.21.15.98:0/4172315924 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508034060 con 0x55f839310f40 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f64f77fe640 1 -- 172.21.15.98:0/4172315924 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f6508005250 con 0x55f839310f40 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f65215e4740 1 -- 172.21.15.98:0/4172315924 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f650c005020 con 0x55f839310f40 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f64f77fe640 1 -- 172.21.15.98:0/4172315924 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f650801e040 con 0x55f839310f40 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f65215e4740 10 client.25651 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f65215e4740 1 -- 172.21.15.98:0/4172315924 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f839391b00 con 0x55f839310f40 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f64f77fe640 1 -- 172.21.15.98:0/4172315924 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65080098c0 con 0x55f839310f40 2024-04-25T17:43:45.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f64f77fe640 1 -- 172.21.15.98:0/4172315924 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f650802e070 con 0x55f839310f40 2024-04-25T17:43:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f65215e4740 10 client.25651 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f65215e4740 10 client.25651 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:43:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f65215e4740 20 client.25651 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f65215e4740 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f83932b5c0 0x55f8393940d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f65215e4740 4 client.25651 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:43:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f65215e4740 1 -- 172.21.15.98:0/4172315924 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f650c005020 con 0x55f83932b5c0 2024-04-25T17:43:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.303+0000 7f6517fff640 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f83932b5c0 0x55f8393940d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.304+0000 7f64f77fe640 10 client.25651 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.304+0000 7f6517fff640 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f83932b5c0 0x55f8393940d0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ba233ea5d765c0f server_cookie=1af07568d59a5e2 in_seq=0 out_seq=0 2024-04-25T17:43:45.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.312+0000 7f64f77fe640 1 -- 172.21.15.98:0/4172315924 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f650c005020 con 0x55f83932b5c0 2024-04-25T17:43:45.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.312+0000 7f64f77fe640 10 client.25651 handle_command_reply: tid=0 2024-04-25T17:43:45.312 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.312+0000 7f65215e4740 1 client.25651 shutdown 2024-04-25T17:43:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.313+0000 7f65215e4740 1 -- 172.21.15.98:0/4172315924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310f40 msgr2=0x55f839314e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:45.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.313+0000 7f65215e4740 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310f40 0x55f839314e10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55f839319a90 tx=0x7f6508009310 comp rx=0 tx=0).stop 2024-04-25T17:43:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.314+0000 7f65215e4740 1 -- 172.21.15.98:0/4172315924 shutdown_connections 2024-04-25T17:43:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.314+0000 7f65215e4740 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f83932b5c0 0x55f8393940d0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.314+0000 7f65215e4740 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f83931b800 0x55f839315350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.314+0000 7f65215e4740 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f839310f40 0x55f839314e10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.314+0000 7f64f77fe640 0 client.25651 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.314+0000 7f65215e4740 1 --2- 172.21.15.98:0/4172315924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f839310620 0x55f839323af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.314+0000 7f65215e4740 1 -- 172.21.15.98:0/4172315924 >> 172.21.15.98:0/4172315924 conn(0x55f83930cb30 msgr2=0x55f839311fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:45.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.314+0000 7f65215e4740 1 -- 172.21.15.98:0/4172315924 shutdown_connections 2024-04-25T17:43:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.314+0000 7f65215e4740 1 -- 172.21.15.98:0/4172315924 wait complete. 2024-04-25T17:43:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.314+0000 7f65215e4740 20 client.25651 trim_cache size 0 max 16384 2024-04-25T17:43:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:43:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:43:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:43:45.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 -- 172.21.15.98:0/3343115579 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f65000a3f20 msgr2=0x7f65000a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f65000a3f20 0x7f65000a63e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f650c005f60 tx=0x7f650c009ae0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 -- 172.21.15.98:0/3343115579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6518108f50 msgr2=0x7f65181a0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6518108f50 0x7f65181a0eb0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6510005cc0 tx=0x7f651002f3a0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 -- 172.21.15.98:0/3343115579 shutdown_connections 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f65000a3f20 0x7f65000a63e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65181150a0 0x7f65181a5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6518109870 0x7f65181a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 --2- 172.21.15.98:0/3343115579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6518108f50 0x7f65181a0eb0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 -- 172.21.15.98:0/3343115579 >> 172.21.15.98:0/3343115579 conn(0x7f651806d020 msgr2=0x7f6518070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:45.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 -- 172.21.15.98:0/3343115579 shutdown_connections 2024-04-25T17:43:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:45.315+0000 7f651e7f7640 1 -- 172.21.15.98:0/3343115579 wait complete. 2024-04-25T17:43:46.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:46 smithi098 ceph-mon[29192]: pgmap v243: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.6 MiB/s wr, 579 op/s 2024-04-25T17:43:46.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:46 smithi203 ceph-mon[32834]: pgmap v243: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.6 MiB/s wr, 579 op/s 2024-04-25T17:43:46.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:46 smithi173 ceph-mon[32856]: pgmap v243: 97 pgs: 97 active+clean; 3.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.6 MiB/s wr, 579 op/s 2024-04-25T17:43:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 435.0 s, 2730.4 tps, lat 36.834 ms stddev 39.603 2024-04-25T17:43:46.750 INFO:tasks.quiescer.fs.[cephfs]:Total duration (50.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-25T17:43:46.750 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.3 2024-04-25T17:43:46.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:43:47.082 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:43:48.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:48 smithi098 ceph-mon[29192]: pgmap v244: 97 pgs: 97 active+clean; 3.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.2 MiB/s wr, 856 op/s 2024-04-25T17:43:48.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:48 smithi203 ceph-mon[32834]: pgmap v244: 97 pgs: 97 active+clean; 3.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.2 MiB/s wr, 856 op/s 2024-04-25T17:43:48.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:48 smithi173 ceph-mon[32856]: pgmap v244: 97 pgs: 97 active+clean; 3.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.2 MiB/s wr, 856 op/s 2024-04-25T17:43:48.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.951+0000 7fccfe724700 1 -- 172.21.15.98:0/3129215931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccf8107940 msgr2=0x7fccf8107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.951+0000 7fccfe724700 1 --2- 172.21.15.98:0/3129215931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccf8107940 0x7fccf8107d20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fcce80099c0 tx=0x7fcce8009cd0 comp rx=0 tx=0).stop 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.951+0000 7fccfe724700 1 -- 172.21.15.98:0/3129215931 shutdown_connections 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.951+0000 7fccfe724700 1 --2- 172.21.15.98:0/3129215931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fccf81034b0 0x7fccf810f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.951+0000 7fccfe724700 1 --2- 172.21.15.98:0/3129215931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccf8107940 0x7fccf8107d20 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.951+0000 7fccfe724700 1 --2- 172.21.15.98:0/3129215931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccf8102af0 0x7fccf8102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.951+0000 7fccfe724700 1 -- 172.21.15.98:0/3129215931 >> 172.21.15.98:0/3129215931 conn(0x7fccf8079ba0 msgr2=0x7fccf8079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.951+0000 7fccfe724700 1 -- 172.21.15.98:0/3129215931 shutdown_connections 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccfe724700 1 -- 172.21.15.98:0/3129215931 wait complete. 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccfe724700 1 Processor -- start 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccfe724700 1 -- start start 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccfe724700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccf8102af0 0x7fccf819b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccfe724700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fccf81034b0 0x7fccf819b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccfe724700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccf8107940 0x7fccf819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:48.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccfe724700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fccf8075900 con 0x7fccf8102af0 2024-04-25T17:43:48.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccfe724700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fccf8075780 con 0x7fccf8107940 2024-04-25T17:43:48.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccfe724700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fccf8075a80 con 0x7fccf81034b0 2024-04-25T17:43:48.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccf7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccf8102af0 0x7fccf819b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:48.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccf7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccf8102af0 0x7fccf819b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49120/0 (socket says 172.21.15.98:49120) 2024-04-25T17:43:48.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccf7fff700 1 -- 172.21.15.98:0/440304248 learned_addr learned my addr 172.21.15.98:0/440304248 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:48.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.953+0000 7fcceffff700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fccf81034b0 0x7fccf819b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:48.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.952+0000 7fccfccc1700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccf8107940 0x7fccf819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.953+0000 7fccfccc1700 1 -- 172.21.15.98:0/440304248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fccf81034b0 msgr2=0x7fccf819b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.954+0000 7fccfccc1700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fccf81034b0 0x7fccf819b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.954+0000 7fccfccc1700 1 -- 172.21.15.98:0/440304248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccf8102af0 msgr2=0x7fccf819b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.954+0000 7fccfccc1700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccf8102af0 0x7fccf819b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.954+0000 7fccfccc1700 1 -- 172.21.15.98:0/440304248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf81a0400 con 0x7fccf8107940 2024-04-25T17:43:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.954+0000 7fccfccc1700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccf8107940 0x7fccf819fcc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fccf000efc0 tx=0x7fccf000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:48.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.954+0000 7fccf5ffb700 1 -- 172.21.15.98:0/440304248 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf00092e0 con 0x7fccf8107940 2024-04-25T17:43:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.954+0000 7fccf5ffb700 1 -- 172.21.15.98:0/440304248 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccf0004bb0 con 0x7fccf8107940 2024-04-25T17:43:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.954+0000 7fccf5ffb700 1 -- 172.21.15.98:0/440304248 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0004280 con 0x7fccf8107940 2024-04-25T17:43:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.954+0000 7fccfe724700 1 -- 172.21.15.98:0/440304248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf81a0690 con 0x7fccf8107940 2024-04-25T17:43:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.954+0000 7fccfe724700 1 -- 172.21.15.98:0/440304248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf81a7f70 con 0x7fccf8107940 2024-04-25T17:43:48.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.956+0000 7fccfe724700 1 -- 172.21.15.98:0/440304248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf8069730 con 0x7fccf8107940 2024-04-25T17:43:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.957+0000 7fccf5ffb700 1 -- 172.21.15.98:0/440304248 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fccf001b030 con 0x7fccf8107940 2024-04-25T17:43:48.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.958+0000 7fccf5ffb700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcce40a3e30 0x7fcce40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.958+0000 7fccf7fff700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcce40a3e30 0x7fcce40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.958+0000 7fccf5ffb700 1 -- 172.21.15.98:0/440304248 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fccf00d0710 con 0x7fccf8107940 2024-04-25T17:43:48.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.959+0000 7fccf7fff700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcce40a3e30 0x7fcce40a62f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcce8005cd0 tx=0x7fcce8005c00 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:48.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:48.963+0000 7fccf5ffb700 1 -- 172.21.15.98:0/440304248 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fccf0048080 con 0x7fccf8107940 2024-04-25T17:43:49.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.340+0000 7fccfe724700 1 -- 172.21.15.98:0/440304248 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccf81a81a0 con 0x7fccf8107940 2024-04-25T17:43:49.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.341+0000 7fccf5ffb700 1 -- 172.21.15.98:0/440304248 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7fccf0098030 con 0x7fccf8107940 2024-04-25T17:43:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:43:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:42:52.734463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:43:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 -- 172.21.15.98:0/440304248 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcce40a3e30 msgr2=0x7fcce40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcce40a3e30 0x7fcce40a62f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcce8005cd0 tx=0x7fcce8005c00 comp rx=0 tx=0).stop 2024-04-25T17:43:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 -- 172.21.15.98:0/440304248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccf8107940 msgr2=0x7fccf819fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccf8107940 0x7fccf819fcc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fccf000efc0 tx=0x7fccf000c3a0 comp rx=0 tx=0).stop 2024-04-25T17:43:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 -- 172.21.15.98:0/440304248 shutdown_connections 2024-04-25T17:43:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcce40a3e30 0x7fcce40a62f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:49.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fccf8107940 0x7fccf819fcc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccf8102af0 0x7fccf819b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 --2- 172.21.15.98:0/440304248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fccf81034b0 0x7fccf819b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 -- 172.21.15.98:0/440304248 >> 172.21.15.98:0/440304248 conn(0x7fccf8079ba0 msgr2=0x7fccf80fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 -- 172.21.15.98:0/440304248 shutdown_connections 2024-04-25T17:43:49.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:49.345+0000 7fccfe724700 1 -- 172.21.15.98:0/440304248 wait complete. 2024-04-25T17:43:49.348 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 38 2024-04-25T17:43:50.040 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 85.3 --await-for 42.0' 2024-04-25T17:43:50.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 85.3 --await-for 42.0 2024-04-25T17:43:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:43:50.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:50 smithi098 ceph-mon[29192]: pgmap v245: 97 pgs: 97 active+clean; 3.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.5 MiB/s wr, 574 op/s 2024-04-25T17:43:50.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:50 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/440304248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:43:50.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:50 smithi203 ceph-mon[32834]: pgmap v245: 97 pgs: 97 active+clean; 3.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.5 MiB/s wr, 574 op/s 2024-04-25T17:43:50.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:50 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/440304248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:43:50.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:50 smithi173 ceph-mon[32856]: pgmap v245: 97 pgs: 97 active+clean; 3.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.5 MiB/s wr, 574 op/s 2024-04-25T17:43:50.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:50 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/440304248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:43:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 440.0 s, 3218.0 tps, lat 31.027 ms stddev 32.920 2024-04-25T17:43:52.267 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:52 smithi098 ceph-mon[29192]: pgmap v246: 97 pgs: 97 active+clean; 3.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 707 op/s 2024-04-25T17:43:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f0657fff700 1 -- 172.21.15.98:0/3205751042 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064c014a20 con 0x7f0658100ed0 2024-04-25T17:43:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 -- 172.21.15.98:0/3205751042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0658100ed0 msgr2=0x7f06581012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 --2- 172.21.15.98:0/3205751042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0658100ed0 0x7f06581012b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f064c00ca60 tx=0x7f064c00cd70 comp rx=0 tx=0).stop 2024-04-25T17:43:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 -- 172.21.15.98:0/3205751042 shutdown_connections 2024-04-25T17:43:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 --2- 172.21.15.98:0/3205751042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0658105fb0 0x7f065810db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 --2- 172.21.15.98:0/3205751042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0658100ed0 0x7f06581012b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 --2- 172.21.15.98:0/3205751042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0658101880 0x7f06581058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 -- 172.21.15.98:0/3205751042 >> 172.21.15.98:0/3205751042 conn(0x7f0658077b70 msgr2=0x7f06580feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 -- 172.21.15.98:0/3205751042 shutdown_connections 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 -- 172.21.15.98:0/3205751042 wait complete. 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 Processor -- start 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.437+0000 7f065f9b2700 1 -- start start 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065f9b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0658100ed0 0x7f0658197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065f9b2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0658101880 0x7f0658198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065f9b2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0658105fb0 0x7f0658198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065f9b2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0658110a90 con 0x7f0658100ed0 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065f9b2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0658110910 con 0x7f0658105fb0 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065f9b2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0658110c10 con 0x7f0658101880 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065d74e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0658100ed0 0x7f0658197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065d74e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0658100ed0 0x7f0658197df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49132/0 (socket says 172.21.15.98:49132) 2024-04-25T17:43:52.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065d74e700 1 -- 172.21.15.98:0/3031001971 learned_addr learned my addr 172.21.15.98:0/3031001971 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065df4f700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0658105fb0 0x7f0658198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065cf4d700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0658101880 0x7f0658198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065df4f700 1 -- 172.21.15.98:0/3031001971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0658101880 msgr2=0x7f0658198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065df4f700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0658101880 0x7f0658198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065df4f700 1 -- 172.21.15.98:0/3031001971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0658100ed0 msgr2=0x7f0658197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065df4f700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0658100ed0 0x7f0658197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065df4f700 1 -- 172.21.15.98:0/3031001971 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06581a7600 con 0x7f0658105fb0 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065cf4d700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0658101880 0x7f0658198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065d74e700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0658100ed0 0x7f0658197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.438+0000 7f065df4f700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0658105fb0 0x7f0658198870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f064800e930 tx=0x7f064800ec40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.439+0000 7f06567fc700 1 -- 172.21.15.98:0/3031001971 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648004220 con 0x7f0658105fb0 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.439+0000 7f065f9b2700 1 -- 172.21.15.98:0/3031001971 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06581a78f0 con 0x7f0658105fb0 2024-04-25T17:43:52.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.439+0000 7f065f9b2700 1 -- 172.21.15.98:0/3031001971 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06581a7e30 con 0x7f0658105fb0 2024-04-25T17:43:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.439+0000 7f06567fc700 1 -- 172.21.15.98:0/3031001971 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06480043c0 con 0x7f0658105fb0 2024-04-25T17:43:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.439+0000 7f06567fc700 1 -- 172.21.15.98:0/3031001971 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648003d20 con 0x7f0658105fb0 2024-04-25T17:43:52.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.441+0000 7f06567fc700 1 -- 172.21.15.98:0/3031001971 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f064800c7e0 con 0x7f0658105fb0 2024-04-25T17:43:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.442+0000 7f06567fc700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06400a3ed0 0x7f06400a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.442+0000 7f06567fc700 1 -- 172.21.15.98:0/3031001971 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f06480d0770 con 0x7f0658105fb0 2024-04-25T17:43:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.442+0000 7f065d74e700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06400a3ed0 0x7f06400a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.444+0000 7f066a23bb80 1 Processor -- start 2024-04-25T17:43:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.444+0000 7f066a23bb80 1 -- start start 2024-04-25T17:43:52.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.444+0000 7f065d74e700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06400a3ed0 0x7f06400a6390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f064c000f80 tx=0x7f064c0060c0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.444+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b824f4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.444+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 0x561b824ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.444+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b82500270 0x561b82502660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.444+0000 7f066a23bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561b82447ec0 con 0x561b824f53a0 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.444+0000 7f066a23bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561b823eeaf0 con 0x561b824f4a80 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.444+0000 7f066a23bb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561b82443e80 con 0x561b82500270 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065cf4d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 0x561b824ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065d74e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b824f4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065df4f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b82500270 0x561b82502660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065df4f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b82500270 0x561b82502660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:39036/0 (socket says 172.21.15.98:39036) 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065df4f700 1 -- 172.21.15.98:0/1738218484 learned_addr learned my addr 172.21.15.98:0/1738218484 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065df4f700 1 -- 172.21.15.98:0/1738218484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 msgr2=0x561b824f4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065df4f700 1 --2- 172.21.15.98:0/1738218484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b824f4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065df4f700 1 -- 172.21.15.98:0/1738218484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 msgr2=0x561b824ffd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:52.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065df4f700 1 --2- 172.21.15.98:0/1738218484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 0x561b824ffd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065df4f700 1 -- 172.21.15.98:0/1738218484 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b82395a60 con 0x561b82500270 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f065df4f700 1 --2- 172.21.15.98:0/1738218484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b82500270 0x561b82502660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f06480098a0 tx=0x7f064809c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44685e54ae71ceb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f063f7fe700 1 -- 172.21.15.98:0/1738218484 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064809cdb0 con 0x561b82500270 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f063f7fe700 1 -- 172.21.15.98:0/1738218484 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f064809c6b0 con 0x561b82500270 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.445+0000 7f063f7fe700 1 -- 172.21.15.98:0/1738218484 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064809cb70 con 0x561b82500270 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.446+0000 7f066a23bb80 1 -- 172.21.15.98:0/1738218484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b82500270 msgr2=0x561b82502660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.446+0000 7f066a23bb80 1 --2- 172.21.15.98:0/1738218484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b82500270 0x561b82502660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f06480098a0 tx=0x7f064809c500 comp rx=0 tx=0).stop 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.446+0000 7f066a23bb80 1 -- 172.21.15.98:0/1738218484 shutdown_connections 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.446+0000 7f066a23bb80 1 --2- 172.21.15.98:0/1738218484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b82500270 0x561b82502660 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.446+0000 7f066a23bb80 1 --2- 172.21.15.98:0/1738218484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b824f4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.446+0000 7f066a23bb80 1 --2- 172.21.15.98:0/1738218484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 0x561b824ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.446+0000 7f066a23bb80 1 -- 172.21.15.98:0/1738218484 >> 172.21.15.98:0/1738218484 conn(0x561b82422820 msgr2=0x561b824f0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.446+0000 7f066a23bb80 1 -- 172.21.15.98:0/1738218484 shutdown_connections 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.446+0000 7f066a23bb80 1 -- 172.21.15.98:0/1738218484 wait complete. 2024-04-25T17:43:52.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.447+0000 7f066a23bb80 1 -- start start 2024-04-25T17:43:52.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.447+0000 7f066a23bb80 1 Processor -- start 2024-04-25T17:43:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.447+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b824ff940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.447+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824fcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.447+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b82500270 0x561b824fd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.447+0000 7f066a23bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561b82502cc0 con 0x561b82500270 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.447+0000 7f066a23bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561b823e6d60 con 0x561b824f4a80 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f066a23bb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561b82502e40 con 0x561b824f53a0 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f066a23bb80 10 client.0 _ll_register_callbacks cb 0x561b823e0ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f065cf4d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824fcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f065cf4d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824fcc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:39038/0 (socket says 172.21.15.98:39038) 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f065cf4d700 1 -- 172.21.15.98:0/770828422 learned_addr learned my addr 172.21.15.98:0/770828422 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f065d74e700 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b824ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f065df4f700 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b82500270 0x561b824fd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f065df4f700 1 -- 172.21.15.98:0/770828422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 msgr2=0x561b824fcc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f065df4f700 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824fcc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f065df4f700 1 -- 172.21.15.98:0/770828422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 msgr2=0x561b824ff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f065df4f700 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b824ff940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.448+0000 7f065df4f700 1 -- 172.21.15.98:0/770828422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b823ff400 con 0x561b82500270 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f065df4f700 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b82500270 0x561b824fd180 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f064809a960 tx=0x7f064809cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0db07c9147587 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f063cff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f063cff9700 1 -- 172.21.15.98:0/770828422 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648080060 con 0x561b82500270 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f066a23bb80 1 -- 172.21.15.98:0/770828422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f064809c6b0 con 0x561b82500270 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f063cff9700 1 -- 172.21.15.98:0/770828422 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f06480a50a0 con 0x561b82500270 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f063cff9700 1 -- 172.21.15.98:0/770828422 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f06480b9070 con 0x561b82500270 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f066a23bb80 10 client.16164 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f066a23bb80 1 -- 172.21.15.98:0/770828422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561b82502cc0 con 0x561b82500270 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f063cff9700 1 -- 172.21.15.98:0/770828422 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f064809c6b0 con 0x561b82500270 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f063cff9700 1 -- 172.21.15.98:0/770828422 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f06480bb310 con 0x561b82500270 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f066a23bb80 10 client.16164 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:52.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f066a23bb80 10 client.16164 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:43:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f066a23bb80 20 client.16164 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f066a23bb80 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561b82506960 0x561b82508d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f066a23bb80 4 client.16164 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:43:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.449+0000 7f066a23bb80 1 -- 172.21.15.98:0/770828422 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b824556b0 con 0x561b82506960 2024-04-25T17:43:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.450+0000 7f065d74e700 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561b82506960 0x561b82508d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.450+0000 7f065d74e700 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561b82506960 0x561b82508d80 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f2f46da54ecfe5c server_cookie=66c25cbfa9a67f81 in_seq=0 out_seq=0 2024-04-25T17:43:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.450+0000 7f063cff9700 10 client.16164 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.451+0000 7f063cff9700 1 -- 172.21.15.98:0/770828422 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b824556b0 con 0x561b82506960 2024-04-25T17:43:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.451+0000 7f063cff9700 10 client.16164 handle_command_reply: tid=0 2024-04-25T17:43:52.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.451+0000 7f066a23bb80 1 client.16164 shutdown 2024-04-25T17:43:52.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.451+0000 7f066a23bb80 1 -- 172.21.15.98:0/770828422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b82500270 msgr2=0x561b824fd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:52.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.451+0000 7f066a23bb80 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b82500270 0x561b824fd180 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f064809a960 tx=0x7f064809cb40 comp rx=0 tx=0).stop 2024-04-25T17:43:52.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.453+0000 7f066a23bb80 1 -- 172.21.15.98:0/770828422 shutdown_connections 2024-04-25T17:43:52.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.453+0000 7f066a23bb80 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561b82506960 0x561b82508d80 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.453+0000 7f066a23bb80 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b82500270 0x561b824fd180 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.453+0000 7f066a23bb80 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b824ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.453+0000 7f066a23bb80 1 --2- 172.21.15.98:0/770828422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824fcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.453+0000 7f066a23bb80 1 -- 172.21.15.98:0/770828422 >> 172.21.15.98:0/770828422 conn(0x561b82422820 msgr2=0x561b824f0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.453+0000 7f063cff9700 0 client.16164 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.453+0000 7f066a23bb80 1 -- 172.21.15.98:0/770828422 shutdown_connections 2024-04-25T17:43:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.453+0000 7f066a23bb80 1 -- 172.21.15.98:0/770828422 wait complete. 2024-04-25T17:43:52.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.453+0000 7f066a23bb80 20 client.16164 trim_cache size 0 max 16384 2024-04-25T17:43:52.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:52 smithi203 ceph-mon[32834]: pgmap v246: 97 pgs: 97 active+clean; 3.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 707 op/s 2024-04-25T17:43:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.478+0000 7f066a23bb80 1 Processor -- start 2024-04-25T17:43:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.478+0000 7f066a23bb80 1 -- start start 2024-04-25T17:43:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f4a80 0x561b82508ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 0x561b824fbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b82500270 0x561b8252a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f066a23bb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f064400ff10 con 0x561b824f53a0 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f066a23bb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f064400d6e0 con 0x561b82500270 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f066a23bb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f064400b550 con 0x561b824f4a80 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f065cf4d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 0x561b824fbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f065cf4d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 0x561b824fbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49152/0 (socket says 172.21.15.98:49152) 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f065cf4d700 1 -- 172.21.15.98:0/825433509 learned_addr learned my addr 172.21.15.98:0/825433509 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f065df4f700 1 --2- 172.21.15.98:0/825433509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b82500270 0x561b8252a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f065d74e700 1 --2- 172.21.15.98:0/825433509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f4a80 0x561b82508ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f065df4f700 1 -- 172.21.15.98:0/825433509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f4a80 msgr2=0x561b82508ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f065df4f700 1 --2- 172.21.15.98:0/825433509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f4a80 0x561b82508ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f065df4f700 1 -- 172.21.15.98:0/825433509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 msgr2=0x561b824fbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f065df4f700 1 --2- 172.21.15.98:0/825433509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 0x561b824fbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.479+0000 7f065df4f700 1 -- 172.21.15.98:0/825433509 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b824fb1d0 con 0x561b82500270 2024-04-25T17:43:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.480+0000 7f065df4f700 1 --2- 172.21.15.98:0/825433509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b82500270 0x561b8252a9c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f064809c3b0 tx=0x7f0648098a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f31ceb2612d7586 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.480+0000 7f063effd700 1 -- 172.21.15.98:0/825433509 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648080070 con 0x561b82500270 2024-04-25T17:43:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.480+0000 7f063effd700 1 -- 172.21.15.98:0/825433509 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f064809cdb0 con 0x561b82500270 2024-04-25T17:43:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.480+0000 7f063effd700 1 -- 172.21.15.98:0/825433509 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06480a5320 con 0x561b82500270 2024-04-25T17:43:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.481+0000 7f066a23bb80 1 -- 172.21.15.98:0/825433509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b82500270 msgr2=0x561b8252a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.481+0000 7f066a23bb80 1 --2- 172.21.15.98:0/825433509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b82500270 0x561b8252a9c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f064809c3b0 tx=0x7f0648098a80 comp rx=0 tx=0).stop 2024-04-25T17:43:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.481+0000 7f066a23bb80 1 -- 172.21.15.98:0/825433509 shutdown_connections 2024-04-25T17:43:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.481+0000 7f066a23bb80 1 --2- 172.21.15.98:0/825433509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f4a80 0x561b82508ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.481+0000 7f066a23bb80 1 --2- 172.21.15.98:0/825433509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b82500270 0x561b8252a9c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.481+0000 7f066a23bb80 1 --2- 172.21.15.98:0/825433509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b824f53a0 0x561b824fbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.481+0000 7f066a23bb80 1 -- 172.21.15.98:0/825433509 >> 172.21.15.98:0/825433509 conn(0x561b82422820 msgr2=0x561b824fb550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.481+0000 7f066a23bb80 1 -- 172.21.15.98:0/825433509 shutdown_connections 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.481+0000 7f066a23bb80 1 -- 172.21.15.98:0/825433509 wait complete. 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.482+0000 7f066a23bb80 1 -- start start 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.482+0000 7f066a23bb80 1 Processor -- start 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.482+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b8250e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.482+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824f92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.482+0000 7f066a23bb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b82500270 0x561b824f9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f065cf4d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824f92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f065cf4d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824f92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:39056/0 (socket says 172.21.15.98:39056) 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f065d74e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b8250e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f065d74e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b8250e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:48148/0 (socket says 172.21.15.98:48148) 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f065cf4d700 1 -- 172.21.15.98:0/4262518665 learned_addr learned my addr 172.21.15.98:0/4262518665 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:52.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f064400bba0 con 0x561b82500270 2024-04-25T17:43:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f064400d010 con 0x561b824f4a80 2024-04-25T17:43:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f064400c270 con 0x561b824f53a0 2024-04-25T17:43:52.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f066a23bb80 10 client.0 _ll_register_callbacks cb 0x561b823e0ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f065cf4d700 1 -- 172.21.15.98:0/4262518665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 msgr2=0x561b8250e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.483+0000 7f065cf4d700 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b8250e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.484+0000 7f065cf4d700 1 -- 172.21.15.98:0/4262518665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b82500270 msgr2=0x561b824f9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.484+0000 7f065cf4d700 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b82500270 0x561b824f9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.484+0000 7f065cf4d700 1 -- 172.21.15.98:0/4262518665 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b824f9d70 con 0x561b824f53a0 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.484+0000 7f065cf4d700 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824f92f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0650012ab0 tx=0x7f0650012dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0e776357467fc8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.484+0000 7f063cff9700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.484+0000 7f063cff9700 1 -- 172.21.15.98:0/4262518665 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0650004520 con 0x561b824f53a0 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.484+0000 7f063cff9700 1 -- 172.21.15.98:0/4262518665 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f0650010820 con 0x561b824f53a0 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.485+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f064809cdb0 con 0x561b824f53a0 2024-04-25T17:43:52.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.485+0000 7f063cff9700 1 -- 172.21.15.98:0/4262518665 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f065002b050 con 0x561b824f53a0 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.485+0000 7f066a23bb80 10 client.25675 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.485+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561b82586ef0 con 0x561b824f53a0 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.485+0000 7f063cff9700 1 -- 172.21.15.98:0/4262518665 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f064809cdb0 con 0x561b824f53a0 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.486+0000 7f063cff9700 1 -- 172.21.15.98:0/4262518665 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f065001d480 con 0x561b824f53a0 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.486+0000 7f066a23bb80 10 client.25675 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.486+0000 7f066a23bb80 10 client.25675 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.486+0000 7f066a23bb80 20 client.25675 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.486+0000 7f066a23bb80 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561b82587730 0x561b82589b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.486+0000 7f066a23bb80 4 client.25675 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.3, "await_for": 42.0}] 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.486+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.3, "await_for": 42.0}) -- 0x561b8244f750 con 0x561b82587730 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.486+0000 7f065d74e700 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561b82587730 0x561b82589b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:52.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.487+0000 7f065d74e700 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561b82587730 0x561b82589b50 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=299d3d3f7dcfb9a server_cookie=dee2f37e1037932f in_seq=0 out_seq=0 2024-04-25T17:43:52.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.487+0000 7f063cff9700 10 client.25675 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:52.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:52 smithi173 ceph-mon[32856]: pgmap v246: 97 pgs: 97 active+clean; 3.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 MiB/s wr, 707 op/s 2024-04-25T17:43:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f063cff9700 1 -- 172.21.15.98:0/4262518665 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x561b8244f750 con 0x561b82587730 2024-04-25T17:43:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f063cff9700 10 client.25675 handle_command_reply: tid=0 2024-04-25T17:43:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 client.25675 shutdown 2024-04-25T17:43:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 msgr2=0x561b824f92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824f92f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0650012ab0 tx=0x7f0650012dc0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 shutdown_connections 2024-04-25T17:43:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561b82587730 0x561b82589b50 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561b82500270 0x561b824f9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561b824f4a80 0x561b8250e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 --2- 172.21.15.98:0/4262518665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561b824f53a0 0x561b824f92f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f063cff9700 0 client.25675 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 >> 172.21.15.98:0/4262518665 conn(0x561b82422820 msgr2=0x561b824fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 shutdown_connections 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 1 -- 172.21.15.98:0/4262518665 wait complete. 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f066a23bb80 20 client.25675 trim_cache size 0 max 16384 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 38, 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 22, 2024-04-25T17:43:52.803 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "b20338f": { 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 22, 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 85.3, 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:43:52.804 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:43:52.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:52.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:52.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:52.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:52.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:43:52.805 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 -- 172.21.15.98:0/3031001971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06400a3ed0 msgr2=0x7f06400a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06400a3ed0 0x7f06400a6390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f064c000f80 tx=0x7f064c0060c0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 -- 172.21.15.98:0/3031001971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0658105fb0 msgr2=0x7f0658198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0658105fb0 0x7f0658198870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f064800e930 tx=0x7f064800ec40 comp rx=0 tx=0).stop 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 -- 172.21.15.98:0/3031001971 shutdown_connections 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06400a3ed0 0x7f06400a6390 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0658105fb0 0x7f0658198870 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0658100ed0 0x7f0658197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 --2- 172.21.15.98:0/3031001971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0658101880 0x7f0658198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 -- 172.21.15.98:0/3031001971 >> 172.21.15.98:0/3031001971 conn(0x7f0658077b70 msgr2=0x7f06580ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 -- 172.21.15.98:0/3031001971 shutdown_connections 2024-04-25T17:43:52.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:43:52.802+0000 7f065f9b2700 1 -- 172.21.15.98:0/3031001971 wait complete. 2024-04-25T17:43:53.523 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b20338f', quiesce took 6.8 seconds. Will release after: 25.3 2024-04-25T17:43:54.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:54 smithi098 ceph-mon[29192]: pgmap v247: 97 pgs: 97 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.3 MiB/s wr, 843 op/s 2024-04-25T17:43:54.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:54 smithi203 ceph-mon[32834]: pgmap v247: 97 pgs: 97 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.3 MiB/s wr, 843 op/s 2024-04-25T17:43:54.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:54 smithi173 ceph-mon[32856]: pgmap v247: 97 pgs: 97 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.3 MiB/s wr, 843 op/s 2024-04-25T17:43:56.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:56 smithi098 ceph-mon[29192]: pgmap v248: 97 pgs: 97 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.7 MiB/s wr, 591 op/s 2024-04-25T17:43:56.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:56 smithi203 ceph-mon[32834]: pgmap v248: 97 pgs: 97 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.7 MiB/s wr, 591 op/s 2024-04-25T17:43:56.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:56 smithi173 ceph-mon[32856]: pgmap v248: 97 pgs: 97 active+clean; 3.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.7 MiB/s wr, 591 op/s 2024-04-25T17:43:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 445.0 s, 584.5 tps, lat 33.131 ms stddev 35.230 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.412+0000 7fa7faacf640 1 -- 172.21.15.98:0/2687958088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f41061a0 msgr2=0x7fa7f410dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.412+0000 7fa7faacf640 1 --2- 172.21.15.98:0/2687958088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f41061a0 0x7fa7f410dd70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec009ca0 tx=0x7fa7ec02a8c0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 -- 172.21.15.98:0/2687958088 shutdown_connections 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 --2- 172.21.15.98:0/2687958088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f41061a0 0x7fa7f410dd70 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 --2- 172.21.15.98:0/2687958088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f4101ad0 0x7fa7f4105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 --2- 172.21.15.98:0/2687958088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f4101120 0x7fa7f4101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 -- 172.21.15.98:0/2687958088 >> 172.21.15.98:0/2687958088 conn(0x7fa7f4078060 msgr2=0x7fa7f4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 -- 172.21.15.98:0/2687958088 shutdown_connections 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 -- 172.21.15.98:0/2687958088 wait complete. 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 Processor -- start 2024-04-25T17:43:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 -- start start 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f4101120 0x7fa7f4197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41061a0 0x7fa7f4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f419bdc0 0x7fa7f419c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f4111110 con 0x7fa7f419bdc0 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f4110f90 con 0x7fa7f4101120 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.413+0000 7fa7faacf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f4111290 con 0x7fa7f41061a0 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f906c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f419bdc0 0x7fa7f419c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f3fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41061a0 0x7fa7f4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f906c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f419bdc0 0x7fa7f419c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38722/0 (socket says 172.21.15.98:38722) 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f906c640 1 -- 172.21.15.98:0/1805493066 learned_addr learned my addr 172.21.15.98:0/1805493066 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f886b640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f4101120 0x7fa7f4197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f906c640 1 -- 172.21.15.98:0/1805493066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41061a0 msgr2=0x7fa7f4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f906c640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41061a0 0x7fa7f4197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f906c640 1 -- 172.21.15.98:0/1805493066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f4101120 msgr2=0x7fa7f4197550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f906c640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f4101120 0x7fa7f4197550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f906c640 1 -- 172.21.15.98:0/1805493066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f419c8b0 con 0x7fa7f419bdc0 2024-04-25T17:43:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f906c640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f419bdc0 0x7fa7f419c270 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec005cc0 tx=0x7fa7ec031680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f1ffb640 1 -- 172.21.15.98:0/1805493066 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec04a070 con 0x7fa7f419bdc0 2024-04-25T17:43:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f1ffb640 1 -- 172.21.15.98:0/1805493066 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7ec003e20 con 0x7fa7f419bdc0 2024-04-25T17:43:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7faacf640 1 -- 172.21.15.98:0/1805493066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7f419cb40 con 0x7fa7f419bdc0 2024-04-25T17:43:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.414+0000 7fa7f1ffb640 1 -- 172.21.15.98:0/1805493066 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec02f6c0 con 0x7fa7f419bdc0 2024-04-25T17:43:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.415+0000 7fa7faacf640 1 -- 172.21.15.98:0/1805493066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7f406aab0 con 0x7fa7f419bdc0 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.417+0000 7fa7f1ffb640 1 -- 172.21.15.98:0/1805493066 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa7ec04d070 con 0x7fa7f419bdc0 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.417+0000 7fa7fd7e6740 1 Processor -- start 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.417+0000 7fa7fd7e6740 1 -- start start 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.417+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269620 0x560e1f269a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.417+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 0x560e1f2742c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.417+0000 7fa7f1ffb640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7e40a3dc0 0x7fa7e40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.417+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f274800 0x560e1f276bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.417+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e1f1ce970 con 0x560e1f269620 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.417+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560e1f16b4a0 con 0x560e1f274800 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.417+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560e1f1ca930 con 0x560e1f269f40 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269620 0x560e1f269a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269620 0x560e1f269a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38728/0 (socket says 172.21.15.98:38728) 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 -- 172.21.15.98:0/263756930 learned_addr learned my addr 172.21.15.98:0/263756930 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f886b640 1 --2- 172.21.15.98:0/263756930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 0x560e1f2742c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7e40a3dc0 0x7fa7e40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f1ffb640 1 -- 172.21.15.98:0/1805493066 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fa7ec0f2c90 con 0x7fa7f419bdc0 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f906c640 1 --2- 172.21.15.98:0/263756930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f274800 0x560e1f276bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 -- 172.21.15.98:0/263756930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 msgr2=0x560e1f2742c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/263756930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 0x560e1f2742c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 -- 172.21.15.98:0/263756930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f274800 msgr2=0x560e1f276bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/263756930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f274800 0x560e1f276bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 -- 172.21.15.98:0/263756930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e1eee1210 con 0x560e1f269620 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f886b640 1 --2- 172.21.15.98:0/263756930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 0x560e1f2742c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:43:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/263756930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269620 0x560e1f269a00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0011a20 tx=0x7fa7e0011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6cb2399a9f395be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7e40a3dc0 0x7fa7e40a6280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0005e10 tx=0x7fa7e001c000 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7deffd640 1 -- 172.21.15.98:0/263756930 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0021070 con 0x560e1f269620 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7deffd640 1 -- 172.21.15.98:0/263756930 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7e000de90 con 0x560e1f269620 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.418+0000 7fa7deffd640 1 -- 172.21.15.98:0/263756930 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e00055d0 con 0x560e1f269620 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/263756930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269620 msgr2=0x560e1f269a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/263756930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269620 0x560e1f269a00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0011a20 tx=0x7fa7e0011ee0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/263756930 shutdown_connections 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/263756930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f274800 0x560e1f276bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/263756930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 0x560e1f2742c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/263756930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269620 0x560e1f269a00 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/263756930 >> 172.21.15.98:0/263756930 conn(0x560e1f265b30 msgr2=0x560e1f265f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/263756930 shutdown_connections 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/263756930 wait complete. 2024-04-25T17:43:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 -- start start 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.419+0000 7fa7fd7e6740 1 Processor -- start 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f274200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 0x560e1f2714f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f274800 0x560e1f271b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e1f2773d0 con 0x560e1f274800 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560e1f1690b0 con 0x560e1f269620 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560e1f277550 con 0x560e1f269f40 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7fd7e6740 10 client.0 _ll_register_callbacks cb 0x560e1f156d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f906c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f274800 0x560e1f271b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f906c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f274800 0x560e1f271b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38744/0 (socket says 172.21.15.98:38744) 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f906c640 1 -- 172.21.15.98:0/2468553419 learned_addr learned my addr 172.21.15.98:0/2468553419 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f886b640 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f274200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 0x560e1f2714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f906c640 1 -- 172.21.15.98:0/2468553419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 msgr2=0x560e1f2714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f906c640 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 0x560e1f2714f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f906c640 1 -- 172.21.15.98:0/2468553419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 msgr2=0x560e1f274200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f906c640 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f274200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f906c640 1 -- 172.21.15.98:0/2468553419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e1efad620 con 0x560e1f274800 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.420+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 0x560e1f2714f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7f906c640 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f274800 0x560e1f271b00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0bb100 tx=0x7fa7ec0be7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a4b2f711327921 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2468553419 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec0dde60 con 0x560e1f274800 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2468553419 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fa7ec0e2740 con 0x560e1f274800 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2468553419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7e000de90 con 0x560e1f274800 2024-04-25T17:43:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2468553419 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa7ec0bf3a0 con 0x560e1f274800 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7fd7e6740 10 client.16179 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2468553419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560e1f2773d0 con 0x560e1f274800 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2468553419 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7e000de90 con 0x560e1f274800 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2468553419 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fa7ec0f0660 con 0x560e1f274800 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7fd7e6740 10 client.16179 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7fd7e6740 10 client.16179 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7fd7e6740 20 client.16179 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560e1f2848d0 0x560e1f286cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7fd7e6740 4 client.16179 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.421+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2468553419 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e1f1dc390 con 0x560e1f2848d0 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.422+0000 7fa7f886b640 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560e1f2848d0 0x560e1f286cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.422+0000 7fa7f886b640 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560e1f2848d0 0x560e1f286cf0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8396a5e220aeb92 server_cookie=b5675088ce1629de in_seq=0 out_seq=0 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.422+0000 7fa7d3fff640 10 client.16179 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2468553419 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e1f1dc390 con 0x560e1f2848d0 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7d3fff640 10 client.16179 handle_command_reply: tid=0 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 client.16179 shutdown 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2468553419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f274800 msgr2=0x560e1f271b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f274800 0x560e1f271b00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0bb100 tx=0x7fa7ec0be7e0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2468553419 shutdown_connections 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560e1f2848d0 0x560e1f286cf0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f274800 0x560e1f271b00 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f269f40 0x560e1f2714f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2468553419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f274200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7d3fff640 0 client.16179 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2468553419 >> 172.21.15.98:0/2468553419 conn(0x560e1f265b30 msgr2=0x560e1f26b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2468553419 shutdown_connections 2024-04-25T17:43:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2468553419 wait complete. 2024-04-25T17:43:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.423+0000 7fa7fd7e6740 20 client.16179 trim_cache size 0 max 16384 2024-04-25T17:43:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7fd7e6740 1 Processor -- start 2024-04-25T17:43:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7fd7e6740 1 -- start start 2024-04-25T17:43:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f280290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f264b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f274800 0x560e1f273d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f41ae0a0 con 0x560e1f269f40 2024-04-25T17:43:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f41ab8b0 con 0x560e1f269620 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f41a9720 con 0x560e1f274800 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f264b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f264b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38760/0 (socket says 172.21.15.98:38760) 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7f3fff640 1 -- 172.21.15.98:0/674151159 learned_addr learned my addr 172.21.15.98:0/674151159 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7f3fff640 1 -- 172.21.15.98:0/674151159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f274800 msgr2=0x560e1f273d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/674151159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f274800 0x560e1f273d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7f3fff640 1 -- 172.21.15.98:0/674151159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 msgr2=0x560e1f280290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7f886b640 1 --2- 172.21.15.98:0/674151159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f280290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/674151159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f280290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.438+0000 7fa7f3fff640 1 -- 172.21.15.98:0/674151159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e1f287370 con 0x560e1f269f40 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/674151159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f264b30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00119f0 tx=0x7fa7e000ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74c05fbe30d7ea5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7de7fc640 1 -- 172.21.15.98:0/674151159 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0032070 con 0x560e1f269f40 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7de7fc640 1 -- 172.21.15.98:0/674151159 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7e000de90 con 0x560e1f269f40 2024-04-25T17:43:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7de7fc640 1 -- 172.21.15.98:0/674151159 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0021040 con 0x560e1f269f40 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/674151159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 msgr2=0x560e1f264b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/674151159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f264b30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00119f0 tx=0x7fa7e000ae60 comp rx=0 tx=0).stop 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/674151159 shutdown_connections 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/674151159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f274800 0x560e1f273d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/674151159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f264b30 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/674151159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f280290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.439+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/674151159 >> 172.21.15.98:0/674151159 conn(0x560e1f265b30 msgr2=0x560e1f264420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/674151159 shutdown_connections 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/674151159 wait complete. 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 1 -- start start 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 1 Processor -- start 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f26e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f26e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f274800 0x560e1f26eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f41a9d70 con 0x560e1f269f40 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f41ab1e0 con 0x560e1f269620 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f41aa440 con 0x560e1f274800 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7fd7e6740 10 client.0 _ll_register_callbacks cb 0x7fa7ec0bea70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f26e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7f3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f26e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38768/0 (socket says 172.21.15.98:38768) 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.440+0000 7fa7f3fff640 1 -- 172.21.15.98:0/2465948714 learned_addr learned my addr 172.21.15.98:0/2465948714 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7f906c640 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f274800 0x560e1f26eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7f886b640 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f26e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7f3fff640 1 -- 172.21.15.98:0/2465948714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f274800 msgr2=0x560e1f26eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f274800 0x560e1f26eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7f3fff640 1 -- 172.21.15.98:0/2465948714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 msgr2=0x560e1f26e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f26e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7f3fff640 1 -- 172.21.15.98:0/2465948714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e1f2e9e70 con 0x560e1f269f40 2024-04-25T17:43:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7f886b640 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f26e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7f3fff640 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f26e560 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0030040 tx=0x7fa7e000abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11dd18574d5e23c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2465948714 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0032070 con 0x560e1f269f40 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2465948714 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fa7e000dbe0 con 0x560e1f269f40 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2465948714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7e000de90 con 0x560e1f269f40 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2465948714 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fa7e0021050 con 0x560e1f269f40 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7fd7e6740 10 client.16185 fetch_fsmap learned FSMap version 38 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2465948714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560e1f2eb040 con 0x560e1f269f40 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.441+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2465948714 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7e000de90 con 0x560e1f269f40 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.442+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2465948714 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fa7e002d4d0 con 0x560e1f269f40 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.442+0000 7fa7fd7e6740 10 client.16185 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.442+0000 7fa7fd7e6740 10 client.16185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:43:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.442+0000 7fa7fd7e6740 20 client.16185 populate_metadata read hostname 'smithi098' 2024-04-25T17:43:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.442+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560e1f2eb5f0 0x560e1f2eda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:43:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.442+0000 7fa7fd7e6740 4 client.16185 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:43:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.442+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2465948714 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560e1f1d6200 con 0x560e1f2eb5f0 2024-04-25T17:43:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.442+0000 7fa7f886b640 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560e1f2eb5f0 0x560e1f2eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:43:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.442+0000 7fa7d3fff640 10 client.16185 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.442+0000 7fa7f886b640 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560e1f2eb5f0 0x560e1f2eda10 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5319fd86524036a server_cookie=ec8ece93f5013791 in_seq=0 out_seq=0 2024-04-25T17:43:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7d3fff640 1 -- 172.21.15.98:0/2465948714 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560e1f1d6200 con 0x560e1f2eb5f0 2024-04-25T17:43:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7d3fff640 10 client.16185 handle_command_reply: tid=0 2024-04-25T17:43:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 client.16185 shutdown 2024-04-25T17:43:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2465948714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 msgr2=0x560e1f26e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f26e560 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0030040 tx=0x7fa7e000abc0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2465948714 shutdown_connections 2024-04-25T17:43:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x560e1f2eb5f0 0x560e1f2eda10 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e1f274800 0x560e1f26eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e1f269f40 0x560e1f26e560 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7d3fff640 0 client.16185 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:43:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 --2- 172.21.15.98:0/2465948714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e1f269620 0x560e1f26e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2465948714 >> 172.21.15.98:0/2465948714 conn(0x560e1f265b30 msgr2=0x560e1f26f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2465948714 shutdown_connections 2024-04-25T17:43:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 1 -- 172.21.15.98:0/2465948714 wait complete. 2024-04-25T17:43:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.451+0000 7fa7fd7e6740 20 client.16185 trim_cache size 0 max 16384 2024-04-25T17:43:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:43:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:43:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:43:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:43:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 -- 172.21.15.98:0/1805493066 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7e40a3dc0 msgr2=0x7fa7e40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7e40a3dc0 0x7fa7e40a6280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0005e10 tx=0x7fa7e001c000 comp rx=0 tx=0).stop 2024-04-25T17:43:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 -- 172.21.15.98:0/1805493066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f419bdc0 msgr2=0x7fa7f419c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:43:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f419bdc0 0x7fa7f419c270 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec005cc0 tx=0x7fa7ec031680 comp rx=0 tx=0).stop 2024-04-25T17:43:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 -- 172.21.15.98:0/1805493066 shutdown_connections 2024-04-25T17:43:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7e40a3dc0 0x7fa7e40a6280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7f419bdc0 0x7fa7f419c270 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41061a0 0x7fa7f4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 --2- 172.21.15.98:0/1805493066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7f4101120 0x7fa7f4197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:43:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 -- 172.21.15.98:0/1805493066 >> 172.21.15.98:0/1805493066 conn(0x7fa7f4078060 msgr2=0x7fa7f40fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:43:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 -- 172.21.15.98:0/1805493066 shutdown_connections 2024-04-25T17:43:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:43:58.452+0000 7fa7faacf640 1 -- 172.21.15.98:0/1805493066 wait complete. 2024-04-25T17:43:58.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:43:58 smithi098 ceph-mon[29192]: pgmap v249: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.0 MiB/s wr, 769 op/s 2024-04-25T17:43:58.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:43:58 smithi203 ceph-mon[32834]: pgmap v249: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.0 MiB/s wr, 769 op/s 2024-04-25T17:43:58.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:43:58 smithi173 ceph-mon[32856]: pgmap v249: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.0 MiB/s wr, 769 op/s 2024-04-25T17:44:00.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:00 smithi098 ceph-mon[29192]: pgmap v250: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.4 MiB/s wr, 492 op/s 2024-04-25T17:44:00.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:00 smithi203 ceph-mon[32834]: pgmap v250: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.4 MiB/s wr, 492 op/s 2024-04-25T17:44:00.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:00 smithi173 ceph-mon[32856]: pgmap v250: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.4 MiB/s wr, 492 op/s 2024-04-25T17:44:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 450.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:44:02.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:02 smithi098 ceph-mon[29192]: pgmap v251: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 496 op/s 2024-04-25T17:44:02.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:02 smithi203 ceph-mon[32834]: pgmap v251: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 496 op/s 2024-04-25T17:44:02.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:02 smithi173 ceph-mon[32856]: pgmap v251: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.4 MiB/s wr, 496 op/s 2024-04-25T17:44:04.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:04 smithi098 ceph-mon[29192]: pgmap v252: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.8 MiB/s wr, 367 op/s 2024-04-25T17:44:04.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:04 smithi203 ceph-mon[32834]: pgmap v252: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.8 MiB/s wr, 367 op/s 2024-04-25T17:44:04.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:04 smithi173 ceph-mon[32856]: pgmap v252: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 3.8 MiB/s wr, 367 op/s 2024-04-25T17:44:06.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:06 smithi098 ceph-mon[29192]: pgmap v253: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 2.2 MiB/s wr, 185 op/s 2024-04-25T17:44:06.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:06 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:44:06.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:06 smithi098 ceph-mon[29192]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T17:44:06.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:06 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:44:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:06 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:44:06.456 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:06 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:44:06.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:06 smithi203 ceph-mon[32834]: pgmap v253: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 2.2 MiB/s wr, 185 op/s 2024-04-25T17:44:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:06 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:44:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:06 smithi203 ceph-mon[32834]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T17:44:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:06 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:44:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:06 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:44:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:06 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:44:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:06 smithi173 ceph-mon[32856]: pgmap v253: 97 pgs: 97 active+clean; 3.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 2.2 MiB/s wr, 185 op/s 2024-04-25T17:44:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:06 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:44:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:06 smithi173 ceph-mon[32856]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T17:44:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:06 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:44:06.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:06 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:44:06.523 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:06 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:44:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 455.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:44:08.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:08 smithi098 ceph-mon[29192]: pgmap v255: 97 pgs: 21 active+clean+snaptrim, 42 active+clean, 34 active+clean+snaptrim_wait; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 MiB/s wr, 1.28k op/s 2024-04-25T17:44:08.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:08 smithi203 ceph-mon[32834]: pgmap v255: 97 pgs: 21 active+clean+snaptrim, 42 active+clean, 34 active+clean+snaptrim_wait; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 MiB/s wr, 1.28k op/s 2024-04-25T17:44:08.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:08 smithi173 ceph-mon[32856]: pgmap v255: 97 pgs: 21 active+clean+snaptrim, 42 active+clean, 34 active+clean+snaptrim_wait; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 MiB/s wr, 1.28k op/s 2024-04-25T17:44:10.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:10 smithi098 ceph-mon[29192]: pgmap v256: 97 pgs: 21 active+clean+snaptrim, 42 active+clean, 34 active+clean+snaptrim_wait; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 MiB/s wr, 1.28k op/s 2024-04-25T17:44:10.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:10 smithi203 ceph-mon[32834]: pgmap v256: 97 pgs: 21 active+clean+snaptrim, 42 active+clean, 34 active+clean+snaptrim_wait; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 MiB/s wr, 1.28k op/s 2024-04-25T17:44:10.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:10 smithi173 ceph-mon[32856]: pgmap v256: 97 pgs: 21 active+clean+snaptrim, 42 active+clean, 34 active+clean+snaptrim_wait; 3.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 MiB/s wr, 1.28k op/s 2024-04-25T17:44:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.547+0000 7fe868ea0640 1 -- 172.21.15.98:0/2728441566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe864101130 msgr2=0x7fe8641015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.547+0000 7fe868ea0640 1 --2- 172.21.15.98:0/2728441566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe864101130 0x7fe8641015b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009930 tx=0x7fe84c02a420 comp rx=0 tx=0).stop 2024-04-25T17:44:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.547+0000 7fe868ea0640 1 -- 172.21.15.98:0/2728441566 shutdown_connections 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.547+0000 7fe868ea0640 1 --2- 172.21.15.98:0/2728441566 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe864101af0 0x7fe86410dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.547+0000 7fe868ea0640 1 --2- 172.21.15.98:0/2728441566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe864101130 0x7fe8641015b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.547+0000 7fe868ea0640 1 --2- 172.21.15.98:0/2728441566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864106f70 0x7fe864107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.547+0000 7fe868ea0640 1 -- 172.21.15.98:0/2728441566 >> 172.21.15.98:0/2728441566 conn(0x7fe864077ba0 msgr2=0x7fe864077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.547+0000 7fe868ea0640 1 -- 172.21.15.98:0/2728441566 shutdown_connections 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.547+0000 7fe868ea0640 1 -- 172.21.15.98:0/2728441566 wait complete. 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe868ea0640 1 Processor -- start 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe868ea0640 1 -- start start 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe868ea0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe864101130 0x7fe864102b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe868ea0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe864101af0 0x7fe864103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe868ea0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864106f70 0x7fe864103580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe868ea0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641110b0 con 0x7fe864101af0 2024-04-25T17:44:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe868ea0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe864110f30 con 0x7fe864106f70 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe868ea0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe864111230 con 0x7fe864101130 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe85bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe864101af0 0x7fe864103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe85bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe864101af0 0x7fe864103040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51066/0 (socket says 172.21.15.98:51066) 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.548+0000 7fe85bfff640 1 -- 172.21.15.98:0/2793736747 learned_addr learned my addr 172.21.15.98:0/2793736747 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe862d9d640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864106f70 0x7fe864103580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe86259c640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe864101130 0x7fe864102b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe85bfff640 1 -- 172.21.15.98:0/2793736747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe864101130 msgr2=0x7fe864102b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe85bfff640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe864101130 0x7fe864102b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe85bfff640 1 -- 172.21.15.98:0/2793736747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864106f70 msgr2=0x7fe864103580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe85bfff640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864106f70 0x7fe864103580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe85bfff640 1 -- 172.21.15.98:0/2793736747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8641a6b60 con 0x7fe864101af0 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe85bfff640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe864101af0 0x7fe864103040 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009230 tx=0x7fe84c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe85b7fe640 1 -- 172.21.15.98:0/2793736747 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c03f030 con 0x7fe864101af0 2024-04-25T17:44:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe868ea0640 1 -- 172.21.15.98:0/2793736747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8641a6df0 con 0x7fe864101af0 2024-04-25T17:44:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe85b7fe640 1 -- 172.21.15.98:0/2793736747 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe84c003aa0 con 0x7fe864101af0 2024-04-25T17:44:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe85b7fe640 1 -- 172.21.15.98:0/2793736747 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c035070 con 0x7fe864101af0 2024-04-25T17:44:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.549+0000 7fe868ea0640 1 -- 172.21.15.98:0/2793736747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8641a7260 con 0x7fe864101af0 2024-04-25T17:44:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.552+0000 7fe86bbb9740 1 Processor -- start 2024-04-25T17:44:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.552+0000 7fe85b7fe640 1 -- 172.21.15.98:0/2793736747 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe84c042030 con 0x7fe864101af0 2024-04-25T17:44:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.552+0000 7fe86bbb9740 1 -- start start 2024-04-25T17:44:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf31ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf320a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559bf3164970 con 0x559bf320a800 2024-04-25T17:44:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559bf31014a0 con 0x559bf31ff620 2024-04-25T17:44:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559bf3160930 con 0x559bf31fff40 2024-04-25T17:44:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe85bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe862d9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf31ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe85bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51068/0 (socket says 172.21.15.98:51068) 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe85bfff640 1 -- 172.21.15.98:0/4160652452 learned_addr learned my addr 172.21.15.98:0/4160652452 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe86259c640 1 --2- 172.21.15.98:0/4160652452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf320a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe85b7fe640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe8500a3dc0 0x7fe8500a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe862d9d640 1 -- 172.21.15.98:0/4160652452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 msgr2=0x559bf320a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe862d9d640 1 --2- 172.21.15.98:0/4160652452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf320a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe862d9d640 1 -- 172.21.15.98:0/4160652452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 msgr2=0x559bf320cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe862d9d640 1 --2- 172.21.15.98:0/4160652452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe862d9d640 1 -- 172.21.15.98:0/4160652452 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bf2e77210 con 0x559bf31ff620 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe85bfff640 1 --2- 172.21.15.98:0/4160652452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe862d9d640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe8500a3dc0 0x7fe8500a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe85b7fe640 1 -- 172.21.15.98:0/2793736747 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8557+0+0 (secure 0 0 0) 0x7fe84c0f0e10 con 0x7fe864101af0 2024-04-25T17:44:11.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.553+0000 7fe862d9d640 1 --2- 172.21.15.98:0/4160652452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf31ffa00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009fa0 tx=0x7fe85c010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d4acfa3bfe35e4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe858ff9640 1 -- 172.21.15.98:0/4160652452 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c01c900 con 0x559bf31ff620 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe862d9d640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe8500a3dc0 0x7fe8500a6280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe85c020000 tx=0x7fe85c01d000 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe858ff9640 1 -- 172.21.15.98:0/4160652452 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe85c0113e0 con 0x559bf31ff620 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe858ff9640 1 -- 172.21.15.98:0/4160652452 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c01b630 con 0x559bf31ff620 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe86bbb9740 1 -- 172.21.15.98:0/4160652452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 msgr2=0x559bf31ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/4160652452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf31ffa00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009fa0 tx=0x7fe85c010800 comp rx=0 tx=0).stop 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe86bbb9740 1 -- 172.21.15.98:0/4160652452 shutdown_connections 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/4160652452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/4160652452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf320a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/4160652452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf31ffa00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.554+0000 7fe86bbb9740 1 -- 172.21.15.98:0/4160652452 >> 172.21.15.98:0/4160652452 conn(0x559bf31fbb30 msgr2=0x559bf31fbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 1 -- 172.21.15.98:0/4160652452 shutdown_connections 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 1 -- 172.21.15.98:0/4160652452 wait complete. 2024-04-25T17:44:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 1 -- start start 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 1 Processor -- start 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf320a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf31fff40 0x559bf32073f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf320a800 0x559bf3207930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559bf320d430 con 0x559bf31fff40 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559bf30ff0b0 con 0x559bf31ff620 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559bf320d5b0 con 0x559bf320a800 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe86bbb9740 10 client.0 _ll_register_callbacks cb 0x559bf30ecd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe85bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf31fff40 0x559bf32073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe85bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf31fff40 0x559bf32073f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51080/0 (socket says 172.21.15.98:51080) 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.555+0000 7fe85bfff640 1 -- 172.21.15.98:0/3200120955 learned_addr learned my addr 172.21.15.98:0/3200120955 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe86259c640 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf320a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe862d9d640 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf320a800 0x559bf3207930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe85bfff640 1 -- 172.21.15.98:0/3200120955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf320a800 msgr2=0x559bf3207930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe85bfff640 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf320a800 0x559bf3207930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe85bfff640 1 -- 172.21.15.98:0/3200120955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 msgr2=0x559bf320a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe85bfff640 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf320a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe85bfff640 1 -- 172.21.15.98:0/3200120955 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bf2f43620 con 0x559bf31fff40 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe85bfff640 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf31fff40 0x559bf32073f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe84c0ee5f0 tx=0x7fe84c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9769d092eb4ceb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe849ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe849ffb640 1 -- 172.21.15.98:0/3200120955 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c049040 con 0x559bf31fff40 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3200120955 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe85c0113e0 con 0x559bf31fff40 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe849ffb640 1 -- 172.21.15.98:0/3200120955 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8557+0+0 (secure 0 0 0) 0x7fe84c0ee7c0 con 0x559bf31fff40 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe849ffb640 1 -- 172.21.15.98:0/3200120955 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe84c0bcb60 con 0x559bf31fff40 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe86bbb9740 10 client.16197 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3200120955 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559bf32193a0 con 0x559bf31fff40 2024-04-25T17:44:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.556+0000 7fe849ffb640 1 -- 172.21.15.98:0/3200120955 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe85c0113e0 con 0x559bf31fff40 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.557+0000 7fe849ffb640 1 -- 172.21.15.98:0/3200120955 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fe84c0b9400 con 0x559bf31fff40 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.557+0000 7fe86bbb9740 10 client.16197 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.557+0000 7fe86bbb9740 10 client.16197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.557+0000 7fe86bbb9740 20 client.16197 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.557+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf321a850 0x559bf321ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.557+0000 7fe86bbb9740 4 client.16197 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.557+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3200120955 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bf3172390 con 0x559bf321a850 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.557+0000 7fe86259c640 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf321a850 0x559bf321ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.557+0000 7fe849ffb640 10 client.16197 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.557+0000 7fe86259c640 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf321a850 0x559bf321ccb0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1384ffb3b34b5bb5 server_cookie=59de4046b527e37 in_seq=0 out_seq=0 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe849ffb640 1 -- 172.21.15.98:0/3200120955 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559bf3172390 con 0x559bf321a850 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe849ffb640 10 client.16197 handle_command_reply: tid=0 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 client.16197 shutdown 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3200120955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf31fff40 msgr2=0x559bf32073f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf31fff40 0x559bf32073f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe84c0ee5f0 tx=0x7fe84c005c90 comp rx=0 tx=0).stop 2024-04-25T17:44:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3200120955 shutdown_connections 2024-04-25T17:44:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf321a850 0x559bf321ccb0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf320a800 0x559bf3207930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf31fff40 0x559bf32073f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe849ffb640 0 client.16197 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/3200120955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf320a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3200120955 >> 172.21.15.98:0/3200120955 conn(0x559bf31fbb30 msgr2=0x559bf3201860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3200120955 shutdown_connections 2024-04-25T17:44:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3200120955 wait complete. 2024-04-25T17:44:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.558+0000 7fe86bbb9740 20 client.16197 trim_cache size 0 max 16384 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe86bbb9740 1 Processor -- start 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe86bbb9740 1 -- start start 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf3216350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf3209c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf32225c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641b26c0 con 0x559bf320a800 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641afed0 con 0x559bf31ff620 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641add40 con 0x559bf31fff40 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe862d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf32225c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe862d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf32225c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51088/0 (socket says 172.21.15.98:51088) 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.573+0000 7fe862d9d640 1 -- 172.21.15.98:0/3719915192 learned_addr learned my addr 172.21.15.98:0/3719915192 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.574+0000 7fe862d9d640 1 -- 172.21.15.98:0/3719915192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 msgr2=0x559bf3209c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.574+0000 7fe862d9d640 1 --2- 172.21.15.98:0/3719915192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf3209c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.574+0000 7fe862d9d640 1 -- 172.21.15.98:0/3719915192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 msgr2=0x559bf3216350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.574+0000 7fe862d9d640 1 --2- 172.21.15.98:0/3719915192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf3216350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.574+0000 7fe862d9d640 1 -- 172.21.15.98:0/3719915192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bf321d330 con 0x559bf320a800 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.574+0000 7fe862d9d640 1 --2- 172.21.15.98:0/3719915192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf32225c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe85c005fd0 tx=0x7fe85c011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28bb2f098f58f5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.574+0000 7fe84bfff640 1 -- 172.21.15.98:0/3719915192 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c01b3d0 con 0x559bf320a800 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.574+0000 7fe84bfff640 1 -- 172.21.15.98:0/3719915192 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe85c035070 con 0x559bf320a800 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.574+0000 7fe84bfff640 1 -- 172.21.15.98:0/3719915192 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c01ba80 con 0x559bf320a800 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3719915192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 msgr2=0x559bf32225c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/3719915192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf32225c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe85c005fd0 tx=0x7fe85c011d70 comp rx=0 tx=0).stop 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3719915192 shutdown_connections 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/3719915192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf32225c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/3719915192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf3209c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/3719915192 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf3216350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3719915192 >> 172.21.15.98:0/3719915192 conn(0x559bf31fbb30 msgr2=0x559bf32029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3719915192 shutdown_connections 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 -- 172.21.15.98:0/3719915192 wait complete. 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 -- start start 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 Processor -- start 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf320e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf320e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641ae390 con 0x559bf320a800 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641af800 con 0x559bf31ff620 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641aea60 con 0x559bf31fff40 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe86bbb9740 10 client.0 _ll_register_callbacks cb 0x559bf30ecd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe862d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe862d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51100/0 (socket says 172.21.15.98:51100) 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.575+0000 7fe862d9d640 1 -- 172.21.15.98:0/310346422 learned_addr learned my addr 172.21.15.98:0/310346422 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe85bfff640 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf320e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe86259c640 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf320e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe862d9d640 1 -- 172.21.15.98:0/310346422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 msgr2=0x559bf320e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe862d9d640 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf320e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe862d9d640 1 -- 172.21.15.98:0/310346422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 msgr2=0x559bf320e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe862d9d640 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf320e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe862d9d640 1 -- 172.21.15.98:0/310346422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bf327fe30 con 0x559bf320a800 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe862d9d640 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320ed00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009660 tx=0x7fe85c009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d1ff2e324a9cfa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe849ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe849ffb640 1 -- 172.21.15.98:0/310346422 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c03d070 con 0x559bf320a800 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe849ffb640 1 -- 172.21.15.98:0/310346422 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8557+0+0 (secure 0 0 0) 0x7fe85c013600 con 0x559bf320a800 2024-04-25T17:44:11.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe86bbb9740 1 -- 172.21.15.98:0/310346422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe85c035070 con 0x559bf320a800 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe849ffb640 1 -- 172.21.15.98:0/310346422 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fe85c049050 con 0x559bf320a800 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe86bbb9740 10 client.16203 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.576+0000 7fe86bbb9740 1 -- 172.21.15.98:0/310346422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559bf3281120 con 0x559bf320a800 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe849ffb640 1 -- 172.21.15.98:0/310346422 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe85c035070 con 0x559bf320a800 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe849ffb640 1 -- 172.21.15.98:0/310346422 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fe85c00f040 con 0x559bf320a800 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe86bbb9740 10 client.16203 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe86bbb9740 10 client.16203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe86bbb9740 20 client.16203 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf32816a0 0x559bf3283ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe86bbb9740 4 client.16203 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe86bbb9740 1 -- 172.21.15.98:0/310346422 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559bf3207cc0 con 0x559bf32816a0 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe86259c640 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf32816a0 0x559bf3283ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe86259c640 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf32816a0 0x559bf3283ac0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2289d86685c7261 server_cookie=f4206d24293cc4f4 in_seq=0 out_seq=0 2024-04-25T17:44:11.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.577+0000 7fe849ffb640 10 client.16203 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe849ffb640 1 -- 172.21.15.98:0/310346422 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559bf3207cc0 con 0x559bf32816a0 2024-04-25T17:44:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe849ffb640 10 client.16203 handle_command_reply: tid=0 2024-04-25T17:44:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe86bbb9740 1 client.16203 shutdown 2024-04-25T17:44:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe86bbb9740 1 -- 172.21.15.98:0/310346422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 msgr2=0x559bf320ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320ed00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009660 tx=0x7fe85c009850 comp rx=0 tx=0).stop 2024-04-25T17:44:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe86bbb9740 1 -- 172.21.15.98:0/310346422 shutdown_connections 2024-04-25T17:44:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559bf32816a0 0x559bf3283ac0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bf320a800 0x559bf320ed00 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe849ffb640 0 client.16203 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bf31fff40 0x559bf320e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe86bbb9740 1 --2- 172.21.15.98:0/310346422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bf31ff620 0x559bf320e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.586+0000 7fe86bbb9740 1 -- 172.21.15.98:0/310346422 >> 172.21.15.98:0/310346422 conn(0x559bf31fbb30 msgr2=0x559bf3202140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe86bbb9740 1 -- 172.21.15.98:0/310346422 shutdown_connections 2024-04-25T17:44:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe86bbb9740 1 -- 172.21.15.98:0/310346422 wait complete. 2024-04-25T17:44:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe86bbb9740 20 client.16203 trim_cache size 0 max 16384 2024-04-25T17:44:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:44:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:44:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:44:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:44:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 -- 172.21.15.98:0/2793736747 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe8500a3dc0 msgr2=0x7fe8500a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe8500a3dc0 0x7fe8500a6280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe85c020000 tx=0x7fe85c01d000 comp rx=0 tx=0).stop 2024-04-25T17:44:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 -- 172.21.15.98:0/2793736747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe864101af0 msgr2=0x7fe864103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe864101af0 0x7fe864103040 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009230 tx=0x7fe84c004270 comp rx=0 tx=0).stop 2024-04-25T17:44:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 -- 172.21.15.98:0/2793736747 shutdown_connections 2024-04-25T17:44:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe8500a3dc0 0x7fe8500a6280 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe864106f70 0x7fe864103580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe864101af0 0x7fe864103040 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 --2- 172.21.15.98:0/2793736747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe864101130 0x7fe864102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 -- 172.21.15.98:0/2793736747 >> 172.21.15.98:0/2793736747 conn(0x7fe864077ba0 msgr2=0x7fe8641003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 -- 172.21.15.98:0/2793736747 shutdown_connections 2024-04-25T17:44:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:11.587+0000 7fe868ea0640 1 -- 172.21.15.98:0/2793736747 wait complete. 2024-04-25T17:44:11.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 460.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:44:12.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:12 smithi098 ceph-mon[29192]: pgmap v257: 97 pgs: 15 active+clean+snaptrim, 55 active+clean, 27 active+clean+snaptrim_wait; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 28 MiB/s wr, 1.28k op/s 2024-04-25T17:44:12.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:12 smithi203 ceph-mon[32834]: pgmap v257: 97 pgs: 15 active+clean+snaptrim, 55 active+clean, 27 active+clean+snaptrim_wait; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 28 MiB/s wr, 1.28k op/s 2024-04-25T17:44:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:12 smithi173 ceph-mon[32856]: pgmap v257: 97 pgs: 15 active+clean+snaptrim, 55 active+clean, 27 active+clean+snaptrim_wait; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 28 MiB/s wr, 1.28k op/s 2024-04-25T17:44:14.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:14 smithi098 ceph-mon[29192]: pgmap v258: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 MiB/s wr, 1.34k op/s 2024-04-25T17:44:14.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:14 smithi203 ceph-mon[32834]: pgmap v258: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 MiB/s wr, 1.34k op/s 2024-04-25T17:44:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:14 smithi173 ceph-mon[32856]: pgmap v258: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 MiB/s wr, 1.34k op/s 2024-04-25T17:44:16.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:16 smithi098 ceph-mon[29192]: pgmap v259: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 MiB/s wr, 1.34k op/s 2024-04-25T17:44:16.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:16 smithi098 ceph-mon[29192]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T17:44:16.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:16 smithi203 ceph-mon[32834]: pgmap v259: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 MiB/s wr, 1.34k op/s 2024-04-25T17:44:16.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:16 smithi203 ceph-mon[32834]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T17:44:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:16 smithi173 ceph-mon[32856]: pgmap v259: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 MiB/s wr, 1.34k op/s 2024-04-25T17:44:16.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:16 smithi173 ceph-mon[32856]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T17:44:16.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 465.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:44:18.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:18 smithi098 ceph-mon[29192]: pgmap v261: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-04-25T17:44:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:18 smithi203 ceph-mon[32834]: pgmap v261: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-04-25T17:44:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:18 smithi173 ceph-mon[32856]: pgmap v261: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-04-25T17:44:18.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:44:19.068 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:44:20.373 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:20 smithi098 ceph-mon[29192]: pgmap v262: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-04-25T17:44:20.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:20 smithi203 ceph-mon[32834]: pgmap v262: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-04-25T17:44:20.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:20 smithi173 ceph-mon[32856]: pgmap v262: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-04-25T17:44:20.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.760+0000 7f9f124b3700 1 -- 172.21.15.98:0/853351411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f0c102a90 msgr2=0x7f9f0c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:20.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.760+0000 7f9f124b3700 1 --2- 172.21.15.98:0/853351411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f0c102a90 0x7f9f0c102f10 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9f00009990 tx=0x7f9f00009ca0 comp rx=0 tx=0).stop 2024-04-25T17:44:20.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.760+0000 7f9f124b3700 1 -- 172.21.15.98:0/853351411 shutdown_connections 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.760+0000 7f9f124b3700 1 --2- 172.21.15.98:0/853351411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f0c108a50 0x7f9f0c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.761+0000 7f9f124b3700 1 --2- 172.21.15.98:0/853351411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f0c103450 0x7f9f0c10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.761+0000 7f9f124b3700 1 --2- 172.21.15.98:0/853351411 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f0c102a90 0x7f9f0c102f10 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.761+0000 7f9f124b3700 1 -- 172.21.15.98:0/853351411 >> 172.21.15.98:0/853351411 conn(0x7f9f0c079ba0 msgr2=0x7f9f0c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.761+0000 7f9f124b3700 1 -- 172.21.15.98:0/853351411 shutdown_connections 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.761+0000 7f9f124b3700 1 -- 172.21.15.98:0/853351411 wait complete. 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.761+0000 7f9f124b3700 1 Processor -- start 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.761+0000 7f9f124b3700 1 -- start start 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f124b3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f0c102a90 0x7f9f0c199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f124b3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f0c103450 0x7f9f0c1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f124b3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f0c108a50 0x7f9f0c19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f124b3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c112be0 con 0x7f9f0c102a90 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f124b3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c112a60 con 0x7f9f0c103450 2024-04-25T17:44:20.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f124b3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c112d60 con 0x7f9f0c108a50 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f0c102a90 0x7f9f0c199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f0c102a90 0x7f9f0c199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42518/0 (socket says 172.21.15.98:42518) 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0bfff700 1 -- 172.21.15.98:0/688503214 learned_addr learned my addr 172.21.15.98:0/688503214 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0b7fe700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f0c103450 0x7f9f0c1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0bfff700 1 -- 172.21.15.98:0/688503214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f0c108a50 msgr2=0x7f9f0c19d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0bfff700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f0c108a50 0x7f9f0c19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0bfff700 1 -- 172.21.15.98:0/688503214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f0c103450 msgr2=0x7f9f0c1995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0bfff700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f0c103450 0x7f9f0c1995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0bfff700 1 -- 172.21.15.98:0/688503214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0c19e050 con 0x7f9f0c102a90 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0b7fe700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f0c103450 0x7f9f0c1995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f0bfff700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f0c102a90 0x7f9f0c199090 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00c930 tx=0x7f9efc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f097fa700 1 -- 172.21.15.98:0/688503214 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc007b00 con 0x7f9f0c102a90 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f097fa700 1 -- 172.21.15.98:0/688503214 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9efc007ca0 con 0x7f9f0c102a90 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f124b3700 1 -- 172.21.15.98:0/688503214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f0c19e340 con 0x7f9f0c102a90 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f097fa700 1 -- 172.21.15.98:0/688503214 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc018660 con 0x7f9f0c102a90 2024-04-25T17:44:20.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.762+0000 7f9f124b3700 1 -- 172.21.15.98:0/688503214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0c1a5bc0 con 0x7f9f0c102a90 2024-04-25T17:44:20.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.763+0000 7f9f124b3700 1 -- 172.21.15.98:0/688503214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f0c069730 con 0x7f9f0c102a90 2024-04-25T17:44:20.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.765+0000 7f9f097fa700 1 -- 172.21.15.98:0/688503214 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9efc00f450 con 0x7f9f0c102a90 2024-04-25T17:44:20.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.765+0000 7f9f097fa700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9ef40a3db0 0x7f9ef40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:20.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.766+0000 7f9f0b7fe700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9ef40a3db0 0x7f9ef40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:20.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.766+0000 7f9f097fa700 1 -- 172.21.15.98:0/688503214 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f9efc0d0a40 con 0x7f9f0c102a90 2024-04-25T17:44:20.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.766+0000 7f9f0b7fe700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9ef40a3db0 0x7f9ef40a6270 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9f00006010 tx=0x7f9f0001a040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:20.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:20.769+0000 7f9f097fa700 1 -- 172.21.15.98:0/688503214 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9efc014030 con 0x7f9f0c102a90 2024-04-25T17:44:21.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.013+0000 7f9f124b3700 1 -- 172.21.15.98:0/688503214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f0c02df80 con 0x7f9f0c102a90 2024-04-25T17:44:21.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.013+0000 7f9f097fa700 1 -- 172.21.15.98:0/688503214 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7f9efc09d170 con 0x7f9f0c102a90 2024-04-25T17:44:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:44:21.015 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:42:52.734463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:44:21.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 -- 172.21.15.98:0/688503214 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9ef40a3db0 msgr2=0x7f9ef40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:21.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9ef40a3db0 0x7f9ef40a6270 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9f00006010 tx=0x7f9f0001a040 comp rx=0 tx=0).stop 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 -- 172.21.15.98:0/688503214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f0c102a90 msgr2=0x7f9f0c199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f0c102a90 0x7f9f0c199090 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00c930 tx=0x7f9efc00ccf0 comp rx=0 tx=0).stop 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 -- 172.21.15.98:0/688503214 shutdown_connections 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f9ef40a3db0 0x7f9ef40a6270 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f0c108a50 0x7f9f0c19d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f0c102a90 0x7f9f0c199090 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 --2- 172.21.15.98:0/688503214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f0c103450 0x7f9f0c1995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 -- 172.21.15.98:0/688503214 >> 172.21.15.98:0/688503214 conn(0x7f9f0c079ba0 msgr2=0x7f9f0c101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 -- 172.21.15.98:0/688503214 shutdown_connections 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:21.016+0000 7f9f124b3700 1 -- 172.21.15.98:0/688503214 wait complete. 2024-04-25T17:44:21.017 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 38 2024-04-25T17:44:21.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:21 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/688503214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:44:21.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:21 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/688503214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:44:21.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:21 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/688503214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:44:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 470.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:44:21.689 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id b20338f --release --await' 2024-04-25T17:44:21.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id b20338f --release --await 2024-04-25T17:44:21.928 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:44:22.363 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:22 smithi203 ceph-mon[32834]: pgmap v263: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.3 MiB/s wr, 66 op/s 2024-04-25T17:44:22.380 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:22 smithi098 ceph-mon[29192]: pgmap v263: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.3 MiB/s wr, 66 op/s 2024-04-25T17:44:22.422 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:22 smithi173 ceph-mon[32856]: pgmap v263: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 1.3 MiB/s wr, 66 op/s 2024-04-25T17:44:23.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.555+0000 7fd8c588b700 1 -- 172.21.15.98:0/1196107642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c006b0f0 msgr2=0x7fd8c010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:23.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.555+0000 7fd8c588b700 1 --2- 172.21.15.98:0/1196107642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c006b0f0 0x7fd8c010f900 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800b0a0 tx=0x7fd8b800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.555+0000 7fd8c588b700 1 -- 172.21.15.98:0/1196107642 shutdown_connections 2024-04-25T17:44:23.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.555+0000 7fd8c588b700 1 --2- 172.21.15.98:0/1196107642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c0108a30 0x7fd8c0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.555+0000 7fd8c588b700 1 --2- 172.21.15.98:0/1196107642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c006b0f0 0x7fd8c010f900 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.555+0000 7fd8c588b700 1 --2- 172.21.15.98:0/1196107642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c006a750 0x7fd8c006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.555+0000 7fd8c588b700 1 -- 172.21.15.98:0/1196107642 >> 172.21.15.98:0/1196107642 conn(0x7fd8c00788f0 msgr2=0x7fd8c0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:23.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.555+0000 7fd8c588b700 1 -- 172.21.15.98:0/1196107642 shutdown_connections 2024-04-25T17:44:23.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.555+0000 7fd8c588b700 1 -- 172.21.15.98:0/1196107642 wait complete. 2024-04-25T17:44:23.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.555+0000 7fd8c588b700 1 Processor -- start 2024-04-25T17:44:23.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8c588b700 1 -- start start 2024-04-25T17:44:23.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8c588b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c006a750 0x7fd8c0101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8c588b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c006b0f0 0x7fd8c0101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8c588b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c0108a30 0x7fd8c0102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8c588b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0112bd0 con 0x7fd8c006b0f0 2024-04-25T17:44:23.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8c588b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0112a50 con 0x7fd8c006a750 2024-04-25T17:44:23.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8c588b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c0112d50 con 0x7fd8c0108a30 2024-04-25T17:44:23.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8b67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c006b0f0 0x7fd8c0101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8b67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c006b0f0 0x7fd8c0101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49890/0 (socket says 172.21.15.98:49890) 2024-04-25T17:44:23.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8b67fc700 1 -- 172.21.15.98:0/934852998 learned_addr learned my addr 172.21.15.98:0/934852998 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:23.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8bf7fe700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c0108a30 0x7fd8c0102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8beffd700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c006a750 0x7fd8c0101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8b67fc700 1 -- 172.21.15.98:0/934852998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c0108a30 msgr2=0x7fd8c0102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:23.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c0108a30 0x7fd8c0102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8b67fc700 1 -- 172.21.15.98:0/934852998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c006a750 msgr2=0x7fd8c0101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:23.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c006a750 0x7fd8c0101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8b67fc700 1 -- 172.21.15.98:0/934852998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c01a75d0 con 0x7fd8c006b0f0 2024-04-25T17:44:23.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.556+0000 7fd8beffd700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c006a750 0x7fd8c0101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:44:23.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.557+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c006b0f0 0x7fd8c0101f50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800e9a0 tx=0x7fd8a800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:23.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.557+0000 7fd8c4889700 1 -- 172.21.15.98:0/934852998 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a800cbd0 con 0x7fd8c006b0f0 2024-04-25T17:44:23.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.557+0000 7fd8c4889700 1 -- 172.21.15.98:0/934852998 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8a800cd70 con 0x7fd8c006b0f0 2024-04-25T17:44:23.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.557+0000 7fd8c4889700 1 -- 172.21.15.98:0/934852998 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8018680 con 0x7fd8c006b0f0 2024-04-25T17:44:23.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.557+0000 7fd8c588b700 1 -- 172.21.15.98:0/934852998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c01a78c0 con 0x7fd8c006b0f0 2024-04-25T17:44:23.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.557+0000 7fd8c588b700 1 -- 172.21.15.98:0/934852998 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c01a7e00 con 0x7fd8c006b0f0 2024-04-25T17:44:23.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.559+0000 7fd8c4889700 1 -- 172.21.15.98:0/934852998 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd8a8010970 con 0x7fd8c006b0f0 2024-04-25T17:44:23.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.560+0000 7fd8c4889700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd8ac0a3db0 0x7fd8ac0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.560+0000 7fd8beffd700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd8ac0a3db0 0x7fd8ac0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.560+0000 7fd8c4889700 1 -- 172.21.15.98:0/934852998 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fd8a80d0ba0 con 0x7fd8c006b0f0 2024-04-25T17:44:23.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.560+0000 7fd8beffd700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd8ac0a3db0 0x7fd8ac0a6270 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009bc0 tx=0x7fd8b0009260 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.560+0000 7fd8d0114b80 1 Processor -- start 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.560+0000 7fd8d0114b80 1 -- start start 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c09f0 0x55d3d79c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c1310 0x55d3d79cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 0x55d3d79ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d3d7913e30 con 0x55d3d79c09f0 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d3d78baa60 con 0x55d3d79c1310 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d3d790fdf0 con 0x55d3d79cc1e0 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8beffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c09f0 0x55d3d79c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8beffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c09f0 0x55d3d79c0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49904/0 (socket says 172.21.15.98:49904) 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8beffd700 1 -- 172.21.15.98:0/1206796129 learned_addr learned my addr 172.21.15.98:0/1206796129 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/1206796129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c1310 0x55d3d79cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8beffd700 1 -- 172.21.15.98:0/1206796129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 msgr2=0x55d3d79ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8bf7fe700 1 --2- 172.21.15.98:0/1206796129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 0x55d3d79ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8beffd700 1 --2- 172.21.15.98:0/1206796129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 0x55d3d79ce5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8beffd700 1 -- 172.21.15.98:0/1206796129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c1310 msgr2=0x55d3d79cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8beffd700 1 --2- 172.21.15.98:0/1206796129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c1310 0x55d3d79cbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8beffd700 1 -- 172.21.15.98:0/1206796129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3d78619d0 con 0x55d3d79c09f0 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/1206796129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c1310 0x55d3d79cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8beffd700 1 --2- 172.21.15.98:0/1206796129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c09f0 0x55d3d79c0dd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000fcd0 tx=0x7fd8b0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8747a940c65751b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8b5ffb700 1 -- 172.21.15.98:0/1206796129 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0022070 con 0x55d3d79c09f0 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8b5ffb700 1 -- 172.21.15.98:0/1206796129 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b0004ce0 con 0x55d3d79c09f0 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.561+0000 7fd8b5ffb700 1 -- 172.21.15.98:0/1206796129 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0005580 con 0x55d3d79c09f0 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.562+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1206796129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c09f0 msgr2=0x55d3d79c0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.562+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/1206796129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c09f0 0x55d3d79c0dd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000fcd0 tx=0x7fd8b0012000 comp rx=0 tx=0).stop 2024-04-25T17:44:23.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.562+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1206796129 shutdown_connections 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.562+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/1206796129 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 0x55d3d79ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.562+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/1206796129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c09f0 0x55d3d79c0dd0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.562+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/1206796129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c1310 0x55d3d79cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.562+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1206796129 >> 172.21.15.98:0/1206796129 conn(0x55d3d78ee790 msgr2=0x55d3d79bced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.562+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1206796129 shutdown_connections 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.562+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1206796129 wait complete. 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8d0114b80 1 -- start start 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8d0114b80 1 Processor -- start 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79c09f0 0x55d3d79cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79cc1e0 0x55d3d79c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d3d79cebd0 con 0x55d3d79c1310 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d3d78b2cd0 con 0x55d3d79cc1e0 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d3d79ced50 con 0x55d3d79c09f0 2024-04-25T17:44:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8d0114b80 10 client.0 _ll_register_callbacks cb 0x55d3d78aca50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8b67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8b67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49918/0 (socket says 172.21.15.98:49918) 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8b67fc700 1 -- 172.21.15.98:0/1165405126 learned_addr learned my addr 172.21.15.98:0/1165405126 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.563+0000 7fd8beffd700 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79c09f0 0x55d3d79cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8b67fc700 1 -- 172.21.15.98:0/1165405126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79c09f0 msgr2=0x55d3d79cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79c09f0 0x55d3d79cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8b67fc700 1 -- 172.21.15.98:0/1165405126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79cc1e0 msgr2=0x55d3d79c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79cc1e0 0x55d3d79c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8b67fc700 1 -- 172.21.15.98:0/1165405126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3d78cb370 con 0x55d3d79c1310 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79c8bd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd8a809ca70 tx=0x7fd8a809d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb85a90c3a5e438 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd89effd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd89effd700 1 -- 172.21.15.98:0/1165405126 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a80a54c0 con 0x55d3d79c1310 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd89effd700 1 -- 172.21.15.98:0/1165405126 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fd8a809bf30 con 0x55d3d79c1310 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1165405126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b0004ce0 con 0x55d3d79c1310 2024-04-25T17:44:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd89effd700 1 -- 172.21.15.98:0/1165405126 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd8a80a5750 con 0x55d3d79c1310 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8d0114b80 10 client.16221 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1165405126 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d3d79d93b0 con 0x55d3d79c1310 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd89effd700 1 -- 172.21.15.98:0/1165405126 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b0004ce0 con 0x55d3d79c1310 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd89effd700 1 -- 172.21.15.98:0/1165405126 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fd8a809b9a0 con 0x55d3d79c1310 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8d0114b80 10 client.16221 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8d0114b80 10 client.16221 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8d0114b80 20 client.16221 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55d3d79db990 0x55d3d79dddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8d0114b80 4 client.16221 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.564+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1165405126 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3d7921620 con 0x55d3d79db990 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.565+0000 7fd8beffd700 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55d3d79db990 0x55d3d79dddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.565+0000 7fd8beffd700 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55d3d79db990 0x55d3d79dddb0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83a2abbdcae9efd8 server_cookie=89f58c0f07a88264 in_seq=0 out_seq=0 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.565+0000 7fd89effd700 10 client.16221 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd89effd700 1 -- 172.21.15.98:0/1165405126 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d3d7921620 con 0x55d3d79db990 2024-04-25T17:44:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd89effd700 10 client.16221 handle_command_reply: tid=0 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd8d0114b80 1 client.16221 shutdown 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1165405126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 msgr2=0x55d3d79c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79c8bd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd8a809ca70 tx=0x7fd8a809d320 comp rx=0 tx=0).stop 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1165405126 shutdown_connections 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55d3d79db990 0x55d3d79dddb0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79c09f0 0x55d3d79cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79cc1e0 0x55d3d79c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd89effd700 0 client.16221 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/1165405126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79c8bd0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.566+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1165405126 >> 172.21.15.98:0/1165405126 conn(0x55d3d78ee790 msgr2=0x55d3d79bcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.567+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1165405126 shutdown_connections 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.567+0000 7fd8d0114b80 1 -- 172.21.15.98:0/1165405126 wait complete. 2024-04-25T17:44:23.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.567+0000 7fd8d0114b80 20 client.16221 trim_cache size 0 max 16384 2024-04-25T17:44:23.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.585+0000 7fd8d0114b80 1 Processor -- start 2024-04-25T17:44:23.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8d0114b80 1 -- start start 2024-04-25T17:44:23.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 0x55d3d79d7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79c1310 0x55d3d79c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79cc1e0 0x55d3d79c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01b1f00 con 0x55d3d79cc1e0 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01af6d0 con 0x55d3d79c09f0 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01ad540 con 0x55d3d79c1310 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8bf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79cc1e0 0x55d3d79c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8bf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79cc1e0 0x55d3d79c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49920/0 (socket says 172.21.15.98:49920) 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8bf7fe700 1 -- 172.21.15.98:0/398017549 learned_addr learned my addr 172.21.15.98:0/398017549 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/398017549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79c1310 0x55d3d79c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8beffd700 1 --2- 172.21.15.98:0/398017549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 0x55d3d79d7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8bf7fe700 1 -- 172.21.15.98:0/398017549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79c1310 msgr2=0x55d3d79c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8bf7fe700 1 --2- 172.21.15.98:0/398017549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79c1310 0x55d3d79c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8bf7fe700 1 -- 172.21.15.98:0/398017549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 msgr2=0x55d3d79d7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8bf7fe700 1 --2- 172.21.15.98:0/398017549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 0x55d3d79d7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8bf7fe700 1 -- 172.21.15.98:0/398017549 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3d79c2170 con 0x55d3d79cc1e0 2024-04-25T17:44:23.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8bf7fe700 1 --2- 172.21.15.98:0/398017549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79cc1e0 0x55d3d79c90d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8015000 tx=0x7fd8b8009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed221896fe742606 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8b57fa700 1 -- 172.21.15.98:0/398017549 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8021070 con 0x55d3d79cc1e0 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8b57fa700 1 -- 172.21.15.98:0/398017549 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b8004a50 con 0x55d3d79cc1e0 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.586+0000 7fd8b57fa700 1 -- 172.21.15.98:0/398017549 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8007a80 con 0x55d3d79cc1e0 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.587+0000 7fd8d0114b80 1 -- 172.21.15.98:0/398017549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79cc1e0 msgr2=0x55d3d79c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.587+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/398017549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79cc1e0 0x55d3d79c90d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8015000 tx=0x7fd8b8009de0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.587+0000 7fd8d0114b80 1 -- 172.21.15.98:0/398017549 shutdown_connections 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.587+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/398017549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79cc1e0 0x55d3d79c90d0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.587+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/398017549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 0x55d3d79d7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.587+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/398017549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79c1310 0x55d3d79c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.587+0000 7fd8d0114b80 1 -- 172.21.15.98:0/398017549 >> 172.21.15.98:0/398017549 conn(0x55d3d78ee790 msgr2=0x55d3d79c8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.587+0000 7fd8d0114b80 1 -- 172.21.15.98:0/398017549 shutdown_connections 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.587+0000 7fd8d0114b80 1 -- 172.21.15.98:0/398017549 wait complete. 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.587+0000 7fd8d0114b80 1 -- start start 2024-04-25T17:44:23.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8d0114b80 1 Processor -- start 2024-04-25T17:44:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 0x55d3d79d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79d1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8d0114b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 0x55d3d7a41f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01adb90 con 0x55d3d79c1310 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8b67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79d1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8b67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79d1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49928/0 (socket says 172.21.15.98:49928) 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8d0114b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01af000 con 0x55d3d79c09f0 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8b67fc700 1 -- 172.21.15.98:0/3903675366 learned_addr learned my addr 172.21.15.98:0/3903675366 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8d0114b80 1 -- 172.21.15.98:0/3903675366 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c01ae260 con 0x55d3d79cc1e0 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8beffd700 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 0x55d3d79d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8d0114b80 10 client.0 _ll_register_callbacks cb 0x55d3d78aca50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8bf7fe700 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 0x55d3d7a41f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8beffd700 1 -- 172.21.15.98:0/3903675366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 msgr2=0x55d3d7a41f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.588+0000 7fd8beffd700 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 0x55d3d7a41f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd8beffd700 1 -- 172.21.15.98:0/3903675366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 msgr2=0x55d3d79d1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd8beffd700 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79d1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd8beffd700 1 -- 172.21.15.98:0/3903675366 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3d7a42540 con 0x55d3d79c09f0 2024-04-25T17:44:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79d1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd8bf7fe700 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 0x55d3d7a41f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd8beffd700 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 0x55d3d79d11e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x55d3d79de620 tx=0x7fd8b00050c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c4a9592d819d42a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd89effd700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd89effd700 1 -- 172.21.15.98:0/3903675366 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0022040 con 0x55d3d79c09f0 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd8d0114b80 1 -- 172.21.15.98:0/3903675366 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b8004a50 con 0x55d3d79c09f0 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd89effd700 1 -- 172.21.15.98:0/3903675366 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fd8b0011980 con 0x55d3d79c09f0 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd89effd700 1 -- 172.21.15.98:0/3903675366 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd8b0042050 con 0x55d3d79c09f0 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd8d0114b80 10 client.25715 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd8d0114b80 1 -- 172.21.15.98:0/3903675366 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d3d7a438c0 con 0x55d3d79c09f0 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.589+0000 7fd89effd700 1 -- 172.21.15.98:0/3903675366 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b8004a50 con 0x55d3d79c09f0 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.590+0000 7fd89effd700 1 -- 172.21.15.98:0/3903675366 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fd8b00051d0 con 0x55d3d79c09f0 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.590+0000 7fd8d0114b80 10 client.25715 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.590+0000 7fd8d0114b80 10 client.25715 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:44:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.590+0000 7fd8d0114b80 20 client.25715 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.590+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55d3d7a43cf0 0x55d3d7a46110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.590+0000 7fd8d0114b80 4 client.25715 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b20338f", "release": true, "await": true}] 2024-04-25T17:44:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.590+0000 7fd8d0114b80 1 -- 172.21.15.98:0/3903675366 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b20338f", "release": true, "await": true}) -- 0x55d3d791b6c0 con 0x55d3d7a43cf0 2024-04-25T17:44:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.590+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55d3d7a43cf0 0x55d3d7a46110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.590+0000 7fd89effd700 10 client.25715 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.590+0000 7fd8b67fc700 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55d3d7a43cf0 0x55d3d7a46110 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce10d97d302d38fe server_cookie=64b7ca14298a5556 in_seq=0 out_seq=0 2024-04-25T17:44:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd89effd700 1 -- 172.21.15.98:0/3903675366 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55d3d791b6c0 con 0x55d3d7a43cf0 2024-04-25T17:44:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd89effd700 10 client.25715 handle_command_reply: tid=0 2024-04-25T17:44:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 client.25715 shutdown 2024-04-25T17:44:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 -- 172.21.15.98:0/3903675366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 msgr2=0x55d3d79d11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 0x55d3d79d11e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x55d3d79de620 tx=0x7fd8b00050c0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 -- 172.21.15.98:0/3903675366 shutdown_connections 2024-04-25T17:44:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55d3d7a43cf0 0x55d3d7a46110 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d3d79cc1e0 0x55d3d7a41f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d3d79c09f0 0x55d3d79d11e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 --2- 172.21.15.98:0/3903675366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d3d79c1310 0x55d3d79d1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd89effd700 0 client.25715 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 -- 172.21.15.98:0/3903675366 >> 172.21.15.98:0/3903675366 conn(0x55d3d78ee790 msgr2=0x55d3d79c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 -- 172.21.15.98:0/3903675366 shutdown_connections 2024-04-25T17:44:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 1 -- 172.21.15.98:0/3903675366 wait complete. 2024-04-25T17:44:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.593+0000 7fd8d0114b80 20 client.25715 trim_cache size 0 max 16384 2024-04-25T17:44:23.597 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:44:23.597 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 38, 2024-04-25T17:44:23.597 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 24, 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: "b20338f": { 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 24, 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:44:23.598 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 85.3, 2024-04-25T17:44:23.599 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:44:23.599 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:44:23.599 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:44:23.599 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:44:23.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:44:23.599 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:44:23.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:44:23.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:44:23.600 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:44:23.600 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:44:23.600 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:44:23.600 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:44:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 -- 172.21.15.98:0/934852998 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd8ac0a3db0 msgr2=0x7fd8ac0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd8ac0a3db0 0x7fd8ac0a6270 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009bc0 tx=0x7fd8b0009260 comp rx=0 tx=0).stop 2024-04-25T17:44:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 -- 172.21.15.98:0/934852998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c006b0f0 msgr2=0x7fd8c0101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c006b0f0 0x7fd8c0101f50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800e9a0 tx=0x7fd8a800ed60 comp rx=0 tx=0).stop 2024-04-25T17:44:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 -- 172.21.15.98:0/934852998 shutdown_connections 2024-04-25T17:44:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd8ac0a3db0 0x7fd8ac0a6270 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8c0108a30 0x7fd8c0102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8c006a750 0x7fd8c0101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 --2- 172.21.15.98:0/934852998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8c006b0f0 0x7fd8c0101f50 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 -- 172.21.15.98:0/934852998 >> 172.21.15.98:0/934852998 conn(0x7fd8c00788f0 msgr2=0x7fd8c00ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 -- 172.21.15.98:0/934852998 shutdown_connections 2024-04-25T17:44:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:23.594+0000 7fd8c588b700 1 -- 172.21.15.98:0/934852998 wait complete. 2024-04-25T17:44:24.391 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:24 smithi098 ceph-mon[29192]: pgmap v264: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 KiB/s wr, 9 op/s 2024-04-25T17:44:24.391 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b20338f', seconds elapsed: 37.6 2024-04-25T17:44:24.392 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.3 2024-04-25T17:44:24.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:44:24.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:24 smithi203 ceph-mon[32834]: pgmap v264: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 KiB/s wr, 9 op/s 2024-04-25T17:44:24.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:24 smithi173 ceph-mon[32856]: pgmap v264: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 KiB/s wr, 9 op/s 2024-04-25T17:44:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.733+0000 7f8dad59a640 1 -- 172.21.15.98:0/2688377078 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da40336a0 con 0x7f8db0101aa0 2024-04-25T17:44:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.734+0000 7f8db4ca7640 1 -- 172.21.15.98:0/2688377078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8db0101aa0 msgr2=0x7f8db0105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.734+0000 7f8db4ca7640 1 --2- 172.21.15.98:0/2688377078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8db0101aa0 0x7f8db0105a90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009900 tx=0x7f8da402c630 comp rx=0 tx=0).stop 2024-04-25T17:44:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.734+0000 7f8db4ca7640 1 -- 172.21.15.98:0/2688377078 shutdown_connections 2024-04-25T17:44:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.734+0000 7f8db4ca7640 1 --2- 172.21.15.98:0/2688377078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8db0106170 0x7f8db010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.734+0000 7f8db4ca7640 1 --2- 172.21.15.98:0/2688377078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8db0101aa0 0x7f8db0105a90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.734+0000 7f8db4ca7640 1 --2- 172.21.15.98:0/2688377078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db01010f0 0x7f8db01014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.734+0000 7f8db4ca7640 1 -- 172.21.15.98:0/2688377078 >> 172.21.15.98:0/2688377078 conn(0x7f8db0077fd0 msgr2=0x7f8db00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.734+0000 7f8db4ca7640 1 -- 172.21.15.98:0/2688377078 shutdown_connections 2024-04-25T17:44:24.735 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.735+0000 7f8db4ca7640 1 -- 172.21.15.98:0/2688377078 wait complete. 2024-04-25T17:44:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.735+0000 7f8db4ca7640 1 Processor -- start 2024-04-25T17:44:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8db4ca7640 1 -- start start 2024-04-25T17:44:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8db4ca7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8db01010f0 0x7f8db01965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8db4ca7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0106170 0x7f8db0196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8db4ca7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8db019ae60 0x7f8db019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8db4ca7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8db0110070 con 0x7f8db0106170 2024-04-25T17:44:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8db4ca7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8db010fef0 con 0x7f8db019ae60 2024-04-25T17:44:24.736 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8db4ca7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8db01101f0 con 0x7f8db01010f0 2024-04-25T17:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0106170 0x7f8db0196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0106170 0x7f8db0196b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49932/0 (socket says 172.21.15.98:49932) 2024-04-25T17:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8dadd9b640 1 -- 172.21.15.98:0/1867993250 learned_addr learned my addr 172.21.15.98:0/1867993250 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8daed9d640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8db019ae60 0x7f8db019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.737 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.736+0000 7f8dae59c640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8db01010f0 0x7f8db01965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8daed9d640 1 -- 172.21.15.98:0/1867993250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8db01010f0 msgr2=0x7f8db01965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8daed9d640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8db01010f0 0x7f8db01965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8daed9d640 1 -- 172.21.15.98:0/1867993250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0106170 msgr2=0x7f8db0196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8daed9d640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0106170 0x7f8db0196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8daed9d640 1 -- 172.21.15.98:0/1867993250 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db019b950 con 0x7f8db019ae60 2024-04-25T17:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8daed9d640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8db019ae60 0x7f8db019b310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8da800da10 tx=0x7f8da800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8d9f7fe640 1 -- 172.21.15.98:0/1867993250 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da800bd70 con 0x7f8db019ae60 2024-04-25T17:44:24.738 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8d9f7fe640 1 -- 172.21.15.98:0/1867993250 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da8003e20 con 0x7f8db019ae60 2024-04-25T17:44:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8d9f7fe640 1 -- 172.21.15.98:0/1867993250 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da80106c0 con 0x7f8db019ae60 2024-04-25T17:44:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8db4ca7640 1 -- 172.21.15.98:0/1867993250 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db019bc40 con 0x7f8db019ae60 2024-04-25T17:44:24.739 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.737+0000 7f8db4ca7640 1 -- 172.21.15.98:0/1867993250 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db01a33e0 con 0x7f8db019ae60 2024-04-25T17:44:24.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.741+0000 7f8db79be740 1 Processor -- start 2024-04-25T17:44:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.742+0000 7f8d9f7fe640 1 -- 172.21.15.98:0/1867993250 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8da8004250 con 0x7f8db019ae60 2024-04-25T17:44:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.742+0000 7f8db79be740 1 -- start start 2024-04-25T17:44:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.742+0000 7f8db79be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac2e14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.742+0000 7f8db79be740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac2e1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.742+0000 7f8db79be740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 0x55daac2f7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.742+0000 7f8db79be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55daac246970 con 0x55daac2e2bd0 2024-04-25T17:44:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.742+0000 7f8db79be740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55daac1e34a0 con 0x55daac2e1a00 2024-04-25T17:44:24.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.742+0000 7f8db79be740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55daac242930 con 0x55daac2f4ea0 2024-04-25T17:44:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac2e14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dae59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac2e1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dae59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac2e1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:42064/0 (socket says 172.21.15.98:42064) 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac2e14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49942/0 (socket says 172.21.15.98:49942) 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dae59c640 1 -- 172.21.15.98:0/2973187675 learned_addr learned my addr 172.21.15.98:0/2973187675 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/2973187675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 0x55daac2f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8d9f7fe640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8d980a3e90 0x7f8d980a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dae59c640 1 -- 172.21.15.98:0/2973187675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 msgr2=0x55daac2f7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dae59c640 1 --2- 172.21.15.98:0/2973187675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 0x55daac2f7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dae59c640 1 -- 172.21.15.98:0/2973187675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 msgr2=0x55daac2e14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dae59c640 1 --2- 172.21.15.98:0/2973187675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac2e14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dae59c640 1 -- 172.21.15.98:0/2973187675 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daabf59210 con 0x55daac2e1a00 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8d9f7fe640 1 -- 172.21.15.98:0/1867993250 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f8da80d0bf0 con 0x7f8db019ae60 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8dae59c640 1 --2- 172.21.15.98:0/2973187675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac2e1e80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009a40 tx=0x7f8da0009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e178326bae9808 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8daed9d640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8d980a3e90 0x7f8d980a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8d9cff9640 1 -- 172.21.15.98:0/2973187675 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0017070 con 0x55daac2e1a00 2024-04-25T17:44:24.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8d9cff9640 1 -- 172.21.15.98:0/2973187675 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da0013470 con 0x55daac2e1a00 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.743+0000 7f8d9cff9640 1 -- 172.21.15.98:0/2973187675 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0012610 con 0x55daac2e1a00 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.744+0000 7f8db79be740 1 -- 172.21.15.98:0/2973187675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 msgr2=0x55daac2e1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.744+0000 7f8db79be740 1 --2- 172.21.15.98:0/2973187675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac2e1e80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009a40 tx=0x7f8da0009f00 comp rx=0 tx=0).stop 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.744+0000 7f8db79be740 1 -- 172.21.15.98:0/2973187675 shutdown_connections 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.745+0000 7f8db79be740 1 --2- 172.21.15.98:0/2973187675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 0x55daac2f7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.745+0000 7f8daed9d640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8d980a3e90 0x7f8d980a6350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8da809cac0 tx=0x7f8da8004540 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.745+0000 7f8db79be740 1 --2- 172.21.15.98:0/2973187675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac2e1e80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.745+0000 7f8db79be740 1 --2- 172.21.15.98:0/2973187675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac2e14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.745+0000 7f8db79be740 1 -- 172.21.15.98:0/2973187675 >> 172.21.15.98:0/2973187675 conn(0x55daac2ddb30 msgr2=0x55daac2ddf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.745+0000 7f8db79be740 1 -- 172.21.15.98:0/2973187675 shutdown_connections 2024-04-25T17:44:24.746 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.745+0000 7f8db79be740 1 -- 172.21.15.98:0/2973187675 wait complete. 2024-04-25T17:44:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.745+0000 7f8db79be740 1 -- start start 2024-04-25T17:44:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.745+0000 7f8db79be740 1 Processor -- start 2024-04-25T17:44:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8db79be740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e1a00 0x55daac2ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8db79be740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e2bd0 0x55daac2eece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8db79be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2f4ea0 0x55daac2f4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8db79be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55daac2e8a70 con 0x55daac2f4ea0 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8db79be740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55daac1e10b0 con 0x55daac2e2bd0 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8db79be740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55daac2e8bf0 con 0x55daac2e1a00 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8db79be740 10 client.0 _ll_register_callbacks cb 0x55daac1ced30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e1a00 0x55daac2ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e1a00 0x55daac2ee7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:34306/0 (socket says 172.21.15.98:34306) 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dadd9b640 1 -- 172.21.15.98:0/3443417216 learned_addr learned my addr 172.21.15.98:0/3443417216 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dae59c640 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e2bd0 0x55daac2eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2f4ea0 0x55daac2f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dadd9b640 1 -- 172.21.15.98:0/3443417216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e2bd0 msgr2=0x55daac2eece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e2bd0 0x55daac2eece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dadd9b640 1 -- 172.21.15.98:0/3443417216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2f4ea0 msgr2=0x55daac2f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2f4ea0 0x55daac2f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dadd9b640 1 -- 172.21.15.98:0/3443417216 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daac025620 con 0x55daac2e1a00 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.746+0000 7f8dae59c640 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e2bd0 0x55daac2eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.747+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e1a00 0x55daac2ee7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8da402cc20 tx=0x7f8da402c630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efc7c68d0a5d53d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.747+0000 7f8d95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.747+0000 7f8d95ffb640 1 -- 172.21.15.98:0/3443417216 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da4037040 con 0x55daac2e1a00 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.747+0000 7f8d95ffb640 1 -- 172.21.15.98:0/3443417216 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f8da4004f30 con 0x55daac2e1a00 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.747+0000 7f8db79be740 1 -- 172.21.15.98:0/3443417216 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8da0013470 con 0x55daac2e1a00 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.747+0000 7f8d95ffb640 1 -- 172.21.15.98:0/3443417216 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8da4005150 con 0x55daac2e1a00 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.747+0000 7f8db79be740 10 client.25732 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:24.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.747+0000 7f8db79be740 1 -- 172.21.15.98:0/3443417216 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55daac30c3a0 con 0x55daac2e1a00 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.747+0000 7f8d95ffb640 1 -- 172.21.15.98:0/3443417216 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da40035f0 con 0x55daac2e1a00 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.747+0000 7f8d95ffb640 1 -- 172.21.15.98:0/3443417216 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f8da403e440 con 0x55daac2e1a00 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.748+0000 7f8db79be740 10 client.25732 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.748+0000 7f8db79be740 10 client.25732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.748+0000 7f8db79be740 20 client.25732 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.748+0000 7f8db79be740 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55daac30d7b0 0x55daac30fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.748+0000 7f8db79be740 4 client.25732 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.748+0000 7f8db79be740 1 -- 172.21.15.98:0/3443417216 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8da0013470 con 0x55daac30d7b0 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.748+0000 7f8dae59c640 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55daac30d7b0 0x55daac30fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.748+0000 7f8dae59c640 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55daac30d7b0 0x55daac30fba0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c86262dd450b08ea server_cookie=4dc14965e26fbfd7 in_seq=0 out_seq=0 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.749+0000 7f8d95ffb640 10 client.25732 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.750+0000 7f8d95ffb640 1 -- 172.21.15.98:0/3443417216 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f8da0013470 con 0x55daac30d7b0 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.750+0000 7f8d95ffb640 10 client.25732 handle_command_reply: tid=0 2024-04-25T17:44:24.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.750+0000 7f8db79be740 1 client.25732 shutdown 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.750+0000 7f8db79be740 1 -- 172.21.15.98:0/3443417216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e1a00 msgr2=0x55daac2ee7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.750+0000 7f8db79be740 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e1a00 0x55daac2ee7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8da402cc20 tx=0x7f8da402c630 comp rx=0 tx=0).stop 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.751+0000 7f8db79be740 1 -- 172.21.15.98:0/3443417216 shutdown_connections 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.751+0000 7f8db79be740 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55daac30d7b0 0x55daac30fba0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.751+0000 7f8db79be740 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2f4ea0 0x55daac2f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.751+0000 7f8db79be740 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e2bd0 0x55daac2eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.751+0000 7f8db79be740 1 --2- 172.21.15.98:0/3443417216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e1a00 0x55daac2ee7a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.751+0000 7f8db79be740 1 -- 172.21.15.98:0/3443417216 >> 172.21.15.98:0/3443417216 conn(0x55daac2ddb30 msgr2=0x55daac2e39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.751+0000 7f8d95ffb640 0 client.25732 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.751+0000 7f8db79be740 1 -- 172.21.15.98:0/3443417216 shutdown_connections 2024-04-25T17:44:24.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.751+0000 7f8db79be740 1 -- 172.21.15.98:0/3443417216 wait complete. 2024-04-25T17:44:24.755 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.751+0000 7f8db79be740 20 client.25732 trim_cache size 0 max 16384 2024-04-25T17:44:24.757 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:44:24.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.773+0000 7f8db79be740 1 Processor -- start 2024-04-25T17:44:24.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8db79be740 1 -- start start 2024-04-25T17:44:24.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8db79be740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac2f4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8db79be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac315460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8db79be740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 0x55daac3159a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8db79be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8db01ae3e0 con 0x55daac2e2bd0 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8db79be740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8db01abbf0 con 0x55daac2e1a00 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8db79be740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8db01a9a60 con 0x55daac2f4ea0 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8dae59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac315460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8dae59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac315460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49958/0 (socket says 172.21.15.98:49958) 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8dae59c640 1 -- 172.21.15.98:0/3609465515 learned_addr learned my addr 172.21.15.98:0/3609465515 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/3609465515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac2f4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.774+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/3609465515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 0x55daac3159a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.775+0000 7f8dae59c640 1 -- 172.21.15.98:0/3609465515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 msgr2=0x55daac3159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.775+0000 7f8dae59c640 1 --2- 172.21.15.98:0/3609465515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 0x55daac3159a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.775+0000 7f8dae59c640 1 -- 172.21.15.98:0/3609465515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 msgr2=0x55daac2f4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.775+0000 7f8dae59c640 1 --2- 172.21.15.98:0/3609465515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac2f4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.775+0000 7f8dae59c640 1 -- 172.21.15.98:0/3609465515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daac1ceaf0 con 0x55daac2e2bd0 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.775+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/3609465515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 0x55daac3159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:44:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.775+0000 7f8dae59c640 1 --2- 172.21.15.98:0/3609465515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac315460 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8da000a010 tx=0x7f8da0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4ab84ee7ed7b6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.775+0000 7f8d97fff640 1 -- 172.21.15.98:0/3609465515 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da002c070 con 0x55daac2e2bd0 2024-04-25T17:44:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.775+0000 7f8d97fff640 1 -- 172.21.15.98:0/3609465515 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da0004040 con 0x55daac2e2bd0 2024-04-25T17:44:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.775+0000 7f8d97fff640 1 -- 172.21.15.98:0/3609465515 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0017040 con 0x55daac2e2bd0 2024-04-25T17:44:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.776+0000 7f8db79be740 1 -- 172.21.15.98:0/3609465515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 msgr2=0x55daac315460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.776+0000 7f8db79be740 1 --2- 172.21.15.98:0/3609465515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac315460 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8da000a010 tx=0x7f8da0008040 comp rx=0 tx=0).stop 2024-04-25T17:44:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.776+0000 7f8db79be740 1 -- 172.21.15.98:0/3609465515 shutdown_connections 2024-04-25T17:44:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.776+0000 7f8db79be740 1 --2- 172.21.15.98:0/3609465515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2f4ea0 0x55daac3159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.776+0000 7f8db79be740 1 --2- 172.21.15.98:0/3609465515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2e2bd0 0x55daac315460 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.776+0000 7f8db79be740 1 --2- 172.21.15.98:0/3609465515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac2f4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.776+0000 7f8db79be740 1 -- 172.21.15.98:0/3609465515 >> 172.21.15.98:0/3609465515 conn(0x55daac2ddb30 msgr2=0x55daac2f5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.776+0000 7f8db79be740 1 -- 172.21.15.98:0/3609465515 shutdown_connections 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.776+0000 7f8db79be740 1 -- 172.21.15.98:0/3609465515 wait complete. 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8db79be740 1 -- start start 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8db79be740 1 Processor -- start 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8db79be740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac320d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8db79be740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e2bd0 0x55daac2e6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8db79be740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2f4ea0 0x55daac2e6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8db79be740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8db01aa0b0 con 0x55daac2f4ea0 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8db79be740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8db01ab520 con 0x55daac2e1a00 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8db79be740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8db01aa780 con 0x55daac2e2bd0 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8db79be740 10 client.0 _ll_register_callbacks cb 0x7f8da4004890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac320d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2f4ea0 0x55daac2e6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac320d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:42096/0 (socket says 172.21.15.98:42096) 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8dadd9b640 1 -- 172.21.15.98:0/314865282 learned_addr learned my addr 172.21.15.98:0/314865282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.777+0000 7f8dae59c640 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e2bd0 0x55daac2e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8dadd9b640 1 -- 172.21.15.98:0/314865282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e2bd0 msgr2=0x55daac2e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e2bd0 0x55daac2e6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8dadd9b640 1 -- 172.21.15.98:0/314865282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2f4ea0 msgr2=0x55daac2e6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2f4ea0 0x55daac2e6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8dadd9b640 1 -- 172.21.15.98:0/314865282 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daac2e6fb0 con 0x55daac2e1a00 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8dae59c640 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e2bd0 0x55daac2e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8dadd9b640 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac320d80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8da4004240 tx=0x7f8da4009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae2e7e8a2433f1bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8d95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8d95ffb640 1 -- 172.21.15.98:0/314865282 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da4037040 con 0x55daac2e1a00 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8db79be740 1 -- 172.21.15.98:0/314865282 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8da0004040 con 0x55daac2e1a00 2024-04-25T17:44:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8d95ffb640 1 -- 172.21.15.98:0/314865282 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f8da40054f0 con 0x55daac2e1a00 2024-04-25T17:44:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.778+0000 7f8d95ffb640 1 -- 172.21.15.98:0/314865282 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8da403ee60 con 0x55daac2e1a00 2024-04-25T17:44:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.779+0000 7f8db79be740 10 client.25733 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.779+0000 7f8db79be740 1 -- 172.21.15.98:0/314865282 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55daac3737b0 con 0x55daac2e1a00 2024-04-25T17:44:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.779+0000 7f8d95ffb640 1 -- 172.21.15.98:0/314865282 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da0004040 con 0x55daac2e1a00 2024-04-25T17:44:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.779+0000 7f8d95ffb640 1 -- 172.21.15.98:0/314865282 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f8da404f450 con 0x55daac2e1a00 2024-04-25T17:44:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.779+0000 7f8db79be740 10 client.25733 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.779+0000 7f8db79be740 10 client.25733 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:44:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.779+0000 7f8db79be740 20 client.25733 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.779+0000 7f8db79be740 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55daac373e00 0x55daac376220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.779+0000 7f8db79be740 4 client.25733 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:44:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.779+0000 7f8db79be740 1 -- 172.21.15.98:0/314865282 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55daac254390 con 0x55daac373e00 2024-04-25T17:44:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.780+0000 7f8dae59c640 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55daac373e00 0x55daac376220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.780+0000 7f8d95ffb640 10 client.25733 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.780+0000 7f8dae59c640 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55daac373e00 0x55daac376220 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f365efcf85324c48 server_cookie=ba41a3876daa67e9 in_seq=0 out_seq=0 2024-04-25T17:44:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.788+0000 7f8d95ffb640 1 -- 172.21.15.98:0/314865282 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55daac254390 con 0x55daac373e00 2024-04-25T17:44:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.788+0000 7f8d95ffb640 10 client.25733 handle_command_reply: tid=0 2024-04-25T17:44:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.788+0000 7f8db79be740 1 client.25733 shutdown 2024-04-25T17:44:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8db79be740 1 -- 172.21.15.98:0/314865282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 msgr2=0x55daac320d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8db79be740 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac320d80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8da4004240 tx=0x7f8da4009680 comp rx=0 tx=0).stop 2024-04-25T17:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8db79be740 1 -- 172.21.15.98:0/314865282 shutdown_connections 2024-04-25T17:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8db79be740 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55daac373e00 0x55daac376220 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8d95ffb640 0 client.25733 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8db79be740 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55daac2f4ea0 0x55daac2e6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8db79be740 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55daac2e2bd0 0x55daac2e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8db79be740 1 --2- 172.21.15.98:0/314865282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55daac2e1a00 0x55daac320d80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8db79be740 1 -- 172.21.15.98:0/314865282 >> 172.21.15.98:0/314865282 conn(0x55daac2ddb30 msgr2=0x55daac2f5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8db79be740 1 -- 172.21.15.98:0/314865282 shutdown_connections 2024-04-25T17:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.789+0000 7f8db79be740 1 -- 172.21.15.98:0/314865282 wait complete. 2024-04-25T17:44:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.790+0000 7f8db79be740 20 client.25733 trim_cache size 0 max 16384 2024-04-25T17:44:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:44:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:44:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:44:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:44:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 -- 172.21.15.98:0/1867993250 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8d980a3e90 msgr2=0x7f8d980a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8d980a3e90 0x7f8d980a6350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8da809cac0 tx=0x7f8da8004540 comp rx=0 tx=0).stop 2024-04-25T17:44:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 -- 172.21.15.98:0/1867993250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8db019ae60 msgr2=0x7f8db019b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8db019ae60 0x7f8db019b310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8da800da10 tx=0x7f8da800ded0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 -- 172.21.15.98:0/1867993250 shutdown_connections 2024-04-25T17:44:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8d980a3e90 0x7f8d980a6350 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8db019ae60 0x7f8db019b310 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8db0106170 0x7f8db0196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 --2- 172.21.15.98:0/1867993250 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8db01010f0 0x7f8db01965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 -- 172.21.15.98:0/1867993250 >> 172.21.15.98:0/1867993250 conn(0x7f8db0077fd0 msgr2=0x7f8db00fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 -- 172.21.15.98:0/1867993250 shutdown_connections 2024-04-25T17:44:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:24.791+0000 7f8db4ca7640 1 -- 172.21.15.98:0/1867993250 wait complete. 2024-04-25T17:44:26.238 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:26 smithi098 ceph-mon[29192]: pgmap v265: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 KiB/s wr, 9 op/s 2024-04-25T17:44:26.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:26 smithi203 ceph-mon[32834]: pgmap v265: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 KiB/s wr, 9 op/s 2024-04-25T17:44:26.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:26 smithi173 ceph-mon[32856]: pgmap v265: 97 pgs: 97 active+clean; 3.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 KiB/s wr, 9 op/s 2024-04-25T17:44:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 475.0 s, 1473.1 tps, lat 462.211 ms stddev 3613.457 2024-04-25T17:44:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.046+0000 7f8c637fe700 1 -- 172.21.15.98:0/437643922 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c012650 con 0x7f8c641054b0 2024-04-25T17:44:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.047+0000 7f8c6a81d700 1 -- 172.21.15.98:0/437643922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c641054b0 msgr2=0x7f8c64105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.047+0000 7f8c6a81d700 1 --2- 172.21.15.98:0/437643922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c641054b0 0x7f8c64105940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00b0a0 tx=0x7f8c5c00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:44:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.047+0000 7f8c6a81d700 1 -- 172.21.15.98:0/437643922 shutdown_connections 2024-04-25T17:44:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.047+0000 7f8c6a81d700 1 --2- 172.21.15.98:0/437643922 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c641054b0 0x7f8c64105940 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.047+0000 7f8c6a81d700 1 --2- 172.21.15.98:0/437643922 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c641079a0 0x7f8c6406b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.047+0000 7f8c6a81d700 1 --2- 172.21.15.98:0/437643922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6406b680 0x7f8c6406bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.047+0000 7f8c6a81d700 1 -- 172.21.15.98:0/437643922 >> 172.21.15.98:0/437643922 conn(0x7f8c64078980 msgr2=0x7f8c64078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:27.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.048+0000 7f8c6a81d700 1 -- 172.21.15.98:0/437643922 shutdown_connections 2024-04-25T17:44:27.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.050+0000 7f8c6a81d700 1 -- 172.21.15.98:0/437643922 wait complete. 2024-04-25T17:44:27.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.051+0000 7f8c6a81d700 1 Processor -- start 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.051+0000 7f8c6a81d700 1 -- start start 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.051+0000 7f8c6a81d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6406b680 0x7f8c6410fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.051+0000 7f8c6a81d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c641054b0 0x7f8c64110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.051+0000 7f8c6a81d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c641079a0 0x7f8c64114380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.051+0000 7f8c6a81d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8c64108440 con 0x7f8c641054b0 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.051+0000 7f8c6a81d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641082c0 con 0x7f8c641079a0 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.051+0000 7f8c6a81d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641085c0 con 0x7f8c6406b680 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.052+0000 7f8c63fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6406b680 0x7f8c6410fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.052+0000 7f8c63fff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6406b680 0x7f8c6410fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:34344/0 (socket says 172.21.15.98:34344) 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.052+0000 7f8c63fff700 1 -- 172.21.15.98:0/2385926235 learned_addr learned my addr 172.21.15.98:0/2385926235 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.052+0000 7f8c63fff700 1 -- 172.21.15.98:0/2385926235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c641079a0 msgr2=0x7f8c64114380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.052+0000 7f8c63fff700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c641079a0 0x7f8c64114380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.052+0000 7f8c63fff700 1 -- 172.21.15.98:0/2385926235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c641054b0 msgr2=0x7f8c64110040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T17:44:27.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.052+0000 7f8c63fff700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c641054b0 0x7f8c64110040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:27.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.052+0000 7f8c63fff700 1 -- 172.21.15.98:0/2385926235 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c64114970 con 0x7f8c6406b680 2024-04-25T17:44:27.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.057+0000 7f8c63fff700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6406b680 0x7f8c6410fb00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400c9b0 tx=0x7f8c5400cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:27.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.057+0000 7f8c61ffb700 1 -- 172.21.15.98:0/2385926235 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54007b00 con 0x7f8c6406b680 2024-04-25T17:44:27.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.057+0000 7f8c6a81d700 1 -- 172.21.15.98:0/2385926235 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c64114c60 con 0x7f8c6406b680 2024-04-25T17:44:27.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.057+0000 7f8c6a81d700 1 -- 172.21.15.98:0/2385926235 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c641b4ba0 con 0x7f8c6406b680 2024-04-25T17:44:27.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.057+0000 7f8c61ffb700 1 -- 172.21.15.98:0/2385926235 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c54007ca0 con 0x7f8c6406b680 2024-04-25T17:44:27.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.057+0000 7f8c61ffb700 1 -- 172.21.15.98:0/2385926235 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54018730 con 0x7f8c6406b680 2024-04-25T17:44:27.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.059+0000 7f8c6a81d700 1 -- 172.21.15.98:0/2385926235 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c44005330 con 0x7f8c6406b680 2024-04-25T17:44:27.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.062+0000 7f8c61ffb700 1 -- 172.21.15.98:0/2385926235 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8c5400f990 con 0x7f8c6406b680 2024-04-25T17:44:27.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.063+0000 7f8c61ffb700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8c500a3e30 0x7f8c500a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:27.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.064+0000 7f8c61ffb700 1 -- 172.21.15.98:0/2385926235 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f8c5400de00 con 0x7f8c6406b680 2024-04-25T17:44:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.065+0000 7f8c5bfff700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8c500a3e30 0x7f8c500a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:27.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.066+0000 7f8c5bfff700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8c500a3e30 0x7f8c500a62f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8c641110a0 tx=0x7f8c4c005c10 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:27.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.071+0000 7f8c61ffb700 1 -- 172.21.15.98:0/2385926235 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c54014030 con 0x7f8c6406b680 2024-04-25T17:44:27.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.422+0000 7f8c6a81d700 1 -- 172.21.15.98:0/2385926235 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c440058f0 con 0x7f8c6406b680 2024-04-25T17:44:27.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.423+0000 7f8c61ffb700 1 -- 172.21.15.98:0/2385926235 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7f8c54014030 con 0x7f8c6406b680 2024-04-25T17:44:27.423 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:44:27.423 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:42:52.734463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:44:27.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 -- 172.21.15.98:0/2385926235 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8c500a3e30 msgr2=0x7f8c500a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:27.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8c500a3e30 0x7f8c500a62f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8c641110a0 tx=0x7f8c4c005c10 comp rx=0 tx=0).stop 2024-04-25T17:44:27.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 -- 172.21.15.98:0/2385926235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6406b680 msgr2=0x7f8c6410fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:27.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6406b680 0x7f8c6410fb00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400c9b0 tx=0x7f8c5400cd70 comp rx=0 tx=0).stop 2024-04-25T17:44:27.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 -- 172.21.15.98:0/2385926235 shutdown_connections 2024-04-25T17:44:27.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8c500a3e30 0x7f8c500a62f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:27.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6406b680 0x7f8c6410fb00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:27.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c641079a0 0x7f8c64114380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:27.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 --2- 172.21.15.98:0/2385926235 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8c641054b0 0x7f8c64110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:27.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 -- 172.21.15.98:0/2385926235 >> 172.21.15.98:0/2385926235 conn(0x7f8c64078980 msgr2=0x7f8c64100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:27.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 -- 172.21.15.98:0/2385926235 shutdown_connections 2024-04-25T17:44:27.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:27.427+0000 7f8c6a81d700 1 -- 172.21.15.98:0/2385926235 wait complete. 2024-04-25T17:44:27.428 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 38 2024-04-25T17:44:28.142 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 85.3 --await-for 42.0' 2024-04-25T17:44:28.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 85.3 --await-for 42.0 2024-04-25T17:44:28.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:28 smithi098 ceph-mon[29192]: pgmap v266: 97 pgs: 97 active+clean; 3.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-04-25T17:44:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:28 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2385926235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:44:28.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:28 smithi203 ceph-mon[32834]: pgmap v266: 97 pgs: 97 active+clean; 3.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-04-25T17:44:28.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:28 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2385926235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:44:28.471 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:44:28.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:28 smithi173 ceph-mon[32856]: pgmap v266: 97 pgs: 97 active+clean; 3.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.2 MiB/s wr, 133 op/s 2024-04-25T17:44:28.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:28 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2385926235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:44:30.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:30 smithi098 ceph-mon[29192]: pgmap v267: 97 pgs: 97 active+clean; 3.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-04-25T17:44:30.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:30 smithi203 ceph-mon[32834]: pgmap v267: 97 pgs: 97 active+clean; 3.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-04-25T17:44:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.492+0000 7f2814ad6700 1 -- 172.21.15.98:0/1447295315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281006b990 msgr2=0x7f281010f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.492+0000 7f2814ad6700 1 --2- 172.21.15.98:0/1447295315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281006b990 0x7f281010f920 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f280800b0a0 tx=0x7f280800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.492+0000 7f2814ad6700 1 -- 172.21.15.98:0/1447295315 shutdown_connections 2024-04-25T17:44:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.492+0000 7f2814ad6700 1 --2- 172.21.15.98:0/1447295315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2810108a50 0x7f2810108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.492+0000 7f2814ad6700 1 --2- 172.21.15.98:0/1447295315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281006b990 0x7f281010f920 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.492+0000 7f2814ad6700 1 --2- 172.21.15.98:0/1447295315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f281006aff0 0x7f281006b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.492+0000 7f2814ad6700 1 -- 172.21.15.98:0/1447295315 >> 172.21.15.98:0/1447295315 conn(0x7f2810078980 msgr2=0x7f2810078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.492+0000 7f2814ad6700 1 -- 172.21.15.98:0/1447295315 shutdown_connections 2024-04-25T17:44:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.492+0000 7f2814ad6700 1 -- 172.21.15.98:0/1447295315 wait complete. 2024-04-25T17:44:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.493+0000 7f2814ad6700 1 Processor -- start 2024-04-25T17:44:30.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.493+0000 7f2814ad6700 1 -- start start 2024-04-25T17:44:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.494+0000 7f2814ad6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f281006aff0 0x7f2810198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.494+0000 7f2814ad6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281006b990 0x7f28101993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.494+0000 7f2814ad6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2810108a50 0x7f281019d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.494+0000 7f2814ad6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2810112c50 con 0x7f281006b990 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.494+0000 7f280e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f281006aff0 0x7f2810198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.494+0000 7f280e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f281006aff0 0x7f2810198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:34356/0 (socket says 172.21.15.98:34356) 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.494+0000 7f280e59c700 1 -- 172.21.15.98:0/1149163898 learned_addr learned my addr 172.21.15.98:0/1149163898 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.494+0000 7f2814ad6700 1 -- 172.21.15.98:0/1149163898 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2810112ad0 con 0x7f2810108a50 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.494+0000 7f2814ad6700 1 -- 172.21.15.98:0/1149163898 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2810112dd0 con 0x7f281006aff0 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.494+0000 7f280ed9d700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2810108a50 0x7f281019d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f280e59c700 1 -- 172.21.15.98:0/1149163898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2810108a50 msgr2=0x7f281019d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f280e59c700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2810108a50 0x7f281019d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f280dd9b700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281006b990 0x7f28101993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f280e59c700 1 -- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281006b990 msgr2=0x7f28101993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:30.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f280e59c700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281006b990 0x7f28101993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f280e59c700 1 -- 172.21.15.98:0/1149163898 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f281019de70 con 0x7f281006aff0 2024-04-25T17:44:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f280dd9b700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281006b990 0x7f28101993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:44:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f280e59c700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f281006aff0 0x7f2810198e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f280000b550 tx=0x7f280000b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f27ff7fe700 1 -- 172.21.15.98:0/1149163898 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800004830 con 0x7f281006aff0 2024-04-25T17:44:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f27ff7fe700 1 -- 172.21.15.98:0/1149163898 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2800005020 con 0x7f281006aff0 2024-04-25T17:44:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.495+0000 7f27ff7fe700 1 -- 172.21.15.98:0/1149163898 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280001a6e0 con 0x7f281006aff0 2024-04-25T17:44:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.496+0000 7f2814ad6700 1 -- 172.21.15.98:0/1149163898 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f281019e160 con 0x7f281006aff0 2024-04-25T17:44:30.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.496+0000 7f2814ad6700 1 -- 172.21.15.98:0/1149163898 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2810101930 con 0x7f281006aff0 2024-04-25T17:44:30.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.498+0000 7f27ff7fe700 1 -- 172.21.15.98:0/1149163898 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f28000049d0 con 0x7f281006aff0 2024-04-25T17:44:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.498+0000 7f27ff7fe700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27f80a3fb0 0x7f27f80a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.498+0000 7f27ff7fe700 1 -- 172.21.15.98:0/1149163898 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f28000d85e0 con 0x7f281006aff0 2024-04-25T17:44:30.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.499+0000 7f280dd9b700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27f80a3fb0 0x7f27f80a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.499+0000 7f280dd9b700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27f80a3fb0 0x7f27f80a6470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2804005fd0 tx=0x7f2804005e20 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:30.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.499+0000 7f281f35fb80 1 Processor -- start 2024-04-25T17:44:30.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.500+0000 7f281f35fb80 1 -- start start 2024-04-25T17:44:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.500+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f79696e0 0x55e5f7969ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.500+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f796a000 0x55e5f79749c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.500+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 0x55e5f79772f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f281f35fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e5f78bce20 con 0x55e5f79696e0 2024-04-25T17:44:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f281f35fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e5f7863ac0 con 0x55e5f7974f00 2024-04-25T17:44:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f281f35fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e5f78b8de0 con 0x55e5f796a000 2024-04-25T17:44:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f280ed9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 0x55e5f79772f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f280ed9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 0x55e5f79772f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:42140/0 (socket says 172.21.15.98:42140) 2024-04-25T17:44:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f280ed9d700 1 -- 172.21.15.98:0/794734444 learned_addr learned my addr 172.21.15.98:0/794734444 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:30.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f280dd9b700 1 --2- 172.21.15.98:0/794734444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f796a000 0x55e5f79749c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f280ed9d700 1 -- 172.21.15.98:0/794734444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f796a000 msgr2=0x55e5f79749c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:30.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f280ed9d700 1 --2- 172.21.15.98:0/794734444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f796a000 0x55e5f79749c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f280ed9d700 1 -- 172.21.15.98:0/794734444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f79696e0 msgr2=0x55e5f7969ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:30.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f280ed9d700 1 --2- 172.21.15.98:0/794734444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f79696e0 0x55e5f7969ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.501+0000 7f280ed9d700 1 -- 172.21.15.98:0/794734444 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5f78278d0 con 0x55e5f7974f00 2024-04-25T17:44:30.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.502+0000 7f280ed9d700 1 --2- 172.21.15.98:0/794734444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 0x55e5f79772f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f280800b0a0 tx=0x7f2808009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ef491d19950fe1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:30.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.502+0000 7f27fcff9700 1 -- 172.21.15.98:0/794734444 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2808003a00 con 0x55e5f7974f00 2024-04-25T17:44:30.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.502+0000 7f27fcff9700 1 -- 172.21.15.98:0/794734444 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2808003ba0 con 0x55e5f7974f00 2024-04-25T17:44:30.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.502+0000 7f27fcff9700 1 -- 172.21.15.98:0/794734444 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28080052a0 con 0x55e5f7974f00 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 -- 172.21.15.98:0/794734444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 msgr2=0x55e5f79772f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 --2- 172.21.15.98:0/794734444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 0x55e5f79772f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f280800b0a0 tx=0x7f2808009520 comp rx=0 tx=0).stop 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 -- 172.21.15.98:0/794734444 shutdown_connections 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 --2- 172.21.15.98:0/794734444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 0x55e5f79772f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 --2- 172.21.15.98:0/794734444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f79696e0 0x55e5f7969ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 --2- 172.21.15.98:0/794734444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f796a000 0x55e5f79749c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 -- 172.21.15.98:0/794734444 >> 172.21.15.98:0/794734444 conn(0x55e5f7896a20 msgr2=0x55e5f7965ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 -- 172.21.15.98:0/794734444 shutdown_connections 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 -- 172.21.15.98:0/794734444 wait complete. 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 -- start start 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.503+0000 7f281f35fb80 1 Processor -- start 2024-04-25T17:44:30.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79696e0 0x55e5f7974730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f796a000 0x55e5f7971a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f7974f00 0x55e5f7971f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f281f35fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e5f7977930 con 0x55e5f7974f00 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f281f35fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e5f7876740 con 0x55e5f79696e0 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f281f35fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e5f7977ab0 con 0x55e5f796a000 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f281f35fb80 10 client.0 _ll_register_callbacks cb 0x55e5f7855ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f280e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79696e0 0x55e5f7974730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f280e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79696e0 0x55e5f7974730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:42148/0 (socket says 172.21.15.98:42148) 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f280e59c700 1 -- 172.21.15.98:0/2449509680 learned_addr learned my addr 172.21.15.98:0/2449509680 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f280dd9b700 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f796a000 0x55e5f7971a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f280ed9d700 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f7974f00 0x55e5f7971f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f280e59c700 1 -- 172.21.15.98:0/2449509680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f796a000 msgr2=0x55e5f7971a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f280e59c700 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f796a000 0x55e5f7971a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f280e59c700 1 -- 172.21.15.98:0/2449509680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f7974f00 msgr2=0x55e5f7971f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f280e59c700 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f7974f00 0x55e5f7971f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.504+0000 7f280e59c700 1 -- 172.21.15.98:0/2449509680 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5f780aa60 con 0x55e5f79696e0 2024-04-25T17:44:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.505+0000 7f280e59c700 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79696e0 0x55e5f7974730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2800009230 tx=0x7f28000a3f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ae92917b76d969b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.505+0000 7f27f5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:44:30.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.505+0000 7f27f5ffb700 1 -- 172.21.15.98:0/2449509680 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28000bd060 con 0x55e5f79696e0 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.505+0000 7f281f35fb80 1 -- 172.21.15.98:0/2449509680 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2808003ba0 con 0x55e5f79696e0 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.505+0000 7f27f5ffb700 1 -- 172.21.15.98:0/2449509680 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f28000bfe00 con 0x55e5f79696e0 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.505+0000 7f27f5ffb700 1 -- 172.21.15.98:0/2449509680 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f28000a4c20 con 0x55e5f79696e0 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f281f35fb80 10 client.25745 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f281f35fb80 1 -- 172.21.15.98:0/2449509680 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e5f7982360 con 0x55e5f79696e0 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f27f5ffb700 1 -- 172.21.15.98:0/2449509680 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2808003ba0 con 0x55e5f79696e0 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f27f5ffb700 1 -- 172.21.15.98:0/2449509680 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f28000acd00 con 0x55e5f79696e0 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f281f35fb80 10 client.25745 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f281f35fb80 10 client.25745 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f281f35fb80 20 client.25745 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f281f35fb80 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5f7984d70 0x55e5f7987190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f281f35fb80 4 client.25745 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f281f35fb80 1 -- 172.21.15.98:0/2449509680 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5f78ca610 con 0x55e5f7984d70 2024-04-25T17:44:30.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.507+0000 7f280dd9b700 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5f7984d70 0x55e5f7987190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.508+0000 7f280dd9b700 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5f7984d70 0x55e5f7987190 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d777495a5307f12a server_cookie=cf9608c347ddf86f in_seq=0 out_seq=0 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.508+0000 7f27f5ffb700 10 client.25745 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f27f5ffb700 1 -- 172.21.15.98:0/2449509680 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5f78ca610 con 0x55e5f7984d70 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f27f5ffb700 10 client.25745 handle_command_reply: tid=0 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f281f35fb80 1 client.25745 shutdown 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f281f35fb80 1 -- 172.21.15.98:0/2449509680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79696e0 msgr2=0x55e5f7974730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f281f35fb80 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79696e0 0x55e5f7974730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2800009230 tx=0x7f28000a3f00 comp rx=0 tx=0).stop 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f281f35fb80 1 -- 172.21.15.98:0/2449509680 shutdown_connections 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f281f35fb80 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5f7984d70 0x55e5f7987190 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f281f35fb80 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f7974f00 0x55e5f7971f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f281f35fb80 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79696e0 0x55e5f7974730 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2800009230 tx=0x7f28000a3f00 comp rx=0 tx=0).stop 2024-04-25T17:44:30.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f27f5ffb700 0 client.25745 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f281f35fb80 1 --2- 172.21.15.98:0/2449509680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f796a000 0x55e5f7971a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.509+0000 7f281f35fb80 1 -- 172.21.15.98:0/2449509680 >> 172.21.15.98:0/2449509680 conn(0x55e5f7896a20 msgr2=0x55e5f796c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.510+0000 7f281f35fb80 1 -- 172.21.15.98:0/2449509680 shutdown_connections 2024-04-25T17:44:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.510+0000 7f281f35fb80 1 -- 172.21.15.98:0/2449509680 wait complete. 2024-04-25T17:44:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.510+0000 7f281f35fb80 20 client.25745 trim_cache size 0 max 16384 2024-04-25T17:44:30.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:30 smithi173 ceph-mon[32856]: pgmap v267: 97 pgs: 97 active+clean; 3.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.1 MiB/s wr, 121 op/s 2024-04-25T17:44:30.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f281f35fb80 1 Processor -- start 2024-04-25T17:44:30.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f281f35fb80 1 -- start start 2024-04-25T17:44:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f796d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f7974f00 0x55e5f7974270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79747b0 0x55e5f798ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f281f35fb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f28101ad610 con 0x55e5f796a000 2024-04-25T17:44:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f281f35fb80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f28101aade0 con 0x55e5f79747b0 2024-04-25T17:44:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f281f35fb80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f28101a8c70 con 0x55e5f7974f00 2024-04-25T17:44:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f280e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f796d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f280e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f796d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50000/0 (socket says 172.21.15.98:50000) 2024-04-25T17:44:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.535+0000 7f280e59c700 1 -- 172.21.15.98:0/208086245 learned_addr learned my addr 172.21.15.98:0/208086245 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:30.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f280dd9b700 1 --2- 172.21.15.98:0/208086245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f7974f00 0x55e5f7974270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f280ed9d700 1 --2- 172.21.15.98:0/208086245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79747b0 0x55e5f798ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f280e59c700 1 -- 172.21.15.98:0/208086245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f7974f00 msgr2=0x55e5f7974270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f280e59c700 1 --2- 172.21.15.98:0/208086245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f7974f00 0x55e5f7974270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f280e59c700 1 -- 172.21.15.98:0/208086245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79747b0 msgr2=0x55e5f798ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f280e59c700 1 --2- 172.21.15.98:0/208086245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79747b0 0x55e5f798ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f280e59c700 1 -- 172.21.15.98:0/208086245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5f7982170 con 0x55e5f796a000 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f280ed9d700 1 --2- 172.21.15.98:0/208086245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79747b0 0x55e5f798ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f280e59c700 1 --2- 172.21.15.98:0/208086245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f796d280 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f28000a3ff0 tx=0x7f28000a2a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11f4561538e079f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f27f7fff700 1 -- 172.21.15.98:0/208086245 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28000c1070 con 0x55e5f796a000 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f27f7fff700 1 -- 172.21.15.98:0/208086245 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28000c3170 con 0x55e5f796a000 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.536+0000 7f27f7fff700 1 -- 172.21.15.98:0/208086245 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28000acd00 con 0x55e5f796a000 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 -- 172.21.15.98:0/208086245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 msgr2=0x55e5f796d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:30.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 --2- 172.21.15.98:0/208086245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f796d280 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f28000a3ff0 tx=0x7f28000a2a30 comp rx=0 tx=0).stop 2024-04-25T17:44:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 -- 172.21.15.98:0/208086245 shutdown_connections 2024-04-25T17:44:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 --2- 172.21.15.98:0/208086245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f79747b0 0x55e5f798ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 --2- 172.21.15.98:0/208086245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f796d280 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 --2- 172.21.15.98:0/208086245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f7974f00 0x55e5f7974270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 -- 172.21.15.98:0/208086245 >> 172.21.15.98:0/208086245 conn(0x55e5f7896a20 msgr2=0x55e5f7964cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 -- 172.21.15.98:0/208086245 shutdown_connections 2024-04-25T17:44:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 -- 172.21.15.98:0/208086245 wait complete. 2024-04-25T17:44:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 -- start start 2024-04-25T17:44:30.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.537+0000 7f281f35fb80 1 Processor -- start 2024-04-25T17:44:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f7978ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f79747b0 0x55e5f7979400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f281f35fb80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 0x55e5f79ea2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f280e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f7978ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f280e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f7978ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50004/0 (socket says 172.21.15.98:50004) 2024-04-25T17:44:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f280e59c700 1 -- 172.21.15.98:0/1425816819 learned_addr learned my addr 172.21.15.98:0/1425816819 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f28101a92c0 con 0x55e5f796a000 2024-04-25T17:44:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f28101aa710 con 0x55e5f7974f00 2024-04-25T17:44:30.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f28101a9970 con 0x55e5f79747b0 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f280dd9b700 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f79747b0 0x55e5f7979400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f280ed9d700 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 0x55e5f79ea2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f281f35fb80 10 client.0 _ll_register_callbacks cb 0x55e5f7855ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f280e59c700 1 -- 172.21.15.98:0/1425816819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f79747b0 msgr2=0x55e5f7979400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f280e59c700 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f79747b0 0x55e5f7979400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f280e59c700 1 -- 172.21.15.98:0/1425816819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 msgr2=0x55e5f79ea2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f280e59c700 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 0x55e5f79ea2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.538+0000 7f280e59c700 1 -- 172.21.15.98:0/1425816819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5f79ea890 con 0x55e5f796a000 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.539+0000 7f280e59c700 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f7978ec0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f28000a4b80 tx=0x7f28000a3f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d004d8989d8a99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.539+0000 7f27f5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.540+0000 7f27f5ffb700 1 -- 172.21.15.98:0/1425816819 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28000b2020 con 0x55e5f796a000 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.540+0000 7f27f5ffb700 1 -- 172.21.15.98:0/1425816819 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f28000a3970 con 0x55e5f796a000 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.540+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28000c3170 con 0x55e5f796a000 2024-04-25T17:44:30.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.540+0000 7f27f5ffb700 1 -- 172.21.15.98:0/1425816819 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f28000bd020 con 0x55e5f796a000 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.540+0000 7f281f35fb80 10 client.16254 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.540+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e5f79ebac0 con 0x55e5f796a000 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.540+0000 7f27f5ffb700 1 -- 172.21.15.98:0/1425816819 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28000a88d0 con 0x55e5f796a000 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.540+0000 7f27f5ffb700 1 -- 172.21.15.98:0/1425816819 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f28000a33e0 con 0x55e5f796a000 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.541+0000 7f281f35fb80 10 client.16254 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.541+0000 7f281f35fb80 10 client.16254 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.541+0000 7f281f35fb80 20 client.16254 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.541+0000 7f281f35fb80 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5f79ebfa0 0x55e5f79ee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.541+0000 7f281f35fb80 4 client.16254 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.3, "await_for": 42.0}] 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.541+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 85.3, "await_for": 42.0}) -- 0x7f28000c3170 con 0x55e5f79ebfa0 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.541+0000 7f280dd9b700 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5f79ebfa0 0x55e5f79ee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.541+0000 7f280dd9b700 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5f79ebfa0 0x55e5f79ee3c0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31747884e558a22 server_cookie=98420f5950f5a418 in_seq=0 out_seq=0 2024-04-25T17:44:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.541+0000 7f27f5ffb700 10 client.16254 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f27f5ffb700 1 -- 172.21.15.98:0/1425816819 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x7f28000c3170 con 0x55e5f79ebfa0 2024-04-25T17:44:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f27f5ffb700 10 client.16254 handle_command_reply: tid=0 2024-04-25T17:44:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 client.16254 shutdown 2024-04-25T17:44:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 msgr2=0x55e5f7978ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f7978ec0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f28000a4b80 tx=0x7f28000a3f00 comp rx=0 tx=0).stop 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 shutdown_connections 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5f79ebfa0 0x55e5f79ee3c0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5f7974f00 0x55e5f79ea2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5f796a000 0x55e5f7978ec0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f27f5ffb700 0 client.16254 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 --2- 172.21.15.98:0/1425816819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5f79747b0 0x55e5f7979400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 >> 172.21.15.98:0/1425816819 conn(0x55e5f7896a20 msgr2=0x55e5f796b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 shutdown_connections 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 1 -- 172.21.15.98:0/1425816819 wait complete. 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.760+0000 7f281f35fb80 20 client.16254 trim_cache size 0 max 16384 2024-04-25T17:44:30.761 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 38, 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 26, 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "a4035ce8": { 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 26, 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:44:30.762 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 85.3, 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:44:30.763 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 -- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27f80a3fb0 msgr2=0x7f27f80a6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27f80a3fb0 0x7f27f80a6470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2804005fd0 tx=0x7f2804005e20 comp rx=0 tx=0).stop 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 -- 172.21.15.98:0/1149163898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f281006aff0 msgr2=0x7f2810198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f281006aff0 0x7f2810198e60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f280000b550 tx=0x7f280000b860 comp rx=0 tx=0).stop 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 -- 172.21.15.98:0/1149163898 shutdown_connections 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f27f80a3fb0 0x7f27f80a6470 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f281006aff0 0x7f2810198e60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2810108a50 0x7f281019d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 --2- 172.21.15.98:0/1149163898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281006b990 0x7f28101993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 -- 172.21.15.98:0/1149163898 >> 172.21.15.98:0/1149163898 conn(0x7f2810078980 msgr2=0x7f2810100900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 -- 172.21.15.98:0/1149163898 shutdown_connections 2024-04-25T17:44:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:30.761+0000 7f2814ad6700 1 -- 172.21.15.98:0/1149163898 wait complete. 2024-04-25T17:44:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 480.0 s, 2535.7 tps, lat 32.001 ms stddev 34.608 2024-04-25T17:44:31.971 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a4035ce8', quiesce took 7.6 seconds. Will release after: 25.3 2024-04-25T17:44:32.204 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:32 smithi098 ceph-mon[29192]: pgmap v268: 97 pgs: 97 active+clean; 3.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 1.7 MiB/s wr, 185 op/s 2024-04-25T17:44:32.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:32 smithi203 ceph-mon[32834]: pgmap v268: 97 pgs: 97 active+clean; 3.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 1.7 MiB/s wr, 185 op/s 2024-04-25T17:44:32.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:32 smithi173 ceph-mon[32856]: pgmap v268: 97 pgs: 97 active+clean; 3.6 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 1.7 MiB/s wr, 185 op/s 2024-04-25T17:44:34.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:34 smithi098 ceph-mon[29192]: pgmap v269: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 4.0 MiB/s wr, 416 op/s 2024-04-25T17:44:34.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:34 smithi203 ceph-mon[32834]: pgmap v269: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 4.0 MiB/s wr, 416 op/s 2024-04-25T17:44:34.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:34 smithi173 ceph-mon[32856]: pgmap v269: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 4.0 MiB/s wr, 416 op/s 2024-04-25T17:44:36.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:36 smithi098 ceph-mon[29192]: pgmap v270: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 4.0 MiB/s wr, 414 op/s 2024-04-25T17:44:36.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:36 smithi203 ceph-mon[32834]: pgmap v270: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 4.0 MiB/s wr, 414 op/s 2024-04-25T17:44:36.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:36 smithi173 ceph-mon[32856]: pgmap v270: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 4.0 MiB/s wr, 414 op/s 2024-04-25T17:44:36.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 485.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:44:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.892+0000 7f801a4fc640 1 -- 172.21.15.98:0/4217770131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80141021c0 msgr2=0x7f80141025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.893+0000 7f801a4fc640 1 --2- 172.21.15.98:0/4217770131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80141021c0 0x7f80141025a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8004009900 tx=0x7f800402a410 comp rx=0 tx=0).stop 2024-04-25T17:44:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.893+0000 7f801a4fc640 1 -- 172.21.15.98:0/4217770131 shutdown_connections 2024-04-25T17:44:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.893+0000 7f801a4fc640 1 --2- 172.21.15.98:0/4217770131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8014107290 0x7f801410ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.893+0000 7f801a4fc640 1 --2- 172.21.15.98:0/4217770131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8014102b70 0x7f8014106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.893+0000 7f801a4fc640 1 --2- 172.21.15.98:0/4217770131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80141021c0 0x7f80141025a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.893+0000 7f801a4fc640 1 -- 172.21.15.98:0/4217770131 >> 172.21.15.98:0/4217770131 conn(0x7f80140fda20 msgr2=0x7f80140ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.893+0000 7f801a4fc640 1 -- 172.21.15.98:0/4217770131 shutdown_connections 2024-04-25T17:44:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.893+0000 7f801a4fc640 1 -- 172.21.15.98:0/4217770131 wait complete. 2024-04-25T17:44:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.893+0000 7f801a4fc640 1 Processor -- start 2024-04-25T17:44:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.893+0000 7f801a4fc640 1 -- start start 2024-04-25T17:44:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f801a4fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80141021c0 0x7f8014198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f801a4fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8014107290 0x7f8014198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f801a4fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f801419cf00 0x7f801419d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f801a4fc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8014112200 con 0x7f801419cf00 2024-04-25T17:44:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f801a4fc640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8014112080 con 0x7f8014107290 2024-04-25T17:44:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f801a4fc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8014112380 con 0x7f80141021c0 2024-04-25T17:44:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f8018a99640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f801419cf00 0x7f801419d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f8018a99640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f801419cf00 0x7f801419d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34672/0 (socket says 172.21.15.98:34672) 2024-04-25T17:44:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f8018a99640 1 -- 172.21.15.98:0/3247052262 learned_addr learned my addr 172.21.15.98:0/3247052262 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f8013fff640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80141021c0 0x7f8014198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f80137fe640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8014107290 0x7f8014198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f8018a99640 1 -- 172.21.15.98:0/3247052262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80141021c0 msgr2=0x7f8014198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f8018a99640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80141021c0 0x7f8014198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f8018a99640 1 -- 172.21.15.98:0/3247052262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8014107290 msgr2=0x7f8014198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f8018a99640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8014107290 0x7f8014198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f8018a99640 1 -- 172.21.15.98:0/3247052262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f801419d9f0 con 0x7f801419cf00 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f80137fe640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8014107290 0x7f8014198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f8018a99640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f801419cf00 0x7f801419d3b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f800c00eaa0 tx=0x7f800c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f80117fa640 1 -- 172.21.15.98:0/3247052262 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800c00cdb0 con 0x7f801419cf00 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f80117fa640 1 -- 172.21.15.98:0/3247052262 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f800c005020 con 0x7f801419cf00 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f801a4fc640 1 -- 172.21.15.98:0/3247052262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f801419dce0 con 0x7f801419cf00 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.894+0000 7f80117fa640 1 -- 172.21.15.98:0/3247052262 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800c0116d0 con 0x7f801419cf00 2024-04-25T17:44:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.895+0000 7f801a4fc640 1 -- 172.21.15.98:0/3247052262 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80141a54d0 con 0x7f801419cf00 2024-04-25T17:44:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.897+0000 7f80117fa640 1 -- 172.21.15.98:0/3247052262 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f800c0051d0 con 0x7f801419cf00 2024-04-25T17:44:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.897+0000 7f801d20c740 1 Processor -- start 2024-04-25T17:44:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.897+0000 7f801d20c740 1 -- start start 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f801d20c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8f620 0x5563f1a8fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f801d20c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8ff40 0x5563f1a9a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f801d20c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a9a800 0x5563f1a9cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f801d20c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5563f19f4970 con 0x5563f1a8ff40 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f801d20c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5563f19914a0 con 0x5563f1a9a800 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f801d20c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5563f19f0930 con 0x5563f1a8f620 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f80117fa640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7ffc0a3dc0 0x7f7ffc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8013fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8ff40 0x5563f1a9a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8013fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8ff40 0x5563f1a9a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34684/0 (socket says 172.21.15.98:34684) 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8013fff640 1 -- 172.21.15.98:0/3416677904 learned_addr learned my addr 172.21.15.98:0/3416677904 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8018a99640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7ffc0a3dc0 0x7f7ffc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f80137fe640 1 --2- 172.21.15.98:0/3416677904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a9a800 0x5563f1a9cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f80117fa640 1 -- 172.21.15.98:0/3247052262 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f800c0d09a0 con 0x7f801419cf00 2024-04-25T17:44:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8013fff640 1 -- 172.21.15.98:0/3416677904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8f620 msgr2=0x5563f1a8fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8013fff640 1 --2- 172.21.15.98:0/3416677904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8f620 0x5563f1a8fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8013fff640 1 -- 172.21.15.98:0/3416677904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a9a800 msgr2=0x5563f1a9cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8013fff640 1 --2- 172.21.15.98:0/3416677904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a9a800 0x5563f1a9cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8013fff640 1 -- 172.21.15.98:0/3416677904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563f1707210 con 0x5563f1a8ff40 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f80137fe640 1 --2- 172.21.15.98:0/3416677904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a9a800 0x5563f1a9cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8018a99640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7ffc0a3dc0 0x7f7ffc0a6280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f800c09c870 tx=0x7f800c000f40 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.898+0000 7f8013fff640 1 --2- 172.21.15.98:0/3416677904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8ff40 0x5563f1a9a2c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8004009900 tx=0x7f8004032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee12fc3e8894abe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f7ffa7fc640 1 -- 172.21.15.98:0/3416677904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8004040070 con 0x5563f1a8ff40 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f7ffa7fc640 1 -- 172.21.15.98:0/3416677904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80040316a0 con 0x5563f1a8ff40 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f7ffa7fc640 1 -- 172.21.15.98:0/3416677904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8004035040 con 0x5563f1a8ff40 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f801d20c740 1 -- 172.21.15.98:0/3416677904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8ff40 msgr2=0x5563f1a9a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f801d20c740 1 --2- 172.21.15.98:0/3416677904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8ff40 0x5563f1a9a2c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8004009900 tx=0x7f8004032e60 comp rx=0 tx=0).stop 2024-04-25T17:44:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f801d20c740 1 -- 172.21.15.98:0/3416677904 shutdown_connections 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f801d20c740 1 --2- 172.21.15.98:0/3416677904 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a9a800 0x5563f1a9cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f801d20c740 1 --2- 172.21.15.98:0/3416677904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8ff40 0x5563f1a9a2c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f801d20c740 1 --2- 172.21.15.98:0/3416677904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8f620 0x5563f1a8fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f801d20c740 1 -- 172.21.15.98:0/3416677904 >> 172.21.15.98:0/3416677904 conn(0x5563f1a8bb30 msgr2=0x5563f1a8bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f801d20c740 1 -- 172.21.15.98:0/3416677904 shutdown_connections 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.899+0000 7f801d20c740 1 -- 172.21.15.98:0/3416677904 wait complete. 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f801d20c740 1 -- start start 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f801d20c740 1 Processor -- start 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f801d20c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8f620 0x5563f1a9a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f801d20c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 0x5563f1a973d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f801d20c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a9a800 0x5563f1a97910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f801d20c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5563f1a9d3d0 con 0x5563f1a8f620 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f801d20c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5563f198f0b0 con 0x5563f1a9a800 2024-04-25T17:44:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f801d20c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5563f1a9d550 con 0x5563f1a8ff40 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f801d20c740 10 client.0 _ll_register_callbacks cb 0x5563f197cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f80137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8f620 0x5563f1a9a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f80137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8f620 0x5563f1a9a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34698/0 (socket says 172.21.15.98:34698) 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.900+0000 7f80137fe640 1 -- 172.21.15.98:0/379916195 learned_addr learned my addr 172.21.15.98:0/379916195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f8013fff640 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 0x5563f1a973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f80137fe640 1 -- 172.21.15.98:0/379916195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 msgr2=0x5563f1a973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f80137fe640 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 0x5563f1a973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f80137fe640 1 -- 172.21.15.98:0/379916195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a9a800 msgr2=0x5563f1a97910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f80137fe640 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a9a800 0x5563f1a97910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f80137fe640 1 -- 172.21.15.98:0/379916195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563f17d3620 con 0x5563f1a8f620 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f80137fe640 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8f620 0x5563f1a9a0e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8008008ea0 tx=0x7f8008008f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd90930c4711d7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f7ff37fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f7ff37fe640 1 -- 172.21.15.98:0/379916195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800800eb40 con 0x5563f1a8f620 2024-04-25T17:44:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f7ff37fe640 1 -- 172.21.15.98:0/379916195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f800800ece0 con 0x5563f1a8f620 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f801d20c740 1 -- 172.21.15.98:0/379916195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80040316a0 con 0x5563f1a8f620 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f7ff37fe640 1 -- 172.21.15.98:0/379916195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8008005020 con 0x5563f1a8f620 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f801d20c740 10 client.16266 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f801d20c740 1 -- 172.21.15.98:0/379916195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5563f1aa9390 con 0x5563f1a8f620 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f7ff37fe640 1 -- 172.21.15.98:0/379916195 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80040316a0 con 0x5563f1a8f620 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.901+0000 7f7ff37fe640 1 -- 172.21.15.98:0/379916195 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f8008004d00 con 0x5563f1a8f620 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.902+0000 7f801d20c740 10 client.16266 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.902+0000 7f801d20c740 10 client.16266 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.902+0000 7f801d20c740 20 client.16266 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.902+0000 7f801d20c740 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563f1aaa850 0x5563f1aacc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.902+0000 7f801d20c740 4 client.16266 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.902+0000 7f801d20c740 1 -- 172.21.15.98:0/379916195 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563f1a02390 con 0x5563f1aaa850 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.902+0000 7f8013fff640 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563f1aaa850 0x5563f1aacc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.902+0000 7f8013fff640 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563f1aaa850 0x5563f1aacc90 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d815637368164806 server_cookie=d889680ce83295 in_seq=0 out_seq=0 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.902+0000 7f7ff37fe640 10 client.16266 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f7ff37fe640 1 -- 172.21.15.98:0/379916195 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563f1a02390 con 0x5563f1aaa850 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f7ff37fe640 10 client.16266 handle_command_reply: tid=0 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 client.16266 shutdown 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 -- 172.21.15.98:0/379916195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8f620 msgr2=0x5563f1a9a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8f620 0x5563f1a9a0e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8008008ea0 tx=0x7f8008008f90 comp rx=0 tx=0).stop 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 -- 172.21.15.98:0/379916195 shutdown_connections 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563f1aaa850 0x5563f1aacc90 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a9a800 0x5563f1a97910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f7ff37fe640 0 client.16266 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 0x5563f1a973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 --2- 172.21.15.98:0/379916195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a8f620 0x5563f1a9a0e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 -- 172.21.15.98:0/379916195 >> 172.21.15.98:0/379916195 conn(0x5563f1a8bb30 msgr2=0x5563f1a91860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 -- 172.21.15.98:0/379916195 shutdown_connections 2024-04-25T17:44:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 1 -- 172.21.15.98:0/379916195 wait complete. 2024-04-25T17:44:37.911 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.903+0000 7f801d20c740 20 client.16266 trim_cache size 0 max 16384 2024-04-25T17:44:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f801d20c740 1 Processor -- start 2024-04-25T17:44:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f801d20c740 1 -- start start 2024-04-25T17:44:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f801d20c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a8f620 0x5563f1aa62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f801d20c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 0x5563f1a95b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f801d20c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a99c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f801d20c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80141b0440 con 0x5563f1a9a800 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f801d20c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f80141adc50 con 0x5563f1a8f620 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f801d20c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80141abac0 con 0x5563f1a8ff40 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f80137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a99c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f80137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a99c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34712/0 (socket says 172.21.15.98:34712) 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f80137fe640 1 -- 172.21.15.98:0/851378781 learned_addr learned my addr 172.21.15.98:0/851378781 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f8013fff640 1 --2- 172.21.15.98:0/851378781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 0x5563f1a95b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f80137fe640 1 -- 172.21.15.98:0/851378781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 msgr2=0x5563f1a95b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f80137fe640 1 --2- 172.21.15.98:0/851378781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 0x5563f1a95b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f80137fe640 1 -- 172.21.15.98:0/851378781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a8f620 msgr2=0x5563f1aa62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.918+0000 7f80137fe640 1 --2- 172.21.15.98:0/851378781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a8f620 0x5563f1aa62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f80137fe640 1 -- 172.21.15.98:0/851378781 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563f1a8f3e0 con 0x5563f1a9a800 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f8013fff640 1 --2- 172.21.15.98:0/851378781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 0x5563f1a95b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f80137fe640 1 --2- 172.21.15.98:0/851378781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a99c10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8008008850 tx=0x7f8008012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b858449c74a7946 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f7ff9ffb640 1 -- 172.21.15.98:0/851378781 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8008029070 con 0x5563f1a9a800 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f7ff9ffb640 1 -- 172.21.15.98:0/851378781 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8008016070 con 0x5563f1a9a800 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f7ff9ffb640 1 -- 172.21.15.98:0/851378781 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800800eda0 con 0x5563f1a9a800 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f801d20c740 1 -- 172.21.15.98:0/851378781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 msgr2=0x5563f1a99c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f801d20c740 1 --2- 172.21.15.98:0/851378781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a99c10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8008008850 tx=0x7f8008012f90 comp rx=0 tx=0).stop 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f801d20c740 1 -- 172.21.15.98:0/851378781 shutdown_connections 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f801d20c740 1 --2- 172.21.15.98:0/851378781 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a99c10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f801d20c740 1 --2- 172.21.15.98:0/851378781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8ff40 0x5563f1a95b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f801d20c740 1 --2- 172.21.15.98:0/851378781 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a8f620 0x5563f1aa62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.919+0000 7f801d20c740 1 -- 172.21.15.98:0/851378781 >> 172.21.15.98:0/851378781 conn(0x5563f1a8bb30 msgr2=0x5563f1a91860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 1 -- 172.21.15.98:0/851378781 shutdown_connections 2024-04-25T17:44:37.924 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 1 -- 172.21.15.98:0/851378781 wait complete. 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 1 -- start start 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 1 Processor -- start 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8f620 0x5563f1a94100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a8ff40 0x5563f1a94640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a94b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80141ac110 con 0x5563f1a9a800 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f80141ad580 con 0x5563f1a8ff40 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80141ac7e0 con 0x5563f1a8f620 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f801d20c740 10 client.0 _ll_register_callbacks cb 0x7f8008012620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f80137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a94b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f80137fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a94b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34718/0 (socket says 172.21.15.98:34718) 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.920+0000 7f80137fe640 1 -- 172.21.15.98:0/3470997133 learned_addr learned my addr 172.21.15.98:0/3470997133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f8013fff640 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a8ff40 0x5563f1a94640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f80137fe640 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8f620 0x5563f1a94100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f80137fe640 1 -- 172.21.15.98:0/3470997133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8f620 msgr2=0x5563f1a94100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f80137fe640 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8f620 0x5563f1a94100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f80137fe640 1 -- 172.21.15.98:0/3470997133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a8ff40 msgr2=0x5563f1a94640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f80137fe640 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a8ff40 0x5563f1a94640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f80137fe640 1 -- 172.21.15.98:0/3470997133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563f1b0fdd0 con 0x5563f1a9a800 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f80137fe640 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a94b80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8008009010 tx=0x7f80080238f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78bc796d589fe63e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f7ff37fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f7ff37fe640 1 -- 172.21.15.98:0/3470997133 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8008029030 con 0x5563f1a9a800 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f7ff37fe640 1 -- 172.21.15.98:0/3470997133 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f80080053c0 con 0x5563f1a9a800 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f801d20c740 1 -- 172.21.15.98:0/3470997133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8008016070 con 0x5563f1a9a800 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f7ff37fe640 1 -- 172.21.15.98:0/3470997133 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f800800e720 con 0x5563f1a9a800 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f801d20c740 10 client.16272 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f801d20c740 1 -- 172.21.15.98:0/3470997133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5563f1b10ff0 con 0x5563f1a9a800 2024-04-25T17:44:37.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f7ff37fe640 1 -- 172.21.15.98:0/3470997133 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8008016070 con 0x5563f1a9a800 2024-04-25T17:44:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f7ff37fe640 1 -- 172.21.15.98:0/3470997133 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f8008035030 con 0x5563f1a9a800 2024-04-25T17:44:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f801d20c740 10 client.16272 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f801d20c740 10 client.16272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:44:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f801d20c740 20 client.16272 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f801d20c740 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563f1b11650 0x5563f1b13a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f801d20c740 4 client.16272 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:44:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.921+0000 7f801d20c740 1 -- 172.21.15.98:0/3470997133 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563f19fc200 con 0x5563f1b11650 2024-04-25T17:44:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.922+0000 7f8013fff640 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563f1b11650 0x5563f1b13a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.922+0000 7f7ff37fe640 10 client.16272 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:37.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.922+0000 7f8013fff640 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563f1b11650 0x5563f1b13a70 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=775fe41c137406c9 server_cookie=1e35cd5ac3a34df2 in_seq=0 out_seq=0 2024-04-25T17:44:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f7ff37fe640 1 -- 172.21.15.98:0/3470997133 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563f19fc200 con 0x5563f1b11650 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f7ff37fe640 10 client.16272 handle_command_reply: tid=0 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f801d20c740 1 client.16272 shutdown 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f801d20c740 1 -- 172.21.15.98:0/3470997133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 msgr2=0x5563f1a94b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f801d20c740 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a94b80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8008009010 tx=0x7f80080238f0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f801d20c740 1 -- 172.21.15.98:0/3470997133 shutdown_connections 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f801d20c740 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5563f1b11650 0x5563f1b13a70 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f801d20c740 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5563f1a9a800 0x5563f1a94b80 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f801d20c740 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5563f1a8ff40 0x5563f1a94640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f7ff37fe640 0 client.16272 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f801d20c740 1 --2- 172.21.15.98:0/3470997133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5563f1a8f620 0x5563f1a94100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.930+0000 7f801d20c740 1 -- 172.21.15.98:0/3470997133 >> 172.21.15.98:0/3470997133 conn(0x5563f1a8bb30 msgr2=0x5563f1a96c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801d20c740 1 -- 172.21.15.98:0/3470997133 shutdown_connections 2024-04-25T17:44:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801d20c740 1 -- 172.21.15.98:0/3470997133 wait complete. 2024-04-25T17:44:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801d20c740 20 client.16272 trim_cache size 0 max 16384 2024-04-25T17:44:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:44:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:44:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:44:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:44:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 -- 172.21.15.98:0/3247052262 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7ffc0a3dc0 msgr2=0x7f7ffc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7ffc0a3dc0 0x7f7ffc0a6280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f800c09c870 tx=0x7f800c000f40 comp rx=0 tx=0).stop 2024-04-25T17:44:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 -- 172.21.15.98:0/3247052262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f801419cf00 msgr2=0x7f801419d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f801419cf00 0x7f801419d3b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f800c00eaa0 tx=0x7f800c00ef60 comp rx=0 tx=0).stop 2024-04-25T17:44:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 -- 172.21.15.98:0/3247052262 shutdown_connections 2024-04-25T17:44:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7ffc0a3dc0 0x7f7ffc0a6280 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f801419cf00 0x7f801419d3b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8014107290 0x7f8014198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 --2- 172.21.15.98:0/3247052262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80141021c0 0x7f8014198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:37.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 -- 172.21.15.98:0/3247052262 >> 172.21.15.98:0/3247052262 conn(0x7f80140fda20 msgr2=0x7f80140fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:37.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 -- 172.21.15.98:0/3247052262 shutdown_connections 2024-04-25T17:44:37.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:37.931+0000 7f801a4fc640 1 -- 172.21.15.98:0/3247052262 wait complete. 2024-04-25T17:44:38.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:38 smithi098 ceph-mon[29192]: pgmap v271: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 4.0 MiB/s wr, 426 op/s 2024-04-25T17:44:38.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:38 smithi203 ceph-mon[32834]: pgmap v271: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 4.0 MiB/s wr, 426 op/s 2024-04-25T17:44:38.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:38 smithi173 ceph-mon[32856]: pgmap v271: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 4.0 MiB/s wr, 426 op/s 2024-04-25T17:44:40.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:40 smithi098 ceph-mon[29192]: pgmap v272: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 2.9 MiB/s wr, 307 op/s 2024-04-25T17:44:40.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:40 smithi203 ceph-mon[32834]: pgmap v272: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 2.9 MiB/s wr, 307 op/s 2024-04-25T17:44:40.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:40 smithi173 ceph-mon[32856]: pgmap v272: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 2.9 MiB/s wr, 307 op/s 2024-04-25T17:44:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 490.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:44:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:42 smithi098 ceph-mon[29192]: pgmap v273: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 3.0 MiB/s wr, 310 op/s 2024-04-25T17:44:42.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:42 smithi203 ceph-mon[32834]: pgmap v273: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 3.0 MiB/s wr, 310 op/s 2024-04-25T17:44:42.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:42 smithi173 ceph-mon[32856]: pgmap v273: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 3.0 MiB/s wr, 310 op/s 2024-04-25T17:44:44.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:44 smithi098 ceph-mon[29192]: pgmap v274: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 2.3 MiB/s wr, 249 op/s 2024-04-25T17:44:44.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:44 smithi203 ceph-mon[32834]: pgmap v274: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 2.3 MiB/s wr, 249 op/s 2024-04-25T17:44:44.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:44 smithi173 ceph-mon[32856]: pgmap v274: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 2.3 MiB/s wr, 249 op/s 2024-04-25T17:44:46.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:46 smithi098 ceph-mon[29192]: pgmap v275: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 95 KiB/s wr, 17 op/s 2024-04-25T17:44:46.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:46 smithi203 ceph-mon[32834]: pgmap v275: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 95 KiB/s wr, 17 op/s 2024-04-25T17:44:46.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:46 smithi173 ceph-mon[32856]: pgmap v275: 97 pgs: 97 active+clean; 3.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 95 KiB/s wr, 17 op/s 2024-04-25T17:44:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 495.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:44:48.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:48 smithi098 ceph-mon[29192]: pgmap v276: 97 pgs: 97 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 9.2 MiB/s wr, 891 op/s 2024-04-25T17:44:48.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:48 smithi203 ceph-mon[32834]: pgmap v276: 97 pgs: 97 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 9.2 MiB/s wr, 891 op/s 2024-04-25T17:44:48.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:48 smithi173 ceph-mon[32856]: pgmap v276: 97 pgs: 97 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 9.2 MiB/s wr, 891 op/s 2024-04-25T17:44:50.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:50 smithi098 ceph-mon[29192]: pgmap v277: 97 pgs: 97 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 879 op/s 2024-04-25T17:44:50.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:50 smithi203 ceph-mon[32834]: pgmap v277: 97 pgs: 97 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 879 op/s 2024-04-25T17:44:50.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:50 smithi173 ceph-mon[32856]: pgmap v277: 97 pgs: 97 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 879 op/s 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.026+0000 7fefceefa640 1 -- 172.21.15.98:0/416008280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefc8106170 msgr2=0x7fefc810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.026+0000 7fefceefa640 1 --2- 172.21.15.98:0/416008280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefc8106170 0x7fefc810dd40 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009ca0 tx=0x7fefc002a8c0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.026+0000 7fefceefa640 1 -- 172.21.15.98:0/416008280 shutdown_connections 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.026+0000 7fefceefa640 1 --2- 172.21.15.98:0/416008280 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefc8106170 0x7fefc810dd40 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.026+0000 7fefceefa640 1 --2- 172.21.15.98:0/416008280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc8101aa0 0x7fefc8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.026+0000 7fefceefa640 1 --2- 172.21.15.98:0/416008280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefc81010f0 0x7fefc81014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.026+0000 7fefceefa640 1 -- 172.21.15.98:0/416008280 >> 172.21.15.98:0/416008280 conn(0x7fefc8077fd0 msgr2=0x7fefc80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.026+0000 7fefceefa640 1 -- 172.21.15.98:0/416008280 shutdown_connections 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.026+0000 7fefceefa640 1 -- 172.21.15.98:0/416008280 wait complete. 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefceefa640 1 Processor -- start 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefceefa640 1 -- start start 2024-04-25T17:44:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefceefa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc81010f0 0x7fefc8199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefceefa640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefc8106170 0x7fefc8199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefceefa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefc8199fd0 0x7fefc81a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefceefa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81110e0 con 0x7fefc8199fd0 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefceefa640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fefc8110f60 con 0x7fefc8106170 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefceefa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefc8111260 con 0x7fefc81010f0 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefcd497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefc8199fd0 0x7fefc81a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefcd497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefc8199fd0 0x7fefc81a4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56348/0 (socket says 172.21.15.98:56348) 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefcd497640 1 -- 172.21.15.98:0/1384598023 learned_addr learned my addr 172.21.15.98:0/1384598023 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefccc96640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc81010f0 0x7fefc8199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.027+0000 7fefc7fff640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefc8106170 0x7fefc8199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefcd497640 1 -- 172.21.15.98:0/1384598023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc81010f0 msgr2=0x7fefc8199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefcd497640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc81010f0 0x7fefc8199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefcd497640 1 -- 172.21.15.98:0/1384598023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefc8106170 msgr2=0x7fefc8199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefcd497640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefc8106170 0x7fefc8199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefcd497640 1 -- 172.21.15.98:0/1384598023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc81a4d80 con 0x7fefc8199fd0 2024-04-25T17:44:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefc7fff640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefc8106170 0x7fefc8199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:44:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefcd497640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefc8199fd0 0x7fefc81a4580 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fefc0005cc0 tx=0x7fefc002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefc5ffb640 1 -- 172.21.15.98:0/1384598023 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0003bf0 con 0x7fefc8199fd0 2024-04-25T17:44:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefc5ffb640 1 -- 172.21.15.98:0/1384598023 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefc0003d90 con 0x7fefc8199fd0 2024-04-25T17:44:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefceefa640 1 -- 172.21.15.98:0/1384598023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc81a5010 con 0x7fefc8199fd0 2024-04-25T17:44:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefc5ffb640 1 -- 172.21.15.98:0/1384598023 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc002fbc0 con 0x7fefc8199fd0 2024-04-25T17:44:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.028+0000 7fefceefa640 1 -- 172.21.15.98:0/1384598023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc81a55b0 con 0x7fefc8199fd0 2024-04-25T17:44:51.037 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.030+0000 7fefc5ffb640 1 -- 172.21.15.98:0/1384598023 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fefc002fd60 con 0x7fefc8199fd0 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefd1c12740 1 Processor -- start 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefc5ffb640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fefb80a3e40 0x7fefb80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefd1c12740 1 -- start start 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefc5ffb640 1 -- 172.21.15.98:0/1384598023 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fefc00f1e70 con 0x7fefc8199fd0 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefc7fff640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fefb80a3e40 0x7fefb80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b30500 0x55ab02b308e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 0x55ab02b3f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b42000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefd1c12740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab02a95970 con 0x55ab02b3fc10 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefd1c12740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ab02a324a0 con 0x55ab02b30e20 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefd1c12740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ab02a91930 con 0x55ab02b30500 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefcd497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b42000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefcd497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b42000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56360/0 (socket says 172.21.15.98:56360) 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefcd497640 1 -- 172.21.15.98:0/620606061 learned_addr learned my addr 172.21.15.98:0/620606061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.031+0000 7fefccc96640 1 --2- 172.21.15.98:0/620606061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b30500 0x55ab02b308e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefc7fff640 1 --2- 172.21.15.98:0/620606061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 0x55ab02b3f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefcd497640 1 -- 172.21.15.98:0/620606061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b30500 msgr2=0x55ab02b308e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefcd497640 1 --2- 172.21.15.98:0/620606061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b30500 0x55ab02b308e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefcd497640 1 -- 172.21.15.98:0/620606061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 msgr2=0x55ab02b3f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefcd497640 1 --2- 172.21.15.98:0/620606061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 0x55ab02b3f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefcd497640 1 -- 172.21.15.98:0/620606061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab027a8210 con 0x55ab02b3fc10 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefc7fff640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fefb80a3e40 0x7fefb80a6300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fefb4005fd0 tx=0x7fefb400f360 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefcd497640 1 --2- 172.21.15.98:0/620606061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b42000 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fefc0003fd0 tx=0x7fefc0003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38dd9c27d6f85931 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefb2ffd640 1 -- 172.21.15.98:0/620606061 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc00dc070 con 0x55ab02b3fc10 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefb2ffd640 1 -- 172.21.15.98:0/620606061 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefc0036050 con 0x55ab02b3fc10 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.032+0000 7fefb2ffd640 1 -- 172.21.15.98:0/620606061 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0004610 con 0x55ab02b3fc10 2024-04-25T17:44:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 -- 172.21.15.98:0/620606061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 msgr2=0x55ab02b42000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 --2- 172.21.15.98:0/620606061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b42000 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fefc0003fd0 tx=0x7fefc0003f40 comp rx=0 tx=0).stop 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 -- 172.21.15.98:0/620606061 shutdown_connections 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 --2- 172.21.15.98:0/620606061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b42000 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 --2- 172.21.15.98:0/620606061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 0x55ab02b3f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 --2- 172.21.15.98:0/620606061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b30500 0x55ab02b308e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 -- 172.21.15.98:0/620606061 >> 172.21.15.98:0/620606061 conn(0x55ab02b2cb30 msgr2=0x55ab02b2cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 -- 172.21.15.98:0/620606061 shutdown_connections 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 -- 172.21.15.98:0/620606061 wait complete. 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 -- start start 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.033+0000 7fefd1c12740 1 Processor -- start 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30500 0x55ab02b3f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b3fc10 0x55ab02b3cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefd1c12740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ab02a95970 con 0x55ab02b30e20 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefd1c12740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ab02a300b0 con 0x55ab02b30500 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefd1c12740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ab02b42600 con 0x55ab02b3fc10 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefd1c12740 10 client.0 _ll_register_callbacks cb 0x55ab02a1dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefc7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefccc96640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30500 0x55ab02b3f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefc7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56362/0 (socket says 172.21.15.98:56362) 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefc7fff640 1 -- 172.21.15.98:0/3030699612 learned_addr learned my addr 172.21.15.98:0/3030699612 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefcd497640 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b3fc10 0x55ab02b3cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefc7fff640 1 -- 172.21.15.98:0/3030699612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b3fc10 msgr2=0x55ab02b3cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefc7fff640 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b3fc10 0x55ab02b3cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefc7fff640 1 -- 172.21.15.98:0/3030699612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30500 msgr2=0x55ab02b3f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefc7fff640 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30500 0x55ab02b3f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefc7fff640 1 -- 172.21.15.98:0/3030699612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab02874620 con 0x55ab02b30e20 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefc7fff640 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3c7f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fefb4012620 tx=0x7fefb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb24ea0dfc7cda9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.034+0000 7fefa7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefa7fff640 1 -- 172.21.15.98:0/3030699612 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb402b060 con 0x55ab02b30e20 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefa7fff640 1 -- 172.21.15.98:0/3030699612 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fefb401d910 con 0x55ab02b30e20 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefd1c12740 1 -- 172.21.15.98:0/3030699612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefc0036050 con 0x55ab02b30e20 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefa7fff640 1 -- 172.21.15.98:0/3030699612 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fefb401db10 con 0x55ab02b30e20 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefd1c12740 10 client.16284 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefd1c12740 1 -- 172.21.15.98:0/3030699612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ab02b4e370 con 0x55ab02b30e20 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefa7fff640 1 -- 172.21.15.98:0/3030699612 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefc0036050 con 0x55ab02b30e20 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefa7fff640 1 -- 172.21.15.98:0/3030699612 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fefb401e430 con 0x55ab02b30e20 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefd1c12740 10 client.16284 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefd1c12740 10 client.16284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefd1c12740 20 client.16284 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefd1c12740 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab02b4f8d0 0x55ab02b51ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefd1c12740 4 client.16284 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.035+0000 7fefd1c12740 1 -- 172.21.15.98:0/3030699612 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab02aa3390 con 0x55ab02b4f8d0 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.036+0000 7fefccc96640 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab02b4f8d0 0x55ab02b51ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.036+0000 7fefa7fff640 10 client.16284 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.036+0000 7fefccc96640 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab02b4f8d0 0x55ab02b51ce0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f7978c061e81d24 server_cookie=34d0c04a16be2cdd in_seq=0 out_seq=0 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefa7fff640 1 -- 172.21.15.98:0/3030699612 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab02aa3390 con 0x55ab02b4f8d0 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefa7fff640 10 client.16284 handle_command_reply: tid=0 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 client.16284 shutdown 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 -- 172.21.15.98:0/3030699612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 msgr2=0x55ab02b3c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3c7f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fefb4012620 tx=0x7fefb4004730 comp rx=0 tx=0).stop 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 -- 172.21.15.98:0/3030699612 shutdown_connections 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab02b4f8d0 0x55ab02b51ce0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b3fc10 0x55ab02b3cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3c7f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 --2- 172.21.15.98:0/3030699612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30500 0x55ab02b3f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefa7fff640 0 client.16284 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 -- 172.21.15.98:0/3030699612 >> 172.21.15.98:0/3030699612 conn(0x55ab02b2cb30 msgr2=0x55ab02b329a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 -- 172.21.15.98:0/3030699612 shutdown_connections 2024-04-25T17:44:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 1 -- 172.21.15.98:0/3030699612 wait complete. 2024-04-25T17:44:51.044 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.037+0000 7fefd1c12740 20 client.16284 trim_cache size 0 max 16384 2024-04-25T17:44:51.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefd1c12740 1 Processor -- start 2024-04-25T17:44:51.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefd1c12740 1 -- start start 2024-04-25T17:44:51.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30500 0x55ab02b37d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b3fc10 0x55ab02b575f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefd1c12740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81ae300 con 0x55ab02b30e20 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefd1c12740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81abb10 con 0x55ab02b30500 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefd1c12740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81a9980 con 0x55ab02b3fc10 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefc7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefc7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56372/0 (socket says 172.21.15.98:56372) 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefc7fff640 1 -- 172.21.15.98:0/1052188931 learned_addr learned my addr 172.21.15.98:0/1052188931 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefcd497640 1 --2- 172.21.15.98:0/1052188931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b3fc10 0x55ab02b575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefc7fff640 1 -- 172.21.15.98:0/1052188931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b3fc10 msgr2=0x55ab02b575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefc7fff640 1 --2- 172.21.15.98:0/1052188931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b3fc10 0x55ab02b575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefc7fff640 1 -- 172.21.15.98:0/1052188931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30500 msgr2=0x55ab02b37d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefccc96640 1 --2- 172.21.15.98:0/1052188931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30500 0x55ab02b37d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefc7fff640 1 --2- 172.21.15.98:0/1052188931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30500 0x55ab02b37d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefc7fff640 1 -- 172.21.15.98:0/1052188931 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab02b52360 con 0x55ab02b30e20 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.052+0000 7fefc7fff640 1 --2- 172.21.15.98:0/1052188931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3f030 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fefb401ea40 tx=0x7fefb400fe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a0bacc6c0e00fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefb27fc640 1 -- 172.21.15.98:0/1052188931 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb401e430 con 0x55ab02b30e20 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefb27fc640 1 -- 172.21.15.98:0/1052188931 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefb401e5d0 con 0x55ab02b30e20 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefb27fc640 1 -- 172.21.15.98:0/1052188931 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb40054f0 con 0x55ab02b30e20 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefd1c12740 1 -- 172.21.15.98:0/1052188931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 msgr2=0x55ab02b3f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefd1c12740 1 --2- 172.21.15.98:0/1052188931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3f030 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fefb401ea40 tx=0x7fefb400fe40 comp rx=0 tx=0).stop 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefd1c12740 1 -- 172.21.15.98:0/1052188931 shutdown_connections 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefd1c12740 1 --2- 172.21.15.98:0/1052188931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b3fc10 0x55ab02b575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefd1c12740 1 --2- 172.21.15.98:0/1052188931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b30e20 0x55ab02b3f030 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefd1c12740 1 --2- 172.21.15.98:0/1052188931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30500 0x55ab02b37d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefd1c12740 1 -- 172.21.15.98:0/1052188931 >> 172.21.15.98:0/1052188931 conn(0x55ab02b2cb30 msgr2=0x55ab02b37670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefd1c12740 1 -- 172.21.15.98:0/1052188931 shutdown_connections 2024-04-25T17:44:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.053+0000 7fefd1c12740 1 -- 172.21.15.98:0/1052188931 wait complete. 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefd1c12740 1 -- start start 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefd1c12740 1 Processor -- start 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b30500 0x55ab02b394a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 0x55ab02b399e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefd1c12740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b39f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefd1c12740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81a9fd0 con 0x55ab02b3fc10 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefd1c12740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81ab440 con 0x55ab02b30e20 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefd1c12740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81aa6a0 con 0x55ab02b30500 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefd1c12740 10 client.0 _ll_register_callbacks cb 0x7fefb400ff10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefcd497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b39f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefcd497640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b39f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56388/0 (socket says 172.21.15.98:56388) 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefcd497640 1 -- 172.21.15.98:0/1374020332 learned_addr learned my addr 172.21.15.98:0/1374020332 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefc7fff640 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 0x55ab02b399e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefccc96640 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b30500 0x55ab02b394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefcd497640 1 -- 172.21.15.98:0/1374020332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b30500 msgr2=0x55ab02b394a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefcd497640 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b30500 0x55ab02b394a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefcd497640 1 -- 172.21.15.98:0/1374020332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 msgr2=0x55ab02b399e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.054+0000 7fefcd497640 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 0x55ab02b399e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefcd497640 1 -- 172.21.15.98:0/1374020332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab02b2c960 con 0x55ab02b3fc10 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefc7fff640 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 0x55ab02b399e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefcd497640 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b39f20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fefc00ef780 tx=0x7fefc00be880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79375458da29b14f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefa7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefa7fff640 1 -- 172.21.15.98:0/1374020332 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc00dc040 con 0x55ab02b3fc10 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefa7fff640 1 -- 172.21.15.98:0/1374020332 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fefc0004610 con 0x55ab02b3fc10 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefd1c12740 1 -- 172.21.15.98:0/1374020332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefb401e5d0 con 0x55ab02b3fc10 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefa7fff640 1 -- 172.21.15.98:0/1374020332 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fefc00ba2e0 con 0x55ab02b3fc10 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefd1c12740 10 client.16290 fetch_fsmap learned FSMap version 38 2024-04-25T17:44:51.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefd1c12740 1 -- 172.21.15.98:0/1374020332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55ab02bb5f20 con 0x55ab02b3fc10 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefa7fff640 1 -- 172.21.15.98:0/1374020332 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefb401e5d0 con 0x55ab02b3fc10 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefa7fff640 1 -- 172.21.15.98:0/1374020332 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fefc00c6820 con 0x55ab02b3fc10 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefd1c12740 10 client.16290 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefd1c12740 10 client.16290 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefd1c12740 20 client.16290 populate_metadata read hostname 'smithi098' 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefd1c12740 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab02bb64b0 0x55ab02bb88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefd1c12740 4 client.16290 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.055+0000 7fefd1c12740 1 -- 172.21.15.98:0/1374020332 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab02b3d0c0 con 0x55ab02bb64b0 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.056+0000 7fefccc96640 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab02bb64b0 0x55ab02bb88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.056+0000 7fefa7fff640 10 client.16290 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:51.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.056+0000 7fefccc96640 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab02bb64b0 0x55ab02bb88d0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=214b185c0d2cc40b server_cookie=20beed62aaaaaadc in_seq=0 out_seq=0 2024-04-25T17:44:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefa7fff640 1 -- 172.21.15.98:0/1374020332 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab02b3d0c0 con 0x55ab02bb64b0 2024-04-25T17:44:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefa7fff640 10 client.16290 handle_command_reply: tid=0 2024-04-25T17:44:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 client.16290 shutdown 2024-04-25T17:44:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 -- 172.21.15.98:0/1374020332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 msgr2=0x55ab02b39f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b39f20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fefc00ef780 tx=0x7fefc00be880 comp rx=0 tx=0).stop 2024-04-25T17:44:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 -- 172.21.15.98:0/1374020332 shutdown_connections 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55ab02bb64b0 0x55ab02bb88d0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ab02b3fc10 0x55ab02b39f20 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ab02b30e20 0x55ab02b399e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefa7fff640 0 client.16290 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 --2- 172.21.15.98:0/1374020332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ab02b30500 0x55ab02b394a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 -- 172.21.15.98:0/1374020332 >> 172.21.15.98:0/1374020332 conn(0x55ab02b2cb30 msgr2=0x55ab02b2bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 -- 172.21.15.98:0/1374020332 shutdown_connections 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 1 -- 172.21.15.98:0/1374020332 wait complete. 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.064+0000 7fefd1c12740 20 client.16290 trim_cache size 0 max 16384 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:44:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:44:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:44:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:44:51.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 -- 172.21.15.98:0/1384598023 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fefb80a3e40 msgr2=0x7fefb80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fefb80a3e40 0x7fefb80a6300 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fefb4005fd0 tx=0x7fefb400f360 comp rx=0 tx=0).stop 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 -- 172.21.15.98:0/1384598023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefc8199fd0 msgr2=0x7fefc81a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefc8199fd0 0x7fefc81a4580 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fefc0005cc0 tx=0x7fefc002f380 comp rx=0 tx=0).stop 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 -- 172.21.15.98:0/1384598023 shutdown_connections 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fefb80a3e40 0x7fefb80a6300 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fefc8199fd0 0x7fefc81a4580 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefc8106170 0x7fefc8199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 --2- 172.21.15.98:0/1384598023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc81010f0 0x7fefc8199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 -- 172.21.15.98:0/1384598023 >> 172.21.15.98:0/1384598023 conn(0x7fefc8077fd0 msgr2=0x7fefc80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 -- 172.21.15.98:0/1384598023 shutdown_connections 2024-04-25T17:44:51.067 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:44:51.065+0000 7fefceefa640 1 -- 172.21.15.98:0/1384598023 wait complete. 2024-04-25T17:44:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 500.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:44:52.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:52 smithi098 ceph-mon[29192]: pgmap v278: 97 pgs: 97 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 880 op/s 2024-04-25T17:44:52.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:52 smithi203 ceph-mon[32834]: pgmap v278: 97 pgs: 97 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 880 op/s 2024-04-25T17:44:52.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:52 smithi173 ceph-mon[32856]: pgmap v278: 97 pgs: 97 active+clean; 3.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.1 MiB/s wr, 880 op/s 2024-04-25T17:44:54.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:54 smithi098 ceph-mon[29192]: pgmap v279: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 880 op/s 2024-04-25T17:44:54.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:54 smithi203 ceph-mon[32834]: pgmap v279: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 880 op/s 2024-04-25T17:44:54.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:54 smithi173 ceph-mon[32856]: pgmap v279: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 MiB/s wr, 880 op/s 2024-04-25T17:44:56.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:56 smithi098 ceph-mon[29192]: pgmap v280: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.1 MiB/s wr, 877 op/s 2024-04-25T17:44:56.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:56 smithi203 ceph-mon[32834]: pgmap v280: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.1 MiB/s wr, 877 op/s 2024-04-25T17:44:56.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:56 smithi173 ceph-mon[32856]: pgmap v280: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.1 MiB/s wr, 877 op/s 2024-04-25T17:44:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 505.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:44:57.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:44:57.529 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:44:58.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:44:58 smithi098 ceph-mon[29192]: pgmap v281: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 882 op/s 2024-04-25T17:44:58.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:44:58 smithi203 ceph-mon[32834]: pgmap v281: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 882 op/s 2024-04-25T17:44:58.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:44:58 smithi173 ceph-mon[32856]: pgmap v281: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 882 op/s 2024-04-25T17:44:59.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.493+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3815914958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c102d50 msgr2=0x7f3f9c1065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:59.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.493+0000 7f3fa19e1700 1 --2- 172.21.15.98:0/3815914958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c102d50 0x7f3f9c1065e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3f840099c0 tx=0x7f3f84009cd0 comp rx=0 tx=0).stop 2024-04-25T17:44:59.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.493+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3815914958 shutdown_connections 2024-04-25T17:44:59.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.493+0000 7f3fa19e1700 1 --2- 172.21.15.98:0/3815914958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f9c1023a0 0x7f3f9c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.493+0000 7f3fa19e1700 1 --2- 172.21.15.98:0/3815914958 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f9c106cc0 0x7f3f9c10e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.493+0000 7f3fa19e1700 1 --2- 172.21.15.98:0/3815914958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c102d50 0x7f3f9c1065e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.493+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3815914958 >> 172.21.15.98:0/3815914958 conn(0x7f3f9c077fb0 msgr2=0x7f3f9c0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.493+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3815914958 shutdown_connections 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.493+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3815914958 wait complete. 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3fa19e1700 1 Processor -- start 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3fa19e1700 1 -- start start 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3fa19e1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c1023a0 0x7f3f9c199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3fa19e1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f9c102d50 0x7f3f9c19a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3fa19e1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f9c106cc0 0x7f3f9c1a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3fa19e1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c111bc0 con 0x7f3f9c1023a0 2024-04-25T17:44:59.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3fa19e1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c111a40 con 0x7f3f9c102d50 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3fa19e1700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c111d40 con 0x7f3f9c106cc0 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3f9affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c1023a0 0x7f3f9c199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3f9affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c1023a0 0x7f3f9c199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58204/0 (socket says 172.21.15.98:58204) 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.494+0000 7f3f9affd700 1 -- 172.21.15.98:0/3715065495 learned_addr learned my addr 172.21.15.98:0/3715065495 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3f9a7fc700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f9c102d50 0x7f3f9c19a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3f9b7fe700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f9c106cc0 0x7f3f9c1a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3f9affd700 1 -- 172.21.15.98:0/3715065495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f9c106cc0 msgr2=0x7f3f9c1a5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3f9affd700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f9c106cc0 0x7f3f9c1a5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3f9affd700 1 -- 172.21.15.98:0/3715065495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f9c102d50 msgr2=0x7f3f9c19a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3f9affd700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f9c102d50 0x7f3f9c19a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3f9affd700 1 -- 172.21.15.98:0/3715065495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9c1a5580 con 0x7f3f9c1023a0 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3f9a7fc700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f9c102d50 0x7f3f9c19a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3f9affd700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c1023a0 0x7f3f9c199fc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3f94007e20 tx=0x7f3f9400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:59.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3fa09df700 1 -- 172.21.15.98:0/3715065495 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f94016020 con 0x7f3f9c1023a0 2024-04-25T17:44:59.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3715065495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9c1a57b0 con 0x7f3f9c1023a0 2024-04-25T17:44:59.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3fa09df700 1 -- 172.21.15.98:0/3715065495 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f9400ac90 con 0x7f3f9c1023a0 2024-04-25T17:44:59.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3fa09df700 1 -- 172.21.15.98:0/3715065495 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f940115e0 con 0x7f3f9c1023a0 2024-04-25T17:44:59.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.495+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3715065495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f9c1a5d10 con 0x7f3f9c1023a0 2024-04-25T17:44:59.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.496+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3715065495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f9c069730 con 0x7f3f9c1023a0 2024-04-25T17:44:59.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.498+0000 7f3fa09df700 1 -- 172.21.15.98:0/3715065495 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3f94019020 con 0x7f3f9c1023a0 2024-04-25T17:44:59.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.499+0000 7f3fa09df700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3f880a3db0 0x7f3f880a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:44:59.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.499+0000 7f3f9a7fc700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3f880a3db0 0x7f3f880a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:44:59.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.499+0000 7f3fa09df700 1 -- 172.21.15.98:0/3715065495 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f3f940d07c0 con 0x7f3f9c1023a0 2024-04-25T17:44:59.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.499+0000 7f3f9a7fc700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3f880a3db0 0x7f3f880a6270 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3f84006010 tx=0x7f3f84009630 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:44:59.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.502+0000 7f3fa09df700 1 -- 172.21.15.98:0/3715065495 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f94020070 con 0x7f3f9c1023a0 2024-04-25T17:44:59.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.756+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3715065495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f9c103130 con 0x7f3f9c1023a0 2024-04-25T17:44:59.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.757+0000 7f3fa09df700 1 -- 172.21.15.98:0/3715065495 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11883 (secure 0 0 0) 0x7f3f9409cf70 con 0x7f3f9c1023a0 2024-04-25T17:44:59.757 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:44:59.757 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:42:52.734463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:44:59.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3715065495 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3f880a3db0 msgr2=0x7f3f880a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:59.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3f880a3db0 0x7f3f880a6270 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3f84006010 tx=0x7f3f84009630 comp rx=0 tx=0).stop 2024-04-25T17:44:59.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3715065495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c1023a0 msgr2=0x7f3f9c199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:44:59.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c1023a0 0x7f3f9c199fc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3f94007e20 tx=0x7f3f9400a340 comp rx=0 tx=0).stop 2024-04-25T17:44:59.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3715065495 shutdown_connections 2024-04-25T17:44:59.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3f880a3db0 0x7f3f880a6270 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:59.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f9c106cc0 0x7f3f9c1a5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:59.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f9c1023a0 0x7f3f9c199fc0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:59.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 --2- 172.21.15.98:0/3715065495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f9c102d50 0x7f3f9c19a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:44:59.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3715065495 >> 172.21.15.98:0/3715065495 conn(0x7f3f9c077fb0 msgr2=0x7f3f9c0ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:44:59.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3715065495 shutdown_connections 2024-04-25T17:44:59.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:44:59.759+0000 7f3fa19e1700 1 -- 172.21.15.98:0/3715065495 wait complete. 2024-04-25T17:44:59.760 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 38 2024-04-25T17:45:00.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:00 smithi098 ceph-mon[29192]: pgmap v282: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-25T17:45:00.483 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:00 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3715065495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:45:00.483 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:00 smithi203 ceph-mon[32834]: pgmap v282: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-25T17:45:00.483 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:00 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3715065495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:45:00.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:00 smithi173 ceph-mon[32856]: pgmap v282: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-25T17:45:00.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:00 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3715065495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:45:00.616 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id a4035ce8 --release --await' 2024-04-25T17:45:00.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id a4035ce8 --release --await 2024-04-25T17:45:00.861 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:45:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 510.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:45:02.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:02 smithi098 ceph-mon[29192]: pgmap v283: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-04-25T17:45:02.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:02 smithi203 ceph-mon[32834]: pgmap v283: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-04-25T17:45:02.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:02 smithi173 ceph-mon[32856]: pgmap v283: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-04-25T17:45:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 -- 172.21.15.98:0/2739157385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824006bb30 msgr2=0x7f824010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 --2- 172.21.15.98:0/2739157385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824006bb30 0x7f824010f900 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f823800b0a0 tx=0x7f823800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 -- 172.21.15.98:0/2739157385 shutdown_connections 2024-04-25T17:45:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 --2- 172.21.15.98:0/2739157385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8240108a30 0x7f8240108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 --2- 172.21.15.98:0/2739157385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824006bb30 0x7f824010f900 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 --2- 172.21.15.98:0/2739157385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f824006b190 0x7f824006b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 -- 172.21.15.98:0/2739157385 >> 172.21.15.98:0/2739157385 conn(0x7f8240078910 msgr2=0x7f8240078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 -- 172.21.15.98:0/2739157385 shutdown_connections 2024-04-25T17:45:02.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 -- 172.21.15.98:0/2739157385 wait complete. 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 Processor -- start 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.602+0000 7f824746a700 1 -- start start 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f824746a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f824006b190 0x7f82401990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f824746a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824006bb30 0x7f8240199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f824746a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8240108a30 0x7f824019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f824746a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8240112bd0 con 0x7f824006bb30 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f824746a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8240112a50 con 0x7f8240108a30 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f824746a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8240112d50 con 0x7f824006b190 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8244a05700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824006bb30 0x7f8240199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8244a05700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824006bb30 0x7f8240199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58232/0 (socket says 172.21.15.98:58232) 2024-04-25T17:45:02.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8244a05700 1 -- 172.21.15.98:0/1841156195 learned_addr learned my addr 172.21.15.98:0/1841156195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8245206700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f824006b190 0x7f82401990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8245a07700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8240108a30 0x7f824019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8244a05700 1 -- 172.21.15.98:0/1841156195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f824006b190 msgr2=0x7f82401990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8244a05700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f824006b190 0x7f82401990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8244a05700 1 -- 172.21.15.98:0/1841156195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8240108a30 msgr2=0x7f824019d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8244a05700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8240108a30 0x7f824019d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8244a05700 1 -- 172.21.15.98:0/1841156195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824019e0d0 con 0x7f824006bb30 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f8244a05700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824006bb30 0x7f8240199600 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f823000d950 tx=0x7f823000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f82367fc700 1 -- 172.21.15.98:0/1841156195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230016070 con 0x7f824006bb30 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f82367fc700 1 -- 172.21.15.98:0/1841156195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8230010460 con 0x7f824006bb30 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f824746a700 1 -- 172.21.15.98:0/1841156195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f824019e3c0 con 0x7f824006bb30 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f82367fc700 1 -- 172.21.15.98:0/1841156195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823000f620 con 0x7f824006bb30 2024-04-25T17:45:02.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.603+0000 7f824746a700 1 -- 172.21.15.98:0/1841156195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82401a5c40 con 0x7f824006bb30 2024-04-25T17:45:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.605+0000 7f82367fc700 1 -- 172.21.15.98:0/1841156195 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8230019070 con 0x7f824006bb30 2024-04-25T17:45:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.606+0000 7f82367fc700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f822c0a3db0 0x7f822c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.606+0000 7f8245206700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f822c0a3db0 0x7f822c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.606+0000 7f82367fc700 1 -- 172.21.15.98:0/1841156195 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f82300d0480 con 0x7f824006bb30 2024-04-25T17:45:02.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.606+0000 7f8245206700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f822c0a3db0 0x7f822c0a6270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f823c00bbb0 tx=0x7f823c00b590 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.607+0000 7f8251cf3b80 1 Processor -- start 2024-04-25T17:45:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.607+0000 7f8251cf3b80 1 -- start start 2024-04-25T17:45:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a02faa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a0309d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 0x5652a030c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5652a024dd90 con 0x5652a02fa650 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5652a01f4a30 con 0x5652a030a280 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5652a0249d50 con 0x5652a02faf70 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8245206700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a02faa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8245206700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a02faa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58240/0 (socket says 172.21.15.98:58240) 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8245206700 1 -- 172.21.15.98:0/2999672569 learned_addr learned my addr 172.21.15.98:0/2999672569 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8244a05700 1 --2- 172.21.15.98:0/2999672569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a0309d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8245a07700 1 --2- 172.21.15.98:0/2999672569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 0x5652a030c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8245206700 1 -- 172.21.15.98:0/2999672569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 msgr2=0x5652a0309d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8245206700 1 --2- 172.21.15.98:0/2999672569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a0309d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8245206700 1 -- 172.21.15.98:0/2999672569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 msgr2=0x5652a030c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8245206700 1 --2- 172.21.15.98:0/2999672569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 0x5652a030c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8245206700 1 -- 172.21.15.98:0/2999672569 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652a01b8840 con 0x5652a02fa650 2024-04-25T17:45:02.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f8245206700 1 --2- 172.21.15.98:0/2999672569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a02faa30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f823c00fcf0 tx=0x7f823c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b31812fb16d529c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:02.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f822b7fe700 1 -- 172.21.15.98:0/2999672569 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c022070 con 0x5652a02fa650 2024-04-25T17:45:02.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f822b7fe700 1 -- 172.21.15.98:0/2999672569 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f823c011c40 con 0x5652a02fa650 2024-04-25T17:45:02.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.608+0000 7f822b7fe700 1 -- 172.21.15.98:0/2999672569 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c01d610 con 0x5652a02fa650 2024-04-25T17:45:02.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.609+0000 7f8251cf3b80 1 -- 172.21.15.98:0/2999672569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 msgr2=0x5652a02faa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:02.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.609+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/2999672569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a02faa30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f823c00fcf0 tx=0x7f823c012000 comp rx=0 tx=0).stop 2024-04-25T17:45:02.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.609+0000 7f8251cf3b80 1 -- 172.21.15.98:0/2999672569 shutdown_connections 2024-04-25T17:45:02.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.609+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/2999672569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 0x5652a030c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.609+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/2999672569 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a02faa30 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.609+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/2999672569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a0309d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.609+0000 7f8251cf3b80 1 -- 172.21.15.98:0/2999672569 >> 172.21.15.98:0/2999672569 conn(0x5652a0227990 msgr2=0x5652a02f6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.609+0000 7f8251cf3b80 1 -- 172.21.15.98:0/2999672569 shutdown_connections 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.609+0000 7f8251cf3b80 1 -- 172.21.15.98:0/2999672569 wait complete. 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.609+0000 7f8251cf3b80 1 -- start start 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8251cf3b80 1 Processor -- start 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a0309a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a0306d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 0x5652a0307250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5652a030cd10 con 0x5652a02fa650 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5652a02076b0 con 0x5652a030a280 2024-04-25T17:45:02.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5652a030ce90 con 0x5652a02faf70 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8251cf3b80 10 client.0 _ll_register_callbacks cb 0x5652a01e6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8245206700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a0309a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8245206700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a0309a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58254/0 (socket says 172.21.15.98:58254) 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8245206700 1 -- 172.21.15.98:0/1529077238 learned_addr learned my addr 172.21.15.98:0/1529077238 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8244a05700 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a0306d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8245a07700 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 0x5652a0307250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8245206700 1 -- 172.21.15.98:0/1529077238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 msgr2=0x5652a0306d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8245206700 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a0306d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8245206700 1 -- 172.21.15.98:0/1529077238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 msgr2=0x5652a0307250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8245206700 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 0x5652a0307250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8245206700 1 -- 172.21.15.98:0/1529077238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652a019b9d0 con 0x5652a02fa650 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.610+0000 7f8245a07700 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 0x5652a0307250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8245206700 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a0309a10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f823c00f6f0 tx=0x7f823c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e2ef0d0bf08b02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8228ff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8228ff9700 1 -- 172.21.15.98:0/1529077238 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c022050 con 0x5652a02fa650 2024-04-25T17:45:02.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8228ff9700 1 -- 172.21.15.98:0/1529077238 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f823c01e8c0 con 0x5652a02fa650 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1529077238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f823c011c40 con 0x5652a02fa650 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8228ff9700 1 -- 172.21.15.98:0/1529077238 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f823c01eac0 con 0x5652a02fa650 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8251cf3b80 10 client.16308 fetch_fsmap learned FSMap version 38 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1529077238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5652a03175e0 con 0x5652a02fa650 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8228ff9700 1 -- 172.21.15.98:0/1529077238 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f823c011c40 con 0x5652a02fa650 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8228ff9700 1 -- 172.21.15.98:0/1529077238 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f823c03a620 con 0x5652a02fa650 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8251cf3b80 10 client.16308 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.611+0000 7f8251cf3b80 10 client.16308 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.612+0000 7f8251cf3b80 20 client.16308 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.612+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5652a0319b60 0x5652a031bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.612+0000 7f8251cf3b80 4 client.16308 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.612+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1529077238 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652a025b580 con 0x5652a0319b60 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.612+0000 7f8244a05700 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5652a0319b60 0x5652a031bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.612+0000 7f8244a05700 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5652a0319b60 0x5652a031bf50 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bc5d5bd0390403e server_cookie=2434c29076d0d7b5 in_seq=0 out_seq=0 2024-04-25T17:45:02.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.612+0000 7f8228ff9700 10 client.16308 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8228ff9700 1 -- 172.21.15.98:0/1529077238 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5652a025b580 con 0x5652a0319b60 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8228ff9700 10 client.16308 handle_command_reply: tid=0 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 client.16308 shutdown 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1529077238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 msgr2=0x5652a0309a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a0309a10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f823c00f6f0 tx=0x7f823c0057b0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1529077238 shutdown_connections 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5652a0319b60 0x5652a031bf50 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a030a280 0x5652a0307250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a02fa650 0x5652a0309a10 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8228ff9700 0 client.16308 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1529077238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a0306d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1529077238 >> 172.21.15.98:0/1529077238 conn(0x5652a0227990 msgr2=0x5652a02fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1529077238 shutdown_connections 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1529077238 wait complete. 2024-04-25T17:45:02.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.613+0000 7f8251cf3b80 20 client.16308 trim_cache size 0 max 16384 2024-04-25T17:45:02.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.632+0000 7f8251cf3b80 1 Processor -- start 2024-04-25T17:45:02.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.632+0000 7f8251cf3b80 1 -- start start 2024-04-25T17:45:02.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02fa650 0x5652a03025a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02faf70 0x5652a0309550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a03215b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f82401adae0 con 0x5652a030a280 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f82401ab2b0 con 0x5652a02faf70 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f82401a9120 con 0x5652a02fa650 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8245a07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a03215b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8245a07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a03215b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58266/0 (socket says 172.21.15.98:58266) 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8245a07700 1 -- 172.21.15.98:0/2609737657 learned_addr learned my addr 172.21.15.98:0/2609737657 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8245206700 1 --2- 172.21.15.98:0/2609737657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02fa650 0x5652a03025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8244a05700 1 --2- 172.21.15.98:0/2609737657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02faf70 0x5652a0309550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8245a07700 1 -- 172.21.15.98:0/2609737657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02fa650 msgr2=0x5652a03025a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8245a07700 1 --2- 172.21.15.98:0/2609737657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02fa650 0x5652a03025a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8245a07700 1 -- 172.21.15.98:0/2609737657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02faf70 msgr2=0x5652a0309550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8245a07700 1 --2- 172.21.15.98:0/2609737657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02faf70 0x5652a0309550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8245a07700 1 -- 172.21.15.98:0/2609737657 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652a02f62a0 con 0x5652a030a280 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8244a05700 1 --2- 172.21.15.98:0/2609737657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02faf70 0x5652a0309550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:45:02.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f8245a07700 1 --2- 172.21.15.98:0/2609737657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a03215b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8238009260 tx=0x7f8238009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a48618a22d66e26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:02.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f822affd700 1 -- 172.21.15.98:0/2609737657 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8238004a50 con 0x5652a030a280 2024-04-25T17:45:02.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f822affd700 1 -- 172.21.15.98:0/2609737657 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8238004bf0 con 0x5652a030a280 2024-04-25T17:45:02.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.633+0000 7f822affd700 1 -- 172.21.15.98:0/2609737657 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82380052a0 con 0x5652a030a280 2024-04-25T17:45:02.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 -- 172.21.15.98:0/2609737657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 msgr2=0x5652a03215b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/2609737657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a03215b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8238009260 tx=0x7f8238009290 comp rx=0 tx=0).stop 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 -- 172.21.15.98:0/2609737657 shutdown_connections 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/2609737657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02fa650 0x5652a03025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/2609737657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a03215b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/2609737657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02faf70 0x5652a0309550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 -- 172.21.15.98:0/2609737657 >> 172.21.15.98:0/2609737657 conn(0x5652a0227990 msgr2=0x5652a0306e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 -- 172.21.15.98:0/2609737657 shutdown_connections 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 -- 172.21.15.98:0/2609737657 wait complete. 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 -- start start 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.634+0000 7f8251cf3b80 1 Processor -- start 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02fa650 0x5652a0312300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a030f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8251cf3b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a030f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f82401a9770 con 0x5652a030a280 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f82401aabe0 con 0x5652a02fa650 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8251cf3b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f82401a9e40 con 0x5652a02faf70 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8251cf3b80 10 client.0 _ll_register_callbacks cb 0x5652a01e6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8245a07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a030f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8245a07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a030f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58270/0 (socket says 172.21.15.98:58270) 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8245206700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02fa650 0x5652a0312300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8245a07700 1 -- 172.21.15.98:0/1826086023 learned_addr learned my addr 172.21.15.98:0/1826086023 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8244a05700 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a030f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8245a07700 1 -- 172.21.15.98:0/1826086023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 msgr2=0x5652a030f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8245a07700 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a030f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8245a07700 1 -- 172.21.15.98:0/1826086023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02fa650 msgr2=0x5652a0312300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8245a07700 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02fa650 0x5652a0312300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8245a07700 1 -- 172.21.15.98:0/1826086023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652a030fb70 con 0x5652a030a280 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8245a07700 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a030f630 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8238009fd0 tx=0x7f82380060c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be87b1bc833aad8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.635+0000 7f8228ff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8228ff9700 1 -- 172.21.15.98:0/1826086023 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8238004a50 con 0x5652a030a280 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8228ff9700 1 -- 172.21.15.98:0/1826086023 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f8238026430 con 0x5652a030a280 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1826086023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8238004bf0 con 0x5652a030a280 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8228ff9700 1 -- 172.21.15.98:0/1826086023 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8238004da0 con 0x5652a030a280 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8251cf3b80 10 client.16314 fetch_fsmap learned FSMap version 38 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1826086023 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5652a0380b30 con 0x5652a030a280 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8228ff9700 1 -- 172.21.15.98:0/1826086023 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8238004bf0 con 0x5652a030a280 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8228ff9700 1 -- 172.21.15.98:0/1826086023 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f8238011070 con 0x5652a030a280 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8251cf3b80 10 client.16314 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8251cf3b80 10 client.16314 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8251cf3b80 20 client.16314 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5652a0319b60 0x5652a03831c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:02.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8251cf3b80 4 client.16314 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a4035ce8", "release": true, "await": true}] 2024-04-25T17:45:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.636+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1826086023 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a4035ce8", "release": true, "await": true}) -- 0x5652a0255620 con 0x5652a0319b60 2024-04-25T17:45:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.637+0000 7f8245206700 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5652a0319b60 0x5652a03831c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.637+0000 7f8228ff9700 10 client.16314 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:02.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.637+0000 7f8245206700 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5652a0319b60 0x5652a03831c0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b052a658c30662c8 server_cookie=24d0b593f13fe1d8 in_seq=0 out_seq=0 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8228ff9700 1 -- 172.21.15.98:0/1826086023 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5652a0255620 con 0x5652a0319b60 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8228ff9700 10 client.16314 handle_command_reply: tid=0 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 client.16314 shutdown 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1826086023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 msgr2=0x5652a030f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a030f630 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8238009fd0 tx=0x7f82380060c0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1826086023 shutdown_connections 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5652a0319b60 0x5652a03831c0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5652a030a280 0x5652a030f630 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8228ff9700 0 client.16314 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5652a02fa650 0x5652a0312300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 --2- 172.21.15.98:0/1826086023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652a02faf70 0x5652a030f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1826086023 >> 172.21.15.98:0/1826086023 conn(0x5652a0227990 msgr2=0x5652a0306540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:02.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1826086023 shutdown_connections 2024-04-25T17:45:02.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 1 -- 172.21.15.98:0/1826086023 wait complete. 2024-04-25T17:45:02.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.639+0000 7f8251cf3b80 20 client.16314 trim_cache size 0 max 16384 2024-04-25T17:45:02.643 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:45:02.643 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 38, 2024-04-25T17:45:02.643 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:45:02.643 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 28, 2024-04-25T17:45:02.643 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:45:02.643 INFO:teuthology.orchestra.run.smithi098.stdout: "a4035ce8": { 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 28, 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 85.3, 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:45:02.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:45:02.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:45:02.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:45:02.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:45:02.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:45:02.645 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 -- 172.21.15.98:0/1841156195 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f822c0a3db0 msgr2=0x7f822c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f822c0a3db0 0x7f822c0a6270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f823c00bbb0 tx=0x7f823c00b590 comp rx=0 tx=0).stop 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 -- 172.21.15.98:0/1841156195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824006bb30 msgr2=0x7f8240199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824006bb30 0x7f8240199600 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f823000d950 tx=0x7f823000dd10 comp rx=0 tx=0).stop 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 -- 172.21.15.98:0/1841156195 shutdown_connections 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f822c0a3db0 0x7f822c0a6270 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f824006b190 0x7f82401990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8240108a30 0x7f824019d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 --2- 172.21.15.98:0/1841156195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f824006bb30 0x7f8240199600 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 -- 172.21.15.98:0/1841156195 >> 172.21.15.98:0/1841156195 conn(0x7f8240078910 msgr2=0x7f82400ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 -- 172.21.15.98:0/1841156195 shutdown_connections 2024-04-25T17:45:02.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:45:02.640+0000 7f824746a700 1 -- 172.21.15.98:0/1841156195 wait complete. 2024-04-25T17:45:03.499 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a4035ce8', seconds elapsed: 39.1 2024-04-25T17:45:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.209+0000 7ff95c493640 1 -- 172.21.15.98:0/441182593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff95410c650 msgr2=0x7ff954108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.209+0000 7ff95c493640 1 --2- 172.21.15.98:0/441182593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff95410c650 0x7ff954108f50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff94c009ca0 tx=0x7ff94c02a8f0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.209+0000 7ff95c493640 1 -- 172.21.15.98:0/441182593 shutdown_connections 2024-04-25T17:45:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.209+0000 7ff95c493640 1 --2- 172.21.15.98:0/441182593 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff954116fe0 0x7ff9541193d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.209+0000 7ff95c493640 1 --2- 172.21.15.98:0/441182593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff954109490 0x7ff9541098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.209+0000 7ff95c493640 1 --2- 172.21.15.98:0/441182593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff95410c650 0x7ff954108f50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.209+0000 7ff95c493640 1 -- 172.21.15.98:0/441182593 >> 172.21.15.98:0/441182593 conn(0x7ff95406d020 msgr2=0x7ff95406d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.212+0000 7ff95c493640 1 -- 172.21.15.98:0/441182593 shutdown_connections 2024-04-25T17:45:04.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.212+0000 7ff95c493640 1 -- 172.21.15.98:0/441182593 wait complete. 2024-04-25T17:45:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.214+0000 7ff95c493640 1 Processor -- start 2024-04-25T17:45:04.213 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.214+0000 7ff95c493640 1 -- start start 2024-04-25T17:45:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.215+0000 7ff95c493640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff954109490 0x7ff9541a3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.215+0000 7ff95c493640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff95410c650 0x7ff9541a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.215+0000 7ff95c493640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff954116fe0 0x7ff9541a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.215+0000 7ff95c493640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff95411cc90 con 0x7ff954116fe0 2024-04-25T17:45:04.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.215+0000 7ff95c493640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff95411cb10 con 0x7ff95410c650 2024-04-25T17:45:04.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.215+0000 7ff95c493640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff95411ce10 con 0x7ff954109490 2024-04-25T17:45:04.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.216+0000 7ff95aa30640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff954116fe0 0x7ff9541a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.216+0000 7ff95aa30640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff954116fe0 0x7ff9541a7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36186/0 (socket says 172.21.15.98:36186) 2024-04-25T17:45:04.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.216+0000 7ff95aa30640 1 -- 172.21.15.98:0/3382286513 learned_addr learned my addr 172.21.15.98:0/3382286513 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.216+0000 7ff95aa30640 1 -- 172.21.15.98:0/3382286513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff954109490 msgr2=0x7ff9541a3170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T17:45:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.216+0000 7ff95aa30640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff954109490 0x7ff9541a3170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.216+0000 7ff95aa30640 1 -- 172.21.15.98:0/3382286513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff95410c650 msgr2=0x7ff9541a36b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T17:45:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.216+0000 7ff95aa30640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff95410c650 0x7ff9541a36b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.216+0000 7ff95aa30640 1 -- 172.21.15.98:0/3382286513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9541a8160 con 0x7ff954116fe0 2024-04-25T17:45:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.217+0000 7ff95aa30640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff954116fe0 0x7ff9541a7a30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff95000b3f0 tx=0x7ff95000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.217+0000 7ff94b7fe640 1 -- 172.21.15.98:0/3382286513 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950015020 con 0x7ff954116fe0 2024-04-25T17:45:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.217+0000 7ff94b7fe640 1 -- 172.21.15.98:0/3382286513 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff950011ce0 con 0x7ff954116fe0 2024-04-25T17:45:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.218+0000 7ff94b7fe640 1 -- 172.21.15.98:0/3382286513 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950010dd0 con 0x7ff954116fe0 2024-04-25T17:45:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.218+0000 7ff95c493640 1 -- 172.21.15.98:0/3382286513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9541a8450 con 0x7ff954116fe0 2024-04-25T17:45:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.218+0000 7ff95c493640 1 -- 172.21.15.98:0/3382286513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9541afce0 con 0x7ff954116fe0 2024-04-25T17:45:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.222+0000 7ff95f1a8740 1 Processor -- start 2024-04-25T17:45:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.223+0000 7ff94b7fe640 1 -- 172.21.15.98:0/3382286513 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff950018020 con 0x7ff954116fe0 2024-04-25T17:45:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.224+0000 7ff95f1a8740 1 -- start start 2024-04-25T17:45:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10620 0x55c496c10a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 0x55c496c1b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c1b800 0x55c496c1dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c496b75970 con 0x55c496c10620 2024-04-25T17:45:04.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c496b124a0 con 0x55c496c10f40 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c496b71930 con 0x55c496c1b800 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff959a2e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10620 0x55c496c10a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff959a2e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10620 0x55c496c10a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36190/0 (socket says 172.21.15.98:36190) 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff959a2e640 1 -- 172.21.15.98:0/1776269322 learned_addr learned my addr 172.21.15.98:0/1776269322 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff959a2e640 1 --2- 172.21.15.98:0/1776269322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c1b800 0x55c496c1dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff94b7fe640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff9400a3f70 0x7ff9400a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95a22f640 1 --2- 172.21.15.98:0/1776269322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 0x55c496c1b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95aa30640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff9400a3f70 0x7ff9400a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95a22f640 1 -- 172.21.15.98:0/1776269322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c1b800 msgr2=0x55c496c1dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95a22f640 1 --2- 172.21.15.98:0/1776269322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c1b800 0x55c496c1dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95a22f640 1 -- 172.21.15.98:0/1776269322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10620 msgr2=0x55c496c10a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95a22f640 1 --2- 172.21.15.98:0/1776269322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10620 0x55c496c10a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.225+0000 7ff95a22f640 1 -- 172.21.15.98:0/1776269322 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c496888210 con 0x55c496c10f40 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.226+0000 7ff94b7fe640 1 -- 172.21.15.98:0/3382286513 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7ff9500cfd70 con 0x7ff954116fe0 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.226+0000 7ff95a22f640 1 --2- 172.21.15.98:0/1776269322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 0x55c496c1b2c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff94c02ae60 tx=0x7ff94c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5a7ad58f03001f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.226+0000 7ff95aa30640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff9400a3f70 0x7ff9400a6430 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff95009bb20 tx=0x7ff9500912b0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.226+0000 7ff948bfe640 1 -- 172.21.15.98:0/1776269322 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c031460 con 0x55c496c10f40 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.226+0000 7ff948bfe640 1 -- 172.21.15.98:0/1776269322 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff94c03f070 con 0x55c496c10f40 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.226+0000 7ff948bfe640 1 -- 172.21.15.98:0/1776269322 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c031460 con 0x55c496c10f40 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1776269322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 msgr2=0x55c496c1b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1776269322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 0x55c496c1b2c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff94c02ae60 tx=0x7ff94c005ef0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1776269322 shutdown_connections 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1776269322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c1b800 0x55c496c1dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1776269322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 0x55c496c1b2c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1776269322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10620 0x55c496c10a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1776269322 >> 172.21.15.98:0/1776269322 conn(0x55c496c0cb30 msgr2=0x55c496c0cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:04.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1776269322 shutdown_connections 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1776269322 wait complete. 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 -- start start 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.227+0000 7ff95f1a8740 1 Processor -- start 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10620 0x55c496c1b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 0x55c496c18460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c1b800 0x55c496c189c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55c496c1e3d0 con 0x55c496c1b800 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55c496b100b0 con 0x55c496c10f40 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c496c1e550 con 0x55c496c10620 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95f1a8740 10 client.0 _ll_register_callbacks cb 0x55c496afdd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff959a2e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10620 0x55c496c1b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff959a2e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10620 0x55c496c1b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:58840/0 (socket says 172.21.15.98:58840) 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff959a2e640 1 -- 172.21.15.98:0/2288629089 learned_addr learned my addr 172.21.15.98:0/2288629089 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:04.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95a22f640 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 0x55c496c18460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95a22f640 1 -- 172.21.15.98:0/2288629089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10620 msgr2=0x55c496c1b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95a22f640 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10620 0x55c496c1b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95a22f640 1 -- 172.21.15.98:0/2288629089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c1b800 msgr2=0x55c496c189c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95a22f640 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c1b800 0x55c496c189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.228+0000 7ff95a22f640 1 -- 172.21.15.98:0/2288629089 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c496954620 con 0x55c496c10f40 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.229+0000 7ff93dffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.229+0000 7ff95a22f640 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 0x55c496c18460 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff94c02aee0 tx=0x7ff94c0093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0baa723197f022e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.229+0000 7ff93dffb640 1 -- 172.21.15.98:0/2288629089 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c031460 con 0x55c496c10f40 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.229+0000 7ff95f1a8740 1 -- 172.21.15.98:0/2288629089 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff94c03f070 con 0x55c496c10f40 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.229+0000 7ff93dffb640 1 -- 172.21.15.98:0/2288629089 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7ff94c031600 con 0x55c496c10f40 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff93dffb640 1 -- 172.21.15.98:0/2288629089 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff94c031800 con 0x55c496c10f40 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff95f1a8740 10 client.25802 fetch_fsmap learned FSMap version 38 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff95f1a8740 1 -- 172.21.15.98:0/2288629089 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c496c2a340 con 0x55c496c10f40 2024-04-25T17:45:04.235 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff93dffb640 1 -- 172.21.15.98:0/2288629089 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff94c03f070 con 0x55c496c10f40 2024-04-25T17:45:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff93dffb640 1 -- 172.21.15.98:0/2288629089 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7ff94c044690 con 0x55c496c10f40 2024-04-25T17:45:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff95f1a8740 10 client.25802 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:45:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff95f1a8740 10 client.25802 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:45:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff95f1a8740 20 client.25802 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55c496c2b690 0x55c496c2da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff95f1a8740 4 client.25802 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:45:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.230+0000 7ff95f1a8740 1 -- 172.21.15.98:0/2288629089 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c496b83390 con 0x55c496c2b690 2024-04-25T17:45:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.231+0000 7ff959a2e640 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55c496c2b690 0x55c496c2da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.231+0000 7ff959a2e640 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55c496c2b690 0x55c496c2da80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c84eb1aeabaf05e5 server_cookie=783e6393aeff485e in_seq=0 out_seq=0 2024-04-25T17:45:04.236 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.231+0000 7ff93dffb640 10 client.25802 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.232+0000 7ff93dffb640 1 -- 172.21.15.98:0/2288629089 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c496b83390 con 0x55c496c2b690 2024-04-25T17:45:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.232+0000 7ff93dffb640 10 client.25802 handle_command_reply: tid=0 2024-04-25T17:45:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.232+0000 7ff95f1a8740 1 client.25802 shutdown 2024-04-25T17:45:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 1 -- 172.21.15.98:0/2288629089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 msgr2=0x55c496c18460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 0x55c496c18460 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff94c02aee0 tx=0x7ff94c0093e0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 1 -- 172.21.15.98:0/2288629089 shutdown_connections 2024-04-25T17:45:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55c496c2b690 0x55c496c2da80 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c1b800 0x55c496c189c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10f40 0x55c496c18460 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/2288629089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10620 0x55c496c1b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 1 -- 172.21.15.98:0/2288629089 >> 172.21.15.98:0/2288629089 conn(0x55c496c0cb30 msgr2=0x55c496c128b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff93dffb640 0 client.25802 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 1 -- 172.21.15.98:0/2288629089 shutdown_connections 2024-04-25T17:45:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 1 -- 172.21.15.98:0/2288629089 wait complete. 2024-04-25T17:45:04.239 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.233+0000 7ff95f1a8740 20 client.25802 trim_cache size 0 max 16384 2024-04-25T17:45:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95f1a8740 1 Processor -- start 2024-04-25T17:45:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95f1a8740 1 -- start start 2024-04-25T17:45:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 0x55c496c27290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10f40 0x55c496c175c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.257 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c1b800 0x55c496c13970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541bafb0 con 0x55c496c10f40 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541b87c0 con 0x55c496c10620 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541b6600 con 0x55c496c1b800 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95a22f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10f40 0x55c496c175c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95a22f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10f40 0x55c496c175c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36212/0 (socket says 172.21.15.98:36212) 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95a22f640 1 -- 172.21.15.98:0/1958407185 learned_addr learned my addr 172.21.15.98:0/1958407185 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff959a2e640 1 --2- 172.21.15.98:0/1958407185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 0x55c496c27290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff959a2e640 1 --2- 172.21.15.98:0/1958407185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c1b800 0x55c496c13970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95a22f640 1 -- 172.21.15.98:0/1958407185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c1b800 msgr2=0x55c496c13970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95a22f640 1 --2- 172.21.15.98:0/1958407185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c1b800 0x55c496c13970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95a22f640 1 -- 172.21.15.98:0/1958407185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 msgr2=0x55c496c27290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95a22f640 1 --2- 172.21.15.98:0/1958407185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 0x55c496c27290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.258 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff95a22f640 1 -- 172.21.15.98:0/1958407185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c496c10d70 con 0x55c496c10f40 2024-04-25T17:45:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.256+0000 7ff959a2e640 1 --2- 172.21.15.98:0/1958407185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 0x55c496c27290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:45:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.257+0000 7ff95a22f640 1 --2- 172.21.15.98:0/1958407185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10f40 0x55c496c175c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff94c002230 tx=0x7ff94c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d3cd6489573445 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.257+0000 7ff93ffff640 1 -- 172.21.15.98:0/1958407185 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c031800 con 0x55c496c10f40 2024-04-25T17:45:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.257+0000 7ff93ffff640 1 -- 172.21.15.98:0/1958407185 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff94c03f040 con 0x55c496c10f40 2024-04-25T17:45:04.259 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.257+0000 7ff93ffff640 1 -- 172.21.15.98:0/1958407185 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c030b30 con 0x55c496c10f40 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.258+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1958407185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10f40 msgr2=0x55c496c175c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.258+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1958407185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10f40 0x55c496c175c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff94c002230 tx=0x7ff94c000f40 comp rx=0 tx=0).stop 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.258+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1958407185 shutdown_connections 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.258+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1958407185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c1b800 0x55c496c13970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.258+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1958407185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c10f40 0x55c496c175c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.258+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1958407185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 0x55c496c27290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.258+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1958407185 >> 172.21.15.98:0/1958407185 conn(0x55c496c0cb30 msgr2=0x55c496c13260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.259+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1958407185 shutdown_connections 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.259+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1958407185 wait complete. 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.259+0000 7ff95f1a8740 1 -- start start 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.259+0000 7ff95f1a8740 1 Processor -- start 2024-04-25T17:45:04.261 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 0x55c496c1e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10f40 0x55c496c1ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff95f1a8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c1b800 0x55c496c898e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541b6cd0 con 0x55c496c1b800 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541b80f0 con 0x55c496c10620 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff95f1a8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff9541b7350 con 0x55c496c10f40 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff95f1a8740 10 client.0 _ll_register_callbacks cb 0x7ff94c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff95a22f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10f40 0x55c496c1ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff95a22f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10f40 0x55c496c1ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:58852/0 (socket says 172.21.15.98:58852) 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff95a22f640 1 -- 172.21.15.98:0/1025125191 learned_addr learned my addr 172.21.15.98:0/1025125191 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff959a2e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 0x55c496c1e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff959a2e640 1 -- 172.21.15.98:0/1025125191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10f40 msgr2=0x55c496c1ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff959a2e640 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10f40 0x55c496c1ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff959a2e640 1 -- 172.21.15.98:0/1025125191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c1b800 msgr2=0x55c496c898e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff959a2e640 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c1b800 0x55c496c898e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff959a2e640 1 -- 172.21.15.98:0/1025125191 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c496c89f20 con 0x55c496c10620 2024-04-25T17:45:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.260+0000 7ff959a2e640 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 0x55c496c1e9d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff944005eb0 tx=0x7ff9440044f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78dea0275c504625 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.261+0000 7ff93dffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:45:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.261+0000 7ff93dffb640 1 -- 172.21.15.98:0/1025125191 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9440035f0 con 0x55c496c10620 2024-04-25T17:45:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.261+0000 7ff93dffb640 1 -- 172.21.15.98:0/1025125191 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7ff944003790 con 0x55c496c10620 2024-04-25T17:45:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.261+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1025125191 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff94c03f040 con 0x55c496c10620 2024-04-25T17:45:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.261+0000 7ff93dffb640 1 -- 172.21.15.98:0/1025125191 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff944003990 con 0x55c496c10620 2024-04-25T17:45:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.262+0000 7ff95f1a8740 10 client.25808 fetch_fsmap learned FSMap version 38 2024-04-25T17:45:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.262+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1025125191 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55c496c91340 con 0x55c496c10620 2024-04-25T17:45:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.262+0000 7ff93dffb640 1 -- 172.21.15.98:0/1025125191 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff94c03f040 con 0x55c496c10620 2024-04-25T17:45:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.263+0000 7ff93dffb640 1 -- 172.21.15.98:0/1025125191 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7ff94400e3e0 con 0x55c496c10620 2024-04-25T17:45:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.263+0000 7ff95f1a8740 10 client.25808 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T17:45:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.263+0000 7ff95f1a8740 10 client.25808 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:45:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.263+0000 7ff95f1a8740 20 client.25808 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.263+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55c496c91a80 0x55c496c93ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.263+0000 7ff95f1a8740 4 client.25808 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:45:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.263+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1025125191 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c496b7d200 con 0x55c496c91a80 2024-04-25T17:45:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.264+0000 7ff95a22f640 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55c496c91a80 0x55c496c93ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.264+0000 7ff95a22f640 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55c496c91a80 0x55c496c93ea0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=212fc77e8ae60b68 server_cookie=533c5ed373faff9d in_seq=0 out_seq=0 2024-04-25T17:45:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.264+0000 7ff93dffb640 10 client.25808 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff93dffb640 1 -- 172.21.15.98:0/1025125191 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c496b7d200 con 0x55c496c91a80 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff93dffb640 10 client.25808 handle_command_reply: tid=0 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 client.25808 shutdown 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1025125191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 msgr2=0x55c496c1e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 0x55c496c1e9d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff944005eb0 tx=0x7ff9440044f0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1025125191 shutdown_connections 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55c496c91a80 0x55c496c93ea0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55c496c1b800 0x55c496c898e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c496c10f40 0x55c496c1ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff93dffb640 0 client.25808 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 --2- 172.21.15.98:0/1025125191 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55c496c10620 0x55c496c1e9d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1025125191 >> 172.21.15.98:0/1025125191 conn(0x55c496c0cb30 msgr2=0x55c496c13140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1025125191 shutdown_connections 2024-04-25T17:45:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.278+0000 7ff95f1a8740 1 -- 172.21.15.98:0/1025125191 wait complete. 2024-04-25T17:45:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95f1a8740 20 client.25808 trim_cache size 0 max 16384 2024-04-25T17:45:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:45:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:45:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:45:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:45:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 -- 172.21.15.98:0/3382286513 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff9400a3f70 msgr2=0x7ff9400a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff9400a3f70 0x7ff9400a6430 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff95009bb20 tx=0x7ff9500912b0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 -- 172.21.15.98:0/3382286513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff954116fe0 msgr2=0x7ff9541a7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff954116fe0 0x7ff9541a7a30 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff95000b3f0 tx=0x7ff95000b8b0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 -- 172.21.15.98:0/3382286513 shutdown_connections 2024-04-25T17:45:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ff9400a3f70 0x7ff9400a6430 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff954116fe0 0x7ff9541a7a30 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff95410c650 0x7ff9541a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 --2- 172.21.15.98:0/3382286513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff954109490 0x7ff9541a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 -- 172.21.15.98:0/3382286513 >> 172.21.15.98:0/3382286513 conn(0x7ff95406d020 msgr2=0x7ff9541173c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 -- 172.21.15.98:0/3382286513 shutdown_connections 2024-04-25T17:45:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:04.279+0000 7ff95c493640 1 -- 172.21.15.98:0/3382286513 wait complete. 2024-04-25T17:45:04.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:04 smithi098 ceph-mon[29192]: pgmap v284: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 18 KiB/s wr, 10 op/s 2024-04-25T17:45:04.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:04 smithi203 ceph-mon[32834]: pgmap v284: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 18 KiB/s wr, 10 op/s 2024-04-25T17:45:04.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:04 smithi173 ceph-mon[32856]: pgmap v284: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 18 KiB/s wr, 10 op/s 2024-04-25T17:45:06.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:06 smithi098 ceph-mon[29192]: pgmap v285: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-04-25T17:45:06.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:06 smithi098 ceph-mon[29192]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T17:45:06.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:06 smithi203 ceph-mon[32834]: pgmap v285: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-04-25T17:45:06.463 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:06 smithi203 ceph-mon[32834]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T17:45:06.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:06 smithi173 ceph-mon[32856]: pgmap v285: 97 pgs: 97 active+clean; 3.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 16 KiB/s wr, 8 op/s 2024-04-25T17:45:06.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:06 smithi173 ceph-mon[32856]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T17:45:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 515.0 s, 1988.8 tps, lat 362.060 ms stddev 3227.567 2024-04-25T17:45:07.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:07 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:45:07.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:07 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:45:07.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:07 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:45:07.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:07 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:45:07.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:07 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:45:07.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:07 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:45:07.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:07 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:45:07.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:07 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:45:07.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:07 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:45:07.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:07 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:45:07.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:07 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:45:07.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:07 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:45:08.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:08 smithi098 ceph-mon[29192]: pgmap v287: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-04-25T17:45:08.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:08 smithi203 ceph-mon[32834]: pgmap v287: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-04-25T17:45:08.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:08 smithi173 ceph-mon[32856]: pgmap v287: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-04-25T17:45:09.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:09 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:09.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:09 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:09.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:09 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:10.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:10 smithi098 ceph-mon[29192]: pgmap v288: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-04-25T17:45:10.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:10 smithi203 ceph-mon[32834]: pgmap v288: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-04-25T17:45:10.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:10 smithi173 ceph-mon[32856]: pgmap v288: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 3.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-04-25T17:45:11.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 520.0 s, 2741.6 tps, lat 36.244 ms stddev 38.019 2024-04-25T17:45:12.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:12 smithi098 ceph-mon[29192]: pgmap v289: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 346 op/s 2024-04-25T17:45:12.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:12 smithi203 ceph-mon[32834]: pgmap v289: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 346 op/s 2024-04-25T17:45:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:12 smithi173 ceph-mon[32856]: pgmap v289: 97 pgs: 97 active+clean; 3.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 3.2 MiB/s wr, 346 op/s 2024-04-25T17:45:14.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:14 smithi203 ceph-mon[32834]: pgmap v290: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 5.3 MiB/s wr, 574 op/s 2024-04-25T17:45:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:14 smithi173 ceph-mon[32856]: pgmap v290: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 5.3 MiB/s wr, 574 op/s 2024-04-25T17:45:14.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:14 smithi098 ceph-mon[29192]: pgmap v290: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 5.3 MiB/s wr, 574 op/s 2024-04-25T17:45:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:16 smithi098 ceph-mon[29192]: pgmap v291: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 5.3 MiB/s wr, 574 op/s 2024-04-25T17:45:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:16 smithi098 ceph-mon[29192]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T17:45:16.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:16 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:16 smithi203 ceph-mon[32834]: pgmap v291: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 5.3 MiB/s wr, 574 op/s 2024-04-25T17:45:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:16 smithi203 ceph-mon[32834]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T17:45:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:16 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:16 smithi173 ceph-mon[32856]: pgmap v291: 97 pgs: 97 active+clean; 3.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 5.3 MiB/s wr, 574 op/s 2024-04-25T17:45:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:16 smithi173 ceph-mon[32856]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T17:45:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:16 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:16.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 525.0 s, 2781.6 tps, lat 36.057 ms stddev 39.074 2024-04-25T17:45:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.405+0000 7f060c74f640 1 -- 172.21.15.98:0/2709096378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0604107070 msgr2=0x7f0604107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.405+0000 7f060c74f640 1 --2- 172.21.15.98:0/2709096378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0604107070 0x7f0604107450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0600009900 tx=0x7f060002a410 comp rx=0 tx=0).stop 2024-04-25T17:45:17.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.405+0000 7f060c74f640 1 -- 172.21.15.98:0/2709096378 shutdown_connections 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.405+0000 7f060c74f640 1 --2- 172.21.15.98:0/2709096378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06041009e0 0x7f060410ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.405+0000 7f060c74f640 1 --2- 172.21.15.98:0/2709096378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0604100020 0x7f06041004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.405+0000 7f060c74f640 1 --2- 172.21.15.98:0/2709096378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0604107070 0x7f0604107450 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.405+0000 7f060c74f640 1 -- 172.21.15.98:0/2709096378 >> 172.21.15.98:0/2709096378 conn(0x7f0604076f50 msgr2=0x7f06040fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.405+0000 7f060c74f640 1 -- 172.21.15.98:0/2709096378 shutdown_connections 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.405+0000 7f060c74f640 1 -- 172.21.15.98:0/2709096378 wait complete. 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060c74f640 1 Processor -- start 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060c74f640 1 -- start start 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060c74f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604100020 0x7f06041964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060c74f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06041009e0 0x7f0604196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060c74f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0604107070 0x7f060419ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060c74f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0604110000 con 0x7f06041009e0 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060c74f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f060410fe80 con 0x7f0604107070 2024-04-25T17:45:17.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060c74f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0604110180 con 0x7f0604100020 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f0609cea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06041009e0 0x7f0604196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f0609cea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06041009e0 0x7f0604196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48730/0 (socket says 172.21.15.98:48730) 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f0609cea640 1 -- 172.21.15.98:0/637749397 learned_addr learned my addr 172.21.15.98:0/637749397 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060acec640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0604107070 0x7f060419ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060a4eb640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604100020 0x7f06041964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060acec640 1 -- 172.21.15.98:0/637749397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604100020 msgr2=0x7f06041964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060acec640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604100020 0x7f06041964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060acec640 1 -- 172.21.15.98:0/637749397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06041009e0 msgr2=0x7f0604196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060acec640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06041009e0 0x7f0604196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f060acec640 1 -- 172.21.15.98:0/637749397 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060419b480 con 0x7f0604107070 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.406+0000 7f0609cea640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06041009e0 0x7f0604196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.407+0000 7f060acec640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0604107070 0x7f060419ad50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00b500 tx=0x7f05fc00b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.407+0000 7f05fb7fe640 1 -- 172.21.15.98:0/637749397 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc004b20 con 0x7f0604107070 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.407+0000 7f060c74f640 1 -- 172.21.15.98:0/637749397 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f060419b770 con 0x7f0604107070 2024-04-25T17:45:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.407+0000 7f05fb7fe640 1 -- 172.21.15.98:0/637749397 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05fc003e20 con 0x7f0604107070 2024-04-25T17:45:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.407+0000 7f05fb7fe640 1 -- 172.21.15.98:0/637749397 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc0046f0 con 0x7f0604107070 2024-04-25T17:45:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.407+0000 7f060c74f640 1 -- 172.21.15.98:0/637749397 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06041a3000 con 0x7f0604107070 2024-04-25T17:45:17.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.410+0000 7f060f46d740 1 Processor -- start 2024-04-25T17:45:17.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.411+0000 7f05fb7fe640 1 -- 172.21.15.98:0/637749397 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f05fc004000 con 0x7f0604107070 2024-04-25T17:45:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060f46d740 1 -- start start 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060f46d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704cba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060f46d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 0x5621704d62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060f46d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704d8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060f46d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562170430970 con 0x5621704d6800 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060f46d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5621703cd4a0 con 0x5621704cb620 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060f46d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56217042c930 con 0x5621704cbf40 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f0609cea640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f0609cea640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704cba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56624/0 (socket says 172.21.15.98:56624) 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060a4eb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 0x5621704d62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f0609cea640 1 -- 172.21.15.98:0/802209973 learned_addr learned my addr 172.21.15.98:0/802209973 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f0609cea640 1 --2- 172.21.15.98:0/802209973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704d8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f05fb7fe640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f05f00a3dc0 0x7f05f00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060acec640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f05f00a3dc0 0x7f05f00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060a4eb640 1 -- 172.21.15.98:0/802209973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 msgr2=0x5621704cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060a4eb640 1 --2- 172.21.15.98:0/802209973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060a4eb640 1 -- 172.21.15.98:0/802209973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 msgr2=0x5621704d8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060a4eb640 1 --2- 172.21.15.98:0/802209973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704d8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f060a4eb640 1 -- 172.21.15.98:0/802209973 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562170143210 con 0x5621704cbf40 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f0609cea640 1 --2- 172.21.15.98:0/802209973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.412+0000 7f05fb7fe640 1 -- 172.21.15.98:0/637749397 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f05fc0d87a0 con 0x7f0604107070 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.413+0000 7f060a4eb640 1 --2- 172.21.15.98:0/802209973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 0x5621704d62c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0600006010 tx=0x7f0600032af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae1aec601ac03387 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.413+0000 7f060acec640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f05f00a3dc0 0x7f05f00a6280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0ace80 tx=0x7f05fc08f040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.413+0000 7f05f8ff9640 1 -- 172.21.15.98:0/802209973 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600035030 con 0x5621704cbf40 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.413+0000 7f05f8ff9640 1 -- 172.21.15.98:0/802209973 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0600031430 con 0x5621704cbf40 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.413+0000 7f05f8ff9640 1 -- 172.21.15.98:0/802209973 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06000044e0 con 0x5621704cbf40 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.413+0000 7f060f46d740 1 -- 172.21.15.98:0/802209973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 msgr2=0x5621704d62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.413+0000 7f060f46d740 1 --2- 172.21.15.98:0/802209973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 0x5621704d62c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0600006010 tx=0x7f0600032af0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.413+0000 7f060f46d740 1 -- 172.21.15.98:0/802209973 shutdown_connections 2024-04-25T17:45:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.414+0000 7f060f46d740 1 --2- 172.21.15.98:0/802209973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704d8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.414+0000 7f060f46d740 1 --2- 172.21.15.98:0/802209973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 0x5621704d62c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.414+0000 7f060f46d740 1 --2- 172.21.15.98:0/802209973 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.414+0000 7f060f46d740 1 -- 172.21.15.98:0/802209973 >> 172.21.15.98:0/802209973 conn(0x5621704c7b30 msgr2=0x5621704c7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.414+0000 7f060f46d740 1 -- 172.21.15.98:0/802209973 shutdown_connections 2024-04-25T17:45:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.414+0000 7f060f46d740 1 -- 172.21.15.98:0/802209973 wait complete. 2024-04-25T17:45:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.414+0000 7f060f46d740 1 -- start start 2024-04-25T17:45:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.414+0000 7f060f46d740 1 Processor -- start 2024-04-25T17:45:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f060f46d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cb620 0x5621704d6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f060f46d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704d3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f060f46d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704d39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f0609cea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cb620 0x5621704d6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f0609cea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cb620 0x5621704d6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:48684/0 (socket says 172.21.15.98:48684) 2024-04-25T17:45:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f0609cea640 1 -- 172.21.15.98:0/318240540 learned_addr learned my addr 172.21.15.98:0/318240540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:17.420 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f060a4eb640 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704d3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5621704d93d0 con 0x5621704d6800 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5621703cb0b0 con 0x5621704cbf40 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5621704d9550 con 0x5621704cb620 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f060f46d740 10 client.0 _ll_register_callbacks cb 0x5621703b8d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f0609cea640 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704d39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f0609cea640 1 -- 172.21.15.98:0/318240540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 msgr2=0x5621704d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f0609cea640 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704d3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f0609cea640 1 -- 172.21.15.98:0/318240540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 msgr2=0x5621704d39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f0609cea640 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704d39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f0609cea640 1 -- 172.21.15.98:0/318240540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56217020f620 con 0x5621704cb620 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.415+0000 7f060a4eb640 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704d3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f0609cea640 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cb620 0x5621704d6150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f05f4005db0 tx=0x7f05f4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4df7d26e5a52b604 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f05edffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f05edffb640 1 -- 172.21.15.98:0/318240540 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4013c60 con 0x5621704cb620 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f05edffb640 1 -- 172.21.15.98:0/318240540 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f05f4013e00 con 0x5621704cb620 2024-04-25T17:45:17.421 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0600031430 con 0x5621704cb620 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f05edffb640 1 -- 172.21.15.98:0/318240540 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f05f4005020 con 0x5621704cb620 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f060f46d740 10 client.25834 fetch_fsmap learned FSMap version 40 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5621704e5370 con 0x5621704cb620 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f05edffb640 1 -- 172.21.15.98:0/318240540 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0600031430 con 0x5621704cb620 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f05edffb640 1 -- 172.21.15.98:0/318240540 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 40) ==== 8346+0+0 (secure 0 0 0) 0x7f05f400f040 con 0x5621704cb620 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f060f46d740 10 client.25834 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f060f46d740 10 client.25834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f060f46d740 20 client.25834 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.416+0000 7f060f46d740 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5621704e67e0 0x5621704e8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.417+0000 7f060f46d740 4 client.25834 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.417+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56217043e390 con 0x5621704e67e0 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.417+0000 7f060a4eb640 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5621704e67e0 0x5621704e8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.417+0000 7f05edffb640 10 client.25834 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.417+0000 7f060a4eb640 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5621704e67e0 0x5621704e8c30 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74abafcb14977199 server_cookie=f7670cc83f581b5e in_seq=0 out_seq=0 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f05edffb640 1 -- 172.21.15.98:0/318240540 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56217043e390 con 0x5621704e67e0 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f05edffb640 10 client.25834 handle_command_reply: tid=0 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 client.25834 shutdown 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cb620 msgr2=0x5621704d6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cb620 0x5621704d6150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f05f4005db0 tx=0x7f05f4004730 comp rx=0 tx=0).stop 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 shutdown_connections 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5621704e67e0 0x5621704e8c30 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704d39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704d3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 --2- 172.21.15.98:0/318240540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cb620 0x5621704d6150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 >> 172.21.15.98:0/318240540 conn(0x5621704c7b30 msgr2=0x5621704cd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f05edffb640 0 client.25834 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 shutdown_connections 2024-04-25T17:45:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 1 -- 172.21.15.98:0/318240540 wait complete. 2024-04-25T17:45:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.418+0000 7f060f46d740 20 client.25834 trim_cache size 0 max 16384 2024-04-25T17:45:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.438+0000 7f060f46d740 1 Processor -- start 2024-04-25T17:45:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.438+0000 7f060f46d740 1 -- start start 2024-04-25T17:45:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.438+0000 7f060f46d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704e2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.438+0000 7f060f46d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 0x5621704c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f060f46d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704ee540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f060f46d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06041ae160 con 0x5621704d6800 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f060f46d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f06041ab9d0 con 0x5621704cb620 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f060f46d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06041a9840 con 0x5621704cbf40 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f060a4eb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 0x5621704c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f060a4eb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 0x5621704c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:48698/0 (socket says 172.21.15.98:48698) 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f060a4eb640 1 -- 172.21.15.98:0/2161691485 learned_addr learned my addr 172.21.15.98:0/2161691485 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f0609cea640 1 --2- 172.21.15.98:0/2161691485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704e2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f0609cea640 1 --2- 172.21.15.98:0/2161691485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f0609cea640 1 -- 172.21.15.98:0/2161691485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 msgr2=0x5621704c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f0609cea640 1 --2- 172.21.15.98:0/2161691485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 0x5621704c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f0609cea640 1 -- 172.21.15.98:0/2161691485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 msgr2=0x5621704ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f0609cea640 1 --2- 172.21.15.98:0/2161691485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704ee540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.439+0000 7f0609cea640 1 -- 172.21.15.98:0/2161691485 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621704c32e0 con 0x5621704cb620 2024-04-25T17:45:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.440+0000 7f0609cea640 1 --2- 172.21.15.98:0/2161691485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704e2290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f05f4013750 tx=0x7f05f40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=683edd95cd9140e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.440+0000 7f05effff640 1 -- 172.21.15.98:0/2161691485 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f402f020 con 0x5621704cb620 2024-04-25T17:45:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.440+0000 7f05effff640 1 -- 172.21.15.98:0/2161691485 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05f4013a60 con 0x5621704cb620 2024-04-25T17:45:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.440+0000 7f05effff640 1 -- 172.21.15.98:0/2161691485 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4005420 con 0x5621704cb620 2024-04-25T17:45:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.441+0000 7f060f46d740 1 -- 172.21.15.98:0/2161691485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 msgr2=0x5621704e2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.441+0000 7f060f46d740 1 --2- 172.21.15.98:0/2161691485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704e2290 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f05f4013750 tx=0x7f05f40046c0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.441+0000 7f060f46d740 1 -- 172.21.15.98:0/2161691485 shutdown_connections 2024-04-25T17:45:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.441+0000 7f060f46d740 1 --2- 172.21.15.98:0/2161691485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704ee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.441+0000 7f060f46d740 1 --2- 172.21.15.98:0/2161691485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704cbf40 0x5621704c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.441+0000 7f060f46d740 1 --2- 172.21.15.98:0/2161691485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cb620 0x5621704e2290 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f05f4013750 tx=0x7f05f40046c0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.441+0000 7f060f46d740 1 -- 172.21.15.98:0/2161691485 >> 172.21.15.98:0/2161691485 conn(0x5621704c7b30 msgr2=0x5621704c6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.441+0000 7f060f46d740 1 -- 172.21.15.98:0/2161691485 shutdown_connections 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.441+0000 7f060f46d740 1 -- 172.21.15.98:0/2161691485 wait complete. 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f060f46d740 1 -- start start 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f060f46d740 1 Processor -- start 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f060f46d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704dea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f060f46d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704dbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f060f46d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704dc2b0 0x5621704dc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f060f46d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06041a9e90 con 0x5621704d6800 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f060f46d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f06041ab300 con 0x5621704cbf40 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f060f46d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06041aa560 con 0x5621704dc2b0 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f060f46d740 10 client.0 _ll_register_callbacks cb 0x7f05f40040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f0609cea640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704dea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.442+0000 7f060a4eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704dbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f0609cea640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704dea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56656/0 (socket says 172.21.15.98:56656) 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f0609cea640 1 -- 172.21.15.98:0/1022315830 learned_addr learned my addr 172.21.15.98:0/1022315830 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f0609cea640 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704dc2b0 0x5621704dc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f0609cea640 1 -- 172.21.15.98:0/1022315830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704dc2b0 msgr2=0x5621704dc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f0609cea640 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704dc2b0 0x5621704dc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f0609cea640 1 -- 172.21.15.98:0/1022315830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 msgr2=0x5621704dbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f0609cea640 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704dbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f0609cea640 1 -- 172.21.15.98:0/1022315830 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621704d23f0 con 0x5621704cbf40 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f0609cea640 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704dea20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f05f4013570 tx=0x7f05f4017040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43f04a2699686eea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f05edffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f05edffb640 1 -- 172.21.15.98:0/1022315830 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f402f070 con 0x5621704cbf40 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f060f46d740 1 -- 172.21.15.98:0/1022315830 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05f4013a60 con 0x5621704cbf40 2024-04-25T17:45:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f05edffb640 1 -- 172.21.15.98:0/1022315830 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f05f4004730 con 0x5621704cbf40 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f05edffb640 1 -- 172.21.15.98:0/1022315830 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f05f4004930 con 0x5621704cbf40 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f060f46d740 10 client.25826 fetch_fsmap learned FSMap version 40 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f060f46d740 1 -- 172.21.15.98:0/1022315830 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56217054ce90 con 0x5621704cbf40 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.443+0000 7f05edffb640 1 -- 172.21.15.98:0/1022315830 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05f4013a60 con 0x5621704cbf40 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.444+0000 7f05edffb640 1 -- 172.21.15.98:0/1022315830 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 40) ==== 8346+0+0 (secure 0 0 0) 0x7f05f4014590 con 0x5621704cbf40 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.444+0000 7f060f46d740 10 client.25826 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.444+0000 7f060f46d740 10 client.25826 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.444+0000 7f060f46d740 20 client.25826 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.444+0000 7f060f46d740 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56217054d3f0 0x56217054f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.445+0000 7f060f46d740 4 client.25826 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.445+0000 7f060f46d740 1 -- 172.21.15.98:0/1022315830 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562170438200 con 0x56217054d3f0 2024-04-25T17:45:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.445+0000 7f060a4eb640 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56217054d3f0 0x56217054f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.445+0000 7f060a4eb640 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56217054d3f0 0x56217054f7e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5439b7e6e1cbbac4 server_cookie=7bb8998a39f008b0 in_seq=0 out_seq=0 2024-04-25T17:45:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.446+0000 7f05edffb640 10 client.25826 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f05edffb640 1 -- 172.21.15.98:0/1022315830 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562170438200 con 0x56217054d3f0 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f05edffb640 10 client.25826 handle_command_reply: tid=0 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f060f46d740 1 client.25826 shutdown 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f060f46d740 1 -- 172.21.15.98:0/1022315830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 msgr2=0x5621704dea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f060f46d740 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704dea20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f05f4013570 tx=0x7f05f4017040 comp rx=0 tx=0).stop 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f060f46d740 1 -- 172.21.15.98:0/1022315830 shutdown_connections 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f060f46d740 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56217054d3f0 0x56217054f7e0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f060f46d740 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5621704dc2b0 0x5621704dc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f060f46d740 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5621704d6800 0x5621704dbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f05edffb640 0 client.25826 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f060f46d740 1 --2- 172.21.15.98:0/1022315830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5621704cbf40 0x5621704dea20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.454+0000 7f060f46d740 1 -- 172.21.15.98:0/1022315830 >> 172.21.15.98:0/1022315830 conn(0x5621704c7b30 msgr2=0x5621704c6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060f46d740 1 -- 172.21.15.98:0/1022315830 shutdown_connections 2024-04-25T17:45:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060f46d740 1 -- 172.21.15.98:0/1022315830 wait complete. 2024-04-25T17:45:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060f46d740 20 client.25826 trim_cache size 0 max 16384 2024-04-25T17:45:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:45:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:45:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:45:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:45:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 -- 172.21.15.98:0/637749397 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f05f00a3dc0 msgr2=0x7f05f00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f05f00a3dc0 0x7f05f00a6280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0ace80 tx=0x7f05fc08f040 comp rx=0 tx=0).stop 2024-04-25T17:45:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 -- 172.21.15.98:0/637749397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0604107070 msgr2=0x7f060419ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0604107070 0x7f060419ad50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00b500 tx=0x7f05fc00b9c0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 -- 172.21.15.98:0/637749397 shutdown_connections 2024-04-25T17:45:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f05f00a3dc0 0x7f05f00a6280 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0604107070 0x7f060419ad50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06041009e0 0x7f0604196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 --2- 172.21.15.98:0/637749397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0604100020 0x7f06041964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 -- 172.21.15.98:0/637749397 >> 172.21.15.98:0/637749397 conn(0x7f0604076f50 msgr2=0x7f06040fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 -- 172.21.15.98:0/637749397 shutdown_connections 2024-04-25T17:45:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:17.455+0000 7f060c74f640 1 -- 172.21.15.98:0/637749397 wait complete. 2024-04-25T17:45:18.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:18 smithi098 ceph-mon[29192]: pgmap v293: 97 pgs: 97 active+clean; 3.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.6 MiB/s wr, 683 op/s 2024-04-25T17:45:18.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:18 smithi203 ceph-mon[32834]: pgmap v293: 97 pgs: 97 active+clean; 3.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.6 MiB/s wr, 683 op/s 2024-04-25T17:45:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:18 smithi173 ceph-mon[32856]: pgmap v293: 97 pgs: 97 active+clean; 3.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.6 MiB/s wr, 683 op/s 2024-04-25T17:45:20.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:20 smithi098 ceph-mon[29192]: pgmap v294: 97 pgs: 97 active+clean; 3.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.6 MiB/s wr, 683 op/s 2024-04-25T17:45:20.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:20 smithi203 ceph-mon[32834]: pgmap v294: 97 pgs: 97 active+clean; 3.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.6 MiB/s wr, 683 op/s 2024-04-25T17:45:20.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:20 smithi173 ceph-mon[32856]: pgmap v294: 97 pgs: 97 active+clean; 3.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.6 MiB/s wr, 683 op/s 2024-04-25T17:45:21.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:21 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:21.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:21 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:21.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:21 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 530.0 s, 3191.8 tps, lat 31.340 ms stddev 34.662 2024-04-25T17:45:22.422 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:22 smithi173 ceph-mon[32856]: pgmap v295: 97 pgs: 97 active+clean; 3.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.1 MiB/s wr, 723 op/s 2024-04-25T17:45:22.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:22 smithi098 ceph-mon[29192]: pgmap v295: 97 pgs: 97 active+clean; 3.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.1 MiB/s wr, 723 op/s 2024-04-25T17:45:22.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:22 smithi203 ceph-mon[32834]: pgmap v295: 97 pgs: 97 active+clean; 3.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.1 MiB/s wr, 723 op/s 2024-04-25T17:45:24.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:24 smithi098 ceph-mon[29192]: pgmap v296: 97 pgs: 97 active+clean; 3.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.0 MiB/s wr, 712 op/s 2024-04-25T17:45:24.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:24 smithi203 ceph-mon[32834]: pgmap v296: 97 pgs: 97 active+clean; 3.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.0 MiB/s wr, 712 op/s 2024-04-25T17:45:24.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:24 smithi173 ceph-mon[32856]: pgmap v296: 97 pgs: 97 active+clean; 3.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.0 MiB/s wr, 712 op/s 2024-04-25T17:45:26.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:26 smithi098 ceph-mon[29192]: pgmap v297: 97 pgs: 97 active+clean; 3.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.0 MiB/s wr, 712 op/s 2024-04-25T17:45:26.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:26 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:26.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:26 smithi203 ceph-mon[32834]: pgmap v297: 97 pgs: 97 active+clean; 3.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.0 MiB/s wr, 712 op/s 2024-04-25T17:45:26.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:26 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:26.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:26 smithi173 ceph-mon[32856]: pgmap v297: 97 pgs: 97 active+clean; 3.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.0 MiB/s wr, 712 op/s 2024-04-25T17:45:26.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:26 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 535.0 s, 3060.8 tps, lat 32.668 ms stddev 35.533 2024-04-25T17:45:28.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:28 smithi098 ceph-mon[29192]: pgmap v298: 97 pgs: 97 active+clean; 3.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 9.1 MiB/s wr, 932 op/s 2024-04-25T17:45:28.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:28 smithi203 ceph-mon[32834]: pgmap v298: 97 pgs: 97 active+clean; 3.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 9.1 MiB/s wr, 932 op/s 2024-04-25T17:45:28.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:28 smithi173 ceph-mon[32856]: pgmap v298: 97 pgs: 97 active+clean; 3.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 9.1 MiB/s wr, 932 op/s 2024-04-25T17:45:30.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:30 smithi098 ceph-mon[29192]: pgmap v299: 97 pgs: 97 active+clean; 3.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.8 MiB/s wr, 594 op/s 2024-04-25T17:45:30.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:30 smithi203 ceph-mon[32834]: pgmap v299: 97 pgs: 97 active+clean; 3.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.8 MiB/s wr, 594 op/s 2024-04-25T17:45:30.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:30 smithi173 ceph-mon[32856]: pgmap v299: 97 pgs: 97 active+clean; 3.4 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 5.8 MiB/s wr, 594 op/s 2024-04-25T17:45:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.574+0000 7fa78dde2640 1 -- 172.21.15.98:0/4210986881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa788106170 msgr2=0x7fa78810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.574+0000 7fa78dde2640 1 --2- 172.21.15.98:0/4210986881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa788106170 0x7fa78810dd40 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa780009ca0 tx=0x7fa78002a8c0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.574+0000 7fa78dde2640 1 -- 172.21.15.98:0/4210986881 shutdown_connections 2024-04-25T17:45:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.574+0000 7fa78dde2640 1 --2- 172.21.15.98:0/4210986881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa788106170 0x7fa78810dd40 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.574+0000 7fa78dde2640 1 --2- 172.21.15.98:0/4210986881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa788101a50 0x7fa788105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.574+0000 7fa78dde2640 1 --2- 172.21.15.98:0/4210986881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7881010a0 0x7fa788101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.574+0000 7fa78dde2640 1 -- 172.21.15.98:0/4210986881 >> 172.21.15.98:0/4210986881 conn(0x7fa7880fc920 msgr2=0x7fa7880fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.574+0000 7fa78dde2640 1 -- 172.21.15.98:0/4210986881 shutdown_connections 2024-04-25T17:45:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.574+0000 7fa78dde2640 1 -- 172.21.15.98:0/4210986881 wait complete. 2024-04-25T17:45:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa78dde2640 1 Processor -- start 2024-04-25T17:45:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa78dde2640 1 -- start start 2024-04-25T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa78dde2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7881010a0 0x7fa7881a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa78dde2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa788106170 0x7fa7881a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa78dde2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa78819ea60 0x7fa78819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa78dde2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7881187e0 con 0x7fa78819ea60 2024-04-25T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa78dde2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa788118660 con 0x7fa7881010a0 2024-04-25T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa78dde2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa788118960 con 0x7fa788106170 2024-04-25T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa78819ea60 0x7fa78819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa786ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa788106170 0x7fa7881a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa78819ea60 0x7fa78819ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34644/0 (socket says 172.21.15.98:34644) 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.575+0000 7fa787fff640 1 -- 172.21.15.98:0/811208553 learned_addr learned my addr 172.21.15.98:0/811208553 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa7877fe640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7881010a0 0x7fa7881a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa786ffd640 1 -- 172.21.15.98:0/811208553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7881010a0 msgr2=0x7fa7881a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa786ffd640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7881010a0 0x7fa7881a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa786ffd640 1 -- 172.21.15.98:0/811208553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa78819ea60 msgr2=0x7fa78819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa786ffd640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa78819ea60 0x7fa78819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa786ffd640 1 -- 172.21.15.98:0/811208553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78819f580 con 0x7fa788106170 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa786ffd640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa788106170 0x7fa7881a4da0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00e9c0 tx=0x7fa77c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa784ff9640 1 -- 172.21.15.98:0/811208553 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c00cc40 con 0x7fa788106170 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa78dde2640 1 -- 172.21.15.98:0/811208553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa78819f870 con 0x7fa788106170 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa784ff9640 1 -- 172.21.15.98:0/811208553 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa77c00cde0 con 0x7fa788106170 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa784ff9640 1 -- 172.21.15.98:0/811208553 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c0047a0 con 0x7fa788106170 2024-04-25T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.576+0000 7fa78dde2640 1 -- 172.21.15.98:0/811208553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7881aba70 con 0x7fa788106170 2024-04-25T17:45:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.578+0000 7fa784ff9640 1 -- 172.21.15.98:0/811208553 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa77c004aa0 con 0x7fa788106170 2024-04-25T17:45:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.579+0000 7fa790bd5740 1 Processor -- start 2024-04-25T17:45:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.580+0000 7fa790bd5740 1 -- start start 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.580+0000 7fa784ff9640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7700a3e10 0x7fa7700a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.580+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 0x55b7ba37da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.580+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3882c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.580+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba388800 0x55b7ba38abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.580+0000 7fa790bd5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b7ba2e2970 con 0x55b7ba37d620 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.580+0000 7fa790bd5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b7ba27f4a0 con 0x55b7ba388800 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.580+0000 7fa790bd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b7ba2de930 con 0x55b7ba37df40 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.580+0000 7fa784ff9640 1 -- 172.21.15.98:0/811208553 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7fa77c0d0690 con 0x7fa788106170 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa786ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3882c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa786ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3882c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:57648/0 (socket says 172.21.15.98:57648) 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa786ffd640 1 -- 172.21.15.98:0/1279536843 learned_addr learned my addr 172.21.15.98:0/1279536843 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa7877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 0x55b7ba37da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa786ffd640 1 -- 172.21.15.98:0/1279536843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba388800 msgr2=0x55b7ba38abf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T17:45:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa786ffd640 1 --2- 172.21.15.98:0/1279536843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba388800 0x55b7ba38abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa786ffd640 1 -- 172.21.15.98:0/1279536843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 msgr2=0x55b7ba37da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa786ffd640 1 --2- 172.21.15.98:0/1279536843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 0x55b7ba37da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa786ffd640 1 -- 172.21.15.98:0/1279536843 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7b9ff5210 con 0x55b7ba37df40 2024-04-25T17:45:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa787fff640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7700a3e10 0x7fa7700a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.581+0000 7fa786ffd640 1 --2- 172.21.15.98:0/1279536843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3882c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa77c098a90 tx=0x7fa77c087110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e781c523d367e65d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa76dffb640 1 -- 172.21.15.98:0/1279536843 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c01c070 con 0x55b7ba37df40 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa76dffb640 1 -- 172.21.15.98:0/1279536843 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa77c09cd30 con 0x55b7ba37df40 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa76dffb640 1 -- 172.21.15.98:0/1279536843 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c0a52a0 con 0x55b7ba37df40 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa790bd5740 1 -- 172.21.15.98:0/1279536843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 msgr2=0x55b7ba3882c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa787fff640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7700a3e10 0x7fa7700a62d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55b7ba37af50 tx=0x7fa780004a40 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1279536843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3882c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa77c098a90 tx=0x7fa77c087110 comp rx=0 tx=0).stop 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa790bd5740 1 -- 172.21.15.98:0/1279536843 shutdown_connections 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1279536843 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba388800 0x55b7ba38abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1279536843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3882c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1279536843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 0x55b7ba37da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa790bd5740 1 -- 172.21.15.98:0/1279536843 >> 172.21.15.98:0/1279536843 conn(0x55b7ba379b30 msgr2=0x55b7ba379f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa790bd5740 1 -- 172.21.15.98:0/1279536843 shutdown_connections 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.582+0000 7fa790bd5740 1 -- 172.21.15.98:0/1279536843 wait complete. 2024-04-25T17:45:30.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.583+0000 7fa790bd5740 1 -- start start 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.583+0000 7fa790bd5740 1 Processor -- start 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.583+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 0x55b7ba3881f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.583+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3854e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.583+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba388800 0x55b7ba385af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.583+0000 7fa790bd5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b7ba39c440 con 0x55b7ba37d620 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.583+0000 7fa790bd5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b7ba27d0b0 con 0x55b7ba388800 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.583+0000 7fa790bd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b7ba39c5c0 con 0x55b7ba37df40 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.583+0000 7fa790bd5740 10 client.0 _ll_register_callbacks cb 0x55b7ba26ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.583+0000 7fa7877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 0x55b7ba3881f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa7877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 0x55b7ba3881f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34656/0 (socket says 172.21.15.98:34656) 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa7877fe640 1 -- 172.21.15.98:0/1535413766 learned_addr learned my addr 172.21.15.98:0/1535413766 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa786ffd640 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3854e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa787fff640 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba388800 0x55b7ba385af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa7877fe640 1 -- 172.21.15.98:0/1535413766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 msgr2=0x55b7ba3854e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:30.588 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa7877fe640 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3854e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa7877fe640 1 -- 172.21.15.98:0/1535413766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba388800 msgr2=0x55b7ba385af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa7877fe640 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba388800 0x55b7ba385af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa7877fe640 1 -- 172.21.15.98:0/1535413766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7ba0c1620 con 0x55b7ba37d620 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa787fff640 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba388800 0x55b7ba385af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa786ffd640 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3854e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa7877fe640 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 0x55b7ba3881f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa778010a50 tx=0x7fa778010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6064822ccbac117b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa766ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa766ffd640 1 -- 172.21.15.98:0/1535413766 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77801e060 con 0x55b7ba37d620 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa766ffd640 1 -- 172.21.15.98:0/1535413766 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7fa77800fbd0 con 0x55b7ba37d620 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa790bd5740 1 -- 172.21.15.98:0/1535413766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa77c09cd30 con 0x55b7ba37d620 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa766ffd640 1 -- 172.21.15.98:0/1535413766 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fa77800fdd0 con 0x55b7ba37d620 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa790bd5740 10 client.16353 fetch_fsmap learned FSMap version 42 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.584+0000 7fa790bd5740 1 -- 172.21.15.98:0/1535413766 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55b7ba3a8450 con 0x55b7ba37d620 2024-04-25T17:45:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa766ffd640 1 -- 172.21.15.98:0/1535413766 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa77c09cd30 con 0x55b7ba37d620 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa766ffd640 1 -- 172.21.15.98:0/1535413766 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 42) ==== 8354+0+0 (secure 0 0 0) 0x7fa778021440 con 0x55b7ba37d620 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa790bd5740 10 client.16353 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa790bd5740 10 client.16353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa790bd5740 20 client.16353 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b7ba3a9910 0x55b7ba3abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa790bd5740 4 client.16353 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa790bd5740 1 -- 172.21.15.98:0/1535413766 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7ba2f0390 con 0x55b7ba3a9910 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa786ffd640 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b7ba3a9910 0x55b7ba3abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa786ffd640 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b7ba3a9910 0x55b7ba3abd00 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecedbaea23618ace server_cookie=27ebe0abbe276052 in_seq=0 out_seq=0 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.585+0000 7fa766ffd640 10 client.16353 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa766ffd640 1 -- 172.21.15.98:0/1535413766 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b7ba2f0390 con 0x55b7ba3a9910 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa766ffd640 10 client.16353 handle_command_reply: tid=0 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 client.16353 shutdown 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 -- 172.21.15.98:0/1535413766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 msgr2=0x55b7ba3881f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 0x55b7ba3881f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa778010a50 tx=0x7fa778010f10 comp rx=0 tx=0).stop 2024-04-25T17:45:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 -- 172.21.15.98:0/1535413766 shutdown_connections 2024-04-25T17:45:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b7ba3a9910 0x55b7ba3abd00 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba388800 0x55b7ba385af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba3854e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1535413766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba37d620 0x55b7ba3881f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 -- 172.21.15.98:0/1535413766 >> 172.21.15.98:0/1535413766 conn(0x55b7ba379b30 msgr2=0x55b7ba37f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa766ffd640 0 client.16353 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 -- 172.21.15.98:0/1535413766 shutdown_connections 2024-04-25T17:45:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 1 -- 172.21.15.98:0/1535413766 wait complete. 2024-04-25T17:45:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.586+0000 7fa790bd5740 20 client.16353 trim_cache size 0 max 16384 2024-04-25T17:45:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa790bd5740 1 Processor -- start 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa790bd5740 1 -- start start 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37d620 0x55b7ba3a5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba378b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 0x55b7ba3b1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa790bd5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7881ae1c0 con 0x55b7ba388800 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa790bd5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa788073ec0 con 0x55b7ba37d620 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa790bd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa78804fce0 con 0x55b7ba37df40 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 0x55b7ba3b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 0x55b7ba3b1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34660/0 (socket says 172.21.15.98:34660) 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa786ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba378b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa787fff640 1 -- 172.21.15.98:0/106216289 learned_addr learned my addr 172.21.15.98:0/106216289 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa7877fe640 1 --2- 172.21.15.98:0/106216289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37d620 0x55b7ba3a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa786ffd640 1 -- 172.21.15.98:0/106216289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37d620 msgr2=0x55b7ba3a5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa786ffd640 1 --2- 172.21.15.98:0/106216289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37d620 0x55b7ba3a5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa786ffd640 1 -- 172.21.15.98:0/106216289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 msgr2=0x55b7ba3b1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa786ffd640 1 --2- 172.21.15.98:0/106216289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 0x55b7ba3b1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.607+0000 7fa786ffd640 1 -- 172.21.15.98:0/106216289 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7ba385e60 con 0x55b7ba37df40 2024-04-25T17:45:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.608+0000 7fa786ffd640 1 --2- 172.21.15.98:0/106216289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba378b30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x55b7ba37e5b0 tx=0x7fa77c09c170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9267af17e09446a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.608+0000 7fa76d7fa640 1 -- 172.21.15.98:0/106216289 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c0b3030 con 0x55b7ba37df40 2024-04-25T17:45:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.608+0000 7fa76d7fa640 1 -- 172.21.15.98:0/106216289 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa77c050e50 con 0x55b7ba37df40 2024-04-25T17:45:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.608+0000 7fa76d7fa640 1 -- 172.21.15.98:0/106216289 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c01c040 con 0x55b7ba37df40 2024-04-25T17:45:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 -- 172.21.15.98:0/106216289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 msgr2=0x55b7ba378b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 --2- 172.21.15.98:0/106216289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba378b30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x55b7ba37e5b0 tx=0x7fa77c09c170 comp rx=0 tx=0).stop 2024-04-25T17:45:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 -- 172.21.15.98:0/106216289 shutdown_connections 2024-04-25T17:45:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 --2- 172.21.15.98:0/106216289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 0x55b7ba3b1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 --2- 172.21.15.98:0/106216289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37df40 0x55b7ba378b30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 --2- 172.21.15.98:0/106216289 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37d620 0x55b7ba3a5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 -- 172.21.15.98:0/106216289 >> 172.21.15.98:0/106216289 conn(0x55b7ba379b30 msgr2=0x55b7ba378420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 -- 172.21.15.98:0/106216289 shutdown_connections 2024-04-25T17:45:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 -- 172.21.15.98:0/106216289 wait complete. 2024-04-25T17:45:30.613 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 -- start start 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.609+0000 7fa790bd5740 1 Processor -- start 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37d620 0x55b7ba39f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37df40 0x55b7ba39f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa790bd5740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 0x55b7ba39fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa790bd5740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa788072380 con 0x55b7ba388800 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa790bd5740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa7880737f0 con 0x55b7ba37df40 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa790bd5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa788072a50 con 0x55b7ba37d620 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa790bd5740 10 client.0 _ll_register_callbacks cb 0x7fa778010660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa786ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37df40 0x55b7ba39f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa7877fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37d620 0x55b7ba39f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa787fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 0x55b7ba39fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa786ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37df40 0x55b7ba39f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:44324/0 (socket says 172.21.15.98:44324) 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa786ffd640 1 -- 172.21.15.98:0/1013092300 learned_addr learned my addr 172.21.15.98:0/1013092300 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:30.614 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa7877fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37d620 0x55b7ba39f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:57666/0 (socket says 172.21.15.98:57666) 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa786ffd640 1 -- 172.21.15.98:0/1013092300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37d620 msgr2=0x55b7ba39f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa786ffd640 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37d620 0x55b7ba39f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa786ffd640 1 -- 172.21.15.98:0/1013092300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 msgr2=0x55b7ba39fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa786ffd640 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 0x55b7ba39fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa786ffd640 1 -- 172.21.15.98:0/1013092300 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7ba40ef90 con 0x55b7ba37df40 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa787fff640 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 0x55b7ba39fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa786ffd640 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37df40 0x55b7ba39f8a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa77c052010 tx=0x7fa77c009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac04d758339b0513 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.610+0000 7fa766ffd640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa766ffd640 1 -- 172.21.15.98:0/1013092300 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c0b3070 con 0x55b7ba37df40 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa766ffd640 1 -- 172.21.15.98:0/1013092300 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7fa77c0509d0 con 0x55b7ba37df40 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa790bd5740 1 -- 172.21.15.98:0/1013092300 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa77c050e50 con 0x55b7ba37df40 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa766ffd640 1 -- 172.21.15.98:0/1013092300 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fa77c01c040 con 0x55b7ba37df40 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa790bd5740 10 client.25838 fetch_fsmap learned FSMap version 42 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa790bd5740 1 -- 172.21.15.98:0/1013092300 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55b7ba4102a0 con 0x55b7ba37df40 2024-04-25T17:45:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa766ffd640 1 -- 172.21.15.98:0/1013092300 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa77c050e50 con 0x55b7ba37df40 2024-04-25T17:45:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa766ffd640 1 -- 172.21.15.98:0/1013092300 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 42) ==== 8354+0+0 (secure 0 0 0) 0x7fa77c0a4ea0 con 0x55b7ba37df40 2024-04-25T17:45:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa790bd5740 10 client.25838 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T17:45:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa790bd5740 10 client.25838 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:45:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa790bd5740 20 client.25838 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b7ba4102f0 0x55b7ba412710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa790bd5740 4 client.25838 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:45:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa790bd5740 1 -- 172.21.15.98:0/1013092300 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b7ba2ea200 con 0x55b7ba4102f0 2024-04-25T17:45:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.611+0000 7fa7877fe640 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b7ba4102f0 0x55b7ba412710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.612+0000 7fa766ffd640 10 client.25838 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:30.616 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.612+0000 7fa7877fe640 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b7ba4102f0 0x55b7ba412710 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc4c5d2521c03dbe server_cookie=bb5f09d84b856f78 in_seq=0 out_seq=0 2024-04-25T17:45:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.676+0000 7fa766ffd640 1 -- 172.21.15.98:0/1013092300 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b7ba2ea200 con 0x55b7ba4102f0 2024-04-25T17:45:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.676+0000 7fa766ffd640 10 client.25838 handle_command_reply: tid=0 2024-04-25T17:45:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.676+0000 7fa790bd5740 1 client.25838 shutdown 2024-04-25T17:45:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.677+0000 7fa790bd5740 1 -- 172.21.15.98:0/1013092300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37df40 msgr2=0x55b7ba39f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:30.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.677+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37df40 0x55b7ba39f8a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa77c052010 tx=0x7fa77c009620 comp rx=0 tx=0).stop 2024-04-25T17:45:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.677+0000 7fa790bd5740 1 -- 172.21.15.98:0/1013092300 shutdown_connections 2024-04-25T17:45:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.677+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b7ba4102f0 0x55b7ba412710 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.677+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b7ba388800 0x55b7ba39fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.677+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b7ba37df40 0x55b7ba39f8a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.677+0000 7fa790bd5740 1 --2- 172.21.15.98:0/1013092300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7ba37d620 0x55b7ba39f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.677+0000 7fa790bd5740 1 -- 172.21.15.98:0/1013092300 >> 172.21.15.98:0/1013092300 conn(0x55b7ba379b30 msgr2=0x55b7ba378cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.678+0000 7fa766ffd640 0 client.25838 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa790bd5740 1 -- 172.21.15.98:0/1013092300 shutdown_connections 2024-04-25T17:45:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa790bd5740 1 -- 172.21.15.98:0/1013092300 wait complete. 2024-04-25T17:45:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa790bd5740 20 client.25838 trim_cache size 0 max 16384 2024-04-25T17:45:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:45:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:45:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:45:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:45:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa78dde2640 1 -- 172.21.15.98:0/811208553 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7700a3e10 msgr2=0x7fa7700a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa78dde2640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7700a3e10 0x7fa7700a62d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55b7ba37af50 tx=0x7fa780004a40 comp rx=0 tx=0).stop 2024-04-25T17:45:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa78dde2640 1 -- 172.21.15.98:0/811208553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa788106170 msgr2=0x7fa7881a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa78dde2640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa788106170 0x7fa7881a4da0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00e9c0 tx=0x7fa77c00edc0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa78dde2640 1 -- 172.21.15.98:0/811208553 shutdown_connections 2024-04-25T17:45:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa78dde2640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa7700a3e10 0x7fa7700a62d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa78dde2640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa78819ea60 0x7fa78819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa78dde2640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa788106170 0x7fa7881a4da0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa78dde2640 1 --2- 172.21.15.98:0/811208553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa7881010a0 0x7fa7881a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.679+0000 7fa78dde2640 1 -- 172.21.15.98:0/811208553 >> 172.21.15.98:0/811208553 conn(0x7fa7880fc920 msgr2=0x7fa7880fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.680+0000 7fa78dde2640 1 -- 172.21.15.98:0/811208553 shutdown_connections 2024-04-25T17:45:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:30.680+0000 7fa78dde2640 1 -- 172.21.15.98:0/811208553 wait complete. 2024-04-25T17:45:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 540.0 s, 2374.6 tps, lat 41.929 ms stddev 53.892 2024-04-25T17:45:32.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:32 smithi098 ceph-mon[29192]: pgmap v300: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.1 MiB/s wr, 629 op/s 2024-04-25T17:45:32.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:32 smithi203 ceph-mon[32834]: pgmap v300: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.1 MiB/s wr, 629 op/s 2024-04-25T17:45:32.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:32 smithi173 ceph-mon[32856]: pgmap v300: 97 pgs: 97 active+clean; 3.4 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.1 MiB/s wr, 629 op/s 2024-04-25T17:45:34.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:34 smithi098 ceph-mon[29192]: pgmap v301: 97 pgs: 97 active+clean; 3.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 7.3 MiB/s wr, 769 op/s 2024-04-25T17:45:34.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:34 smithi203 ceph-mon[32834]: pgmap v301: 97 pgs: 97 active+clean; 3.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 7.3 MiB/s wr, 769 op/s 2024-04-25T17:45:34.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:34 smithi173 ceph-mon[32856]: pgmap v301: 97 pgs: 97 active+clean; 3.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 7.3 MiB/s wr, 769 op/s 2024-04-25T17:45:36.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:36 smithi098 ceph-mon[29192]: pgmap v302: 97 pgs: 97 active+clean; 3.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 5.6 MiB/s wr, 585 op/s 2024-04-25T17:45:36.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:36 smithi203 ceph-mon[32834]: pgmap v302: 97 pgs: 97 active+clean; 3.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 5.6 MiB/s wr, 585 op/s 2024-04-25T17:45:36.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:36 smithi173 ceph-mon[32856]: pgmap v302: 97 pgs: 97 active+clean; 3.4 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 5.6 MiB/s wr, 585 op/s 2024-04-25T17:45:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 545.0 s, 2147.4 tps, lat 46.796 ms stddev 50.663 2024-04-25T17:45:38.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:38 smithi098 ceph-mon[29192]: pgmap v303: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 8.8 MiB/s wr, 903 op/s 2024-04-25T17:45:38.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:38 smithi203 ceph-mon[32834]: pgmap v303: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 8.8 MiB/s wr, 903 op/s 2024-04-25T17:45:38.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:38 smithi173 ceph-mon[32856]: pgmap v303: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 8.8 MiB/s wr, 903 op/s 2024-04-25T17:45:40.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:40 smithi098 ceph-mon[29192]: pgmap v304: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 6.0 MiB/s wr, 613 op/s 2024-04-25T17:45:40.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:40 smithi203 ceph-mon[32834]: pgmap v304: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 6.0 MiB/s wr, 613 op/s 2024-04-25T17:45:40.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:40 smithi173 ceph-mon[32856]: pgmap v304: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 6.0 MiB/s wr, 613 op/s 2024-04-25T17:45:41.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:41 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:41.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:41 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:41.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:41 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 550.0 s, 2779.0 tps, lat 35.898 ms stddev 39.707 2024-04-25T17:45:42.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:42 smithi098 ceph-mon[29192]: pgmap v305: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.9 MiB/s wr, 670 op/s 2024-04-25T17:45:42.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:42 smithi203 ceph-mon[32834]: pgmap v305: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.9 MiB/s wr, 670 op/s 2024-04-25T17:45:42.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:42 smithi173 ceph-mon[32856]: pgmap v305: 97 pgs: 97 active+clean; 3.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.9 MiB/s wr, 670 op/s 2024-04-25T17:45:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.803+0000 7f80e8dcd640 1 -- 172.21.15.98:0/2796870707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80e4100e10 msgr2=0x7f80e410ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.803+0000 7f80e8dcd640 1 --2- 172.21.15.98:0/2796870707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80e4100e10 0x7f80e410ede0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009cd0 tx=0x7f80dc02a8e0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.804+0000 7f80e8dcd640 1 -- 172.21.15.98:0/2796870707 shutdown_connections 2024-04-25T17:45:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.804+0000 7f80e8dcd640 1 --2- 172.21.15.98:0/2796870707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80e4100e10 0x7f80e410ede0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.804+0000 7f80e8dcd640 1 --2- 172.21.15.98:0/2796870707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80e4100450 0x7f80e41008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.804+0000 7f80e8dcd640 1 --2- 172.21.15.98:0/2796870707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80e4106f30 0x7f80e4107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.804+0000 7f80e8dcd640 1 -- 172.21.15.98:0/2796870707 >> 172.21.15.98:0/2796870707 conn(0x7f80e40773e0 msgr2=0x7f80e40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.804+0000 7f80e8dcd640 1 -- 172.21.15.98:0/2796870707 shutdown_connections 2024-04-25T17:45:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.804+0000 7f80e8dcd640 1 -- 172.21.15.98:0/2796870707 wait complete. 2024-04-25T17:45:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.805+0000 7f80e8dcd640 1 Processor -- start 2024-04-25T17:45:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.805+0000 7f80e8dcd640 1 -- start start 2024-04-25T17:45:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.805+0000 7f80e8dcd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80e4100450 0x7f80e419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.805+0000 7f80e8dcd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80e4100e10 0x7f80e419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.805+0000 7f80e8dcd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80e4106f30 0x7f80e41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.805+0000 7f80e8dcd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80e4112310 con 0x7f80e4100e10 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.805+0000 7f80e8dcd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f80e4112190 con 0x7f80e4106f30 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.805+0000 7f80e8dcd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80e4112490 con 0x7f80e4100450 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80e4100e10 0x7f80e419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e2d9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80e4106f30 0x7f80e41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80e4100450 0x7f80e419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e2d9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80e4106f30 0x7f80e41985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40000/0 (socket says 172.21.15.98:40000) 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e2d9d640 1 -- 172.21.15.98:0/787064658 learned_addr learned my addr 172.21.15.98:0/787064658 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e259c640 1 -- 172.21.15.98:0/787064658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80e4106f30 msgr2=0x7f80e41985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e259c640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80e4106f30 0x7f80e41985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e259c640 1 -- 172.21.15.98:0/787064658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80e4100e10 msgr2=0x7f80e419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e259c640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80e4100e10 0x7f80e419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e259c640 1 -- 172.21.15.98:0/787064658 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e4198d30 con 0x7f80e4100450 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e2d9d640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80e4106f30 0x7f80e41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80e259c640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80e4100450 0x7f80e419e400 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f80d400f780 tx=0x7f80d400fc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80d37fe640 1 -- 172.21.15.98:0/787064658 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d4010750 con 0x7f80e4100450 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80d37fe640 1 -- 172.21.15.98:0/787064658 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80d4010da0 con 0x7f80e4100450 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.806+0000 7f80d37fe640 1 -- 172.21.15.98:0/787064658 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d400bd50 con 0x7f80e4100450 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.807+0000 7f80e8dcd640 1 -- 172.21.15.98:0/787064658 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e4198fc0 con 0x7f80e4100450 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.807+0000 7f80e8dcd640 1 -- 172.21.15.98:0/787064658 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e4101560 con 0x7f80e4100450 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.809+0000 7f80d37fe640 1 -- 172.21.15.98:0/787064658 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f80d40108f0 con 0x7f80e4100450 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.810+0000 7f80d37fe640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80cc0a3dc0 0x7f80cc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.810+0000 7f80d37fe640 1 -- 172.21.15.98:0/787064658 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f80d40d38d0 con 0x7f80e4100450 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.810+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80cc0a3dc0 0x7f80cc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.811+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80cc0a3dc0 0x7f80cc0a6280 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f80e41058f0 tx=0x7f80d8005f90 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.812+0000 7f80ebbc3740 1 Processor -- start 2024-04-25T17:45:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.812+0000 7f80ebbc3740 1 -- start start 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.812+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e55ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.812+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55cf40 0x56414e5672c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.812+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 0x56414e569bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.812+0000 7f80ebbc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56414e4c1970 con 0x56414e55cf40 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e55ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e55ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:47662/0 (socket says 172.21.15.98:47662) 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e259c640 1 -- 172.21.15.98:0/2868997459 learned_addr learned my addr 172.21.15.98:0/2868997459 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/2868997459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55cf40 0x56414e5672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e2d9d640 1 --2- 172.21.15.98:0/2868997459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 0x56414e569bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2868997459 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56414e45e4a0 con 0x56414e567800 2024-04-25T17:45:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2868997459 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56414e4bd930 con 0x56414e55c620 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e1d9b640 1 -- 172.21.15.98:0/2868997459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 msgr2=0x56414e55ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/2868997459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e55ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e1d9b640 1 -- 172.21.15.98:0/2868997459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 msgr2=0x56414e569bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/2868997459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 0x56414e569bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e1d9b640 1 -- 172.21.15.98:0/2868997459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56414e1d4210 con 0x56414e55cf40 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.813+0000 7f80e2d9d640 1 --2- 172.21.15.98:0/2868997459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 0x56414e569bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.814+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/2868997459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55cf40 0x56414e5672c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f80e41996f0 tx=0x7f80d8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61b8c449cc15778 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.814+0000 7f80d0ff9640 1 -- 172.21.15.98:0/2868997459 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d8005020 con 0x56414e55cf40 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.815+0000 7f80d0ff9640 1 -- 172.21.15.98:0/2868997459 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80d80047a0 con 0x56414e55cf40 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.815+0000 7f80d0ff9640 1 -- 172.21.15.98:0/2868997459 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d801e680 con 0x56414e55cf40 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.816+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2868997459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55cf40 msgr2=0x56414e5672c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.816+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2868997459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55cf40 0x56414e5672c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f80e41996f0 tx=0x7f80d8014000 comp rx=0 tx=0).stop 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.816+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2868997459 shutdown_connections 2024-04-25T17:45:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.816+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2868997459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 0x56414e569bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.816+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2868997459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55cf40 0x56414e5672c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.816+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2868997459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e55ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.816+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2868997459 >> 172.21.15.98:0/2868997459 conn(0x56414e558b30 msgr2=0x56414e558f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.817+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2868997459 shutdown_connections 2024-04-25T17:45:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.817+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2868997459 wait complete. 2024-04-25T17:45:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.817+0000 7f80ebbc3740 1 -- start start 2024-04-25T17:45:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80ebbc3740 1 Processor -- start 2024-04-25T17:45:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e567230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e55cf40 0x56414e564520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e567800 0x56414e564c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80ebbc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56414e56a430 con 0x56414e567800 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80ebbc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56414e45c0b0 con 0x56414e55cf40 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80e259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e567230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80e1d9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e55cf40 0x56414e564520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80e259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e567230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:47664/0 (socket says 172.21.15.98:47664) 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80e259c640 1 -- 172.21.15.98:0/2924884208 learned_addr learned my addr 172.21.15.98:0/2924884208 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2924884208 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56414e56a5b0 con 0x56414e55c620 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.818+0000 7f80e2d9d640 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e567800 0x56414e564c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80ebbc3740 10 client.0 _ll_register_callbacks cb 0x56414e449d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80e259c640 1 -- 172.21.15.98:0/2924884208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e55cf40 msgr2=0x56414e564520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80e259c640 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e55cf40 0x56414e564520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80e259c640 1 -- 172.21.15.98:0/2924884208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e567800 msgr2=0x56414e564c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80e259c640 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e567800 0x56414e564c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80e259c640 1 -- 172.21.15.98:0/2924884208 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56414e2a0620 con 0x56414e55c620 2024-04-25T17:45:43.822 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80e259c640 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e567230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f80d400dfd0 tx=0x7f80d40047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c734662b658670c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2924884208 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d40a3d30 con 0x56414e55c620 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2924884208 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f80d40a8160 con 0x56414e55c620 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.819+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2924884208 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80d80047a0 con 0x56414e55c620 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.820+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2924884208 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f80d40a85f0 con 0x56414e55c620 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.820+0000 7f80ebbc3740 10 client.25870 fetch_fsmap learned FSMap version 43 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.820+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2924884208 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56414e576460 con 0x56414e55c620 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.821+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2924884208 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80d80047a0 con 0x56414e55c620 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.821+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2924884208 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 43) ==== 8350+0+0 (secure 0 0 0) 0x7f80d408a0c0 con 0x56414e55c620 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.821+0000 7f80ebbc3740 10 client.25870 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.821+0000 7f80ebbc3740 10 client.25870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.821+0000 7f80ebbc3740 20 client.25870 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.821+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56414e577920 0x56414e579d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.821+0000 7f80ebbc3740 4 client.25870 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:45:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.821+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2924884208 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56414e4cf390 con 0x56414e577920 2024-04-25T17:45:43.824 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.821+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56414e577920 0x56414e579d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.822+0000 7f80c9ffb640 10 client.25870 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.822+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56414e577920 0x56414e579d10 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1af7192a3d895f server_cookie=386328330a3c74e1 in_seq=0 out_seq=0 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.823+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2924884208 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56414e4cf390 con 0x56414e577920 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.823+0000 7f80c9ffb640 10 client.25870 handle_command_reply: tid=0 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.823+0000 7f80ebbc3740 1 client.25870 shutdown 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.823+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2924884208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 msgr2=0x56414e567230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.823+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e567230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f80d400dfd0 tx=0x7f80d40047d0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.823+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2924884208 shutdown_connections 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.823+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56414e577920 0x56414e579d10 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.824+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e567800 0x56414e564c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.824+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e55cf40 0x56414e564520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.824+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2924884208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55c620 0x56414e567230 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.824+0000 7f80c9ffb640 0 client.25870 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:43.825 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.824+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2924884208 >> 172.21.15.98:0/2924884208 conn(0x56414e558b30 msgr2=0x56414e55e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.824+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2924884208 shutdown_connections 2024-04-25T17:45:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.824+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2924884208 wait complete. 2024-04-25T17:45:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.824+0000 7f80ebbc3740 20 client.25870 trim_cache size 0 max 16384 2024-04-25T17:45:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80ebbc3740 1 Processor -- start 2024-04-25T17:45:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80ebbc3740 1 -- start start 2024-04-25T17:45:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55c620 0x56414e5732f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e55cf40 0x56414e57f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e567800 0x56414e57fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80ebbc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41ae280 con 0x56414e55c620 2024-04-25T17:45:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80e2d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e567800 0x56414e57fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80e259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55c620 0x56414e5732f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80e2d9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e567800 0x56414e57fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:47674/0 (socket says 172.21.15.98:47674) 2024-04-25T17:45:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80e2d9d640 1 -- 172.21.15.98:0/696132613 learned_addr learned my addr 172.21.15.98:0/696132613 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/696132613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e55cf40 0x56414e57f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.848+0000 7f80ebbc3740 1 -- 172.21.15.98:0/696132613 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41abb20 con 0x56414e55cf40 2024-04-25T17:45:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.849+0000 7f80ebbc3740 1 -- 172.21.15.98:0/696132613 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41a9990 con 0x56414e567800 2024-04-25T17:45:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.849+0000 7f80e2d9d640 1 -- 172.21.15.98:0/696132613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e55cf40 msgr2=0x56414e57f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.849+0000 7f80e2d9d640 1 --2- 172.21.15.98:0/696132613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e55cf40 0x56414e57f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.849+0000 7f80e2d9d640 1 -- 172.21.15.98:0/696132613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55c620 msgr2=0x56414e5732f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.849+0000 7f80e2d9d640 1 --2- 172.21.15.98:0/696132613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55c620 0x56414e5732f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.849+0000 7f80e2d9d640 1 -- 172.21.15.98:0/696132613 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56414e564880 con 0x56414e567800 2024-04-25T17:45:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.849+0000 7f80e2d9d640 1 --2- 172.21.15.98:0/696132613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e567800 0x56414e57fa80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f80dc005cc0 tx=0x7f80dc005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24e2dc9ac079cc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.850+0000 7f80cbfff640 1 -- 172.21.15.98:0/696132613 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc003970 con 0x56414e567800 2024-04-25T17:45:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.850+0000 7f80cbfff640 1 -- 172.21.15.98:0/696132613 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80dc036040 con 0x56414e567800 2024-04-25T17:45:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.850+0000 7f80cbfff640 1 -- 172.21.15.98:0/696132613 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc0305a0 con 0x56414e567800 2024-04-25T17:45:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 -- 172.21.15.98:0/696132613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e567800 msgr2=0x56414e57fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/696132613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e567800 0x56414e57fa80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f80dc005cc0 tx=0x7f80dc005c50 comp rx=0 tx=0).stop 2024-04-25T17:45:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 -- 172.21.15.98:0/696132613 shutdown_connections 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/696132613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e567800 0x56414e57fa80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/696132613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e55cf40 0x56414e57f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/696132613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55c620 0x56414e5732f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 -- 172.21.15.98:0/696132613 >> 172.21.15.98:0/696132613 conn(0x56414e558b30 msgr2=0x56414e557cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 -- 172.21.15.98:0/696132613 shutdown_connections 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 -- 172.21.15.98:0/696132613 wait complete. 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 -- start start 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.851+0000 7f80ebbc3740 1 Processor -- start 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55c620 0x56414e56d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55cf40 0x56414e56dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80ebbc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 0x56414e5dadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80ebbc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41a9fe0 con 0x56414e55c620 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80ebbc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41ab450 con 0x56414e567800 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80ebbc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f80e41aa6b0 con 0x56414e55cf40 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80ebbc3740 10 client.0 _ll_register_callbacks cb 0x7f80d40c0050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80e1d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55cf40 0x56414e56dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80e1d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55cf40 0x56414e56dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:47676/0 (socket says 172.21.15.98:47676) 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80e1d9b640 1 -- 172.21.15.98:0/2803796713 learned_addr learned my addr 172.21.15.98:0/2803796713 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80e2d9d640 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 0x56414e5dadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.852+0000 7f80e259c640 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55c620 0x56414e56d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.853+0000 7f80e1d9b640 1 -- 172.21.15.98:0/2803796713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 msgr2=0x56414e5dadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.853+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 0x56414e5dadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.853+0000 7f80e1d9b640 1 -- 172.21.15.98:0/2803796713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55c620 msgr2=0x56414e56d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.853+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55c620 0x56414e56d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.853+0000 7f80e1d9b640 1 -- 172.21.15.98:0/2803796713 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56414e5db2f0 con 0x56414e55cf40 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.853+0000 7f80e1d9b640 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55cf40 0x56414e56dec0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f80d8004220 tx=0x7f80d8003d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de35d8594949cacb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.853+0000 7f80c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.853+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2803796713 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d803c060 con 0x56414e55cf40 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.854+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2803796713 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7f80d8004830 con 0x56414e55cf40 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.854+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2803796713 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80dc036040 con 0x56414e55cf40 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.854+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2803796713 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f80d8036170 con 0x56414e55cf40 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.854+0000 7f80ebbc3740 10 client.25876 fetch_fsmap learned FSMap version 43 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.854+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2803796713 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56414e5ef650 con 0x56414e55cf40 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.854+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2803796713 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80dc036040 con 0x56414e55cf40 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.854+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2803796713 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 43) ==== 8350+0+0 (secure 0 0 0) 0x7f80d8033920 con 0x56414e55cf40 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.854+0000 7f80ebbc3740 10 client.25876 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.855+0000 7f80ebbc3740 10 client.25876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.855+0000 7f80ebbc3740 20 client.25876 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.855+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56414e5efaa0 0x56414e5f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.855+0000 7f80ebbc3740 4 client.25876 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:45:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.855+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2803796713 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56414e57a330 con 0x56414e5efaa0 2024-04-25T17:45:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.855+0000 7f80e259c640 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56414e5efaa0 0x56414e5f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.855+0000 7f80c9ffb640 10 client.25876 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.855+0000 7f80e259c640 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56414e5efaa0 0x56414e5f1ec0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dac40faf10b68679 server_cookie=d2d74c1d1e324024 in_seq=0 out_seq=0 2024-04-25T17:45:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.868+0000 7f80c9ffb640 1 -- 172.21.15.98:0/2803796713 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56414e57a330 con 0x56414e5efaa0 2024-04-25T17:45:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.868+0000 7f80c9ffb640 10 client.25876 handle_command_reply: tid=0 2024-04-25T17:45:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.868+0000 7f80ebbc3740 1 client.25876 shutdown 2024-04-25T17:45:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.869+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2803796713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55cf40 msgr2=0x56414e56dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.869+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55cf40 0x56414e56dec0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f80d8004220 tx=0x7f80d8003d10 comp rx=0 tx=0).stop 2024-04-25T17:45:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.869+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2803796713 shutdown_connections 2024-04-25T17:45:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.869+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56414e5efaa0 0x56414e5f1ec0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.869+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56414e567800 0x56414e5dadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.869+0000 7f80c9ffb640 0 client.25876 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.869+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56414e55cf40 0x56414e56dec0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.869+0000 7f80ebbc3740 1 --2- 172.21.15.98:0/2803796713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56414e55c620 0x56414e56d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.869+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2803796713 >> 172.21.15.98:0/2803796713 conn(0x56414e558b30 msgr2=0x56414e5722e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.870+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2803796713 shutdown_connections 2024-04-25T17:45:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.870+0000 7f80ebbc3740 1 -- 172.21.15.98:0/2803796713 wait complete. 2024-04-25T17:45:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.870+0000 7f80ebbc3740 20 client.25876 trim_cache size 0 max 16384 2024-04-25T17:45:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:45:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:45:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:45:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:45:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.870+0000 7f80e8dcd640 1 -- 172.21.15.98:0/787064658 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80cc0a3dc0 msgr2=0x7f80cc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.870+0000 7f80e8dcd640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80cc0a3dc0 0x7f80cc0a6280 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f80e41058f0 tx=0x7f80d8005f90 comp rx=0 tx=0).stop 2024-04-25T17:45:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.870+0000 7f80e8dcd640 1 -- 172.21.15.98:0/787064658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80e4100450 msgr2=0x7f80e419e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.870+0000 7f80e8dcd640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80e4100450 0x7f80e419e400 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f80d400f780 tx=0x7f80d400fc40 comp rx=0 tx=0).stop 2024-04-25T17:45:43.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.871+0000 7f80e8dcd640 1 -- 172.21.15.98:0/787064658 shutdown_connections 2024-04-25T17:45:43.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.871+0000 7f80e8dcd640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f80cc0a3dc0 0x7f80cc0a6280 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.871+0000 7f80e8dcd640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80e4106f30 0x7f80e41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.871+0000 7f80e8dcd640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80e4100e10 0x7f80e419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.871+0000 7f80e8dcd640 1 --2- 172.21.15.98:0/787064658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80e4100450 0x7f80e419e400 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:43.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.871+0000 7f80e8dcd640 1 -- 172.21.15.98:0/787064658 >> 172.21.15.98:0/787064658 conn(0x7f80e40773e0 msgr2=0x7f80e4108560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:43.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.872+0000 7f80e8dcd640 1 -- 172.21.15.98:0/787064658 shutdown_connections 2024-04-25T17:45:43.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:43.872+0000 7f80e8dcd640 1 -- 172.21.15.98:0/787064658 wait complete. 2024-04-25T17:45:44.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:44 smithi098 ceph-mon[29192]: pgmap v306: 97 pgs: 97 active+clean; 3.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 18 MiB/s wr, 971 op/s 2024-04-25T17:45:44.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:44 smithi203 ceph-mon[32834]: pgmap v306: 97 pgs: 97 active+clean; 3.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 18 MiB/s wr, 971 op/s 2024-04-25T17:45:44.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:44 smithi173 ceph-mon[32856]: pgmap v306: 97 pgs: 97 active+clean; 3.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 18 MiB/s wr, 971 op/s 2024-04-25T17:45:46.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:46 smithi098 ceph-mon[29192]: pgmap v307: 97 pgs: 97 active+clean; 3.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 15 MiB/s wr, 710 op/s 2024-04-25T17:45:46.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:46 smithi203 ceph-mon[32834]: pgmap v307: 97 pgs: 97 active+clean; 3.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 15 MiB/s wr, 710 op/s 2024-04-25T17:45:46.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:46 smithi173 ceph-mon[32856]: pgmap v307: 97 pgs: 97 active+clean; 3.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 15 MiB/s wr, 710 op/s 2024-04-25T17:45:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 555.0 s, 2666.4 tps, lat 37.627 ms stddev 76.974 2024-04-25T17:45:48.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:48 smithi098 ceph-mon[29192]: pgmap v308: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 48 MiB/s wr, 1.99k op/s 2024-04-25T17:45:48.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:48 smithi203 ceph-mon[32834]: pgmap v308: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 48 MiB/s wr, 1.99k op/s 2024-04-25T17:45:48.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:48 smithi173 ceph-mon[32856]: pgmap v308: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 48 MiB/s wr, 1.99k op/s 2024-04-25T17:45:50.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:50 smithi098 ceph-mon[29192]: pgmap v309: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 45 MiB/s wr, 1.67k op/s 2024-04-25T17:45:50.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:50 smithi203 ceph-mon[32834]: pgmap v309: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 45 MiB/s wr, 1.67k op/s 2024-04-25T17:45:50.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:50 smithi173 ceph-mon[32856]: pgmap v309: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 45 MiB/s wr, 1.67k op/s 2024-04-25T17:45:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 560.0 s, 2618.4 tps, lat 38.188 ms stddev 41.649 2024-04-25T17:45:52.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:52 smithi203 ceph-mon[32834]: pgmap v310: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 47 MiB/s wr, 1.74k op/s 2024-04-25T17:45:52.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:52 smithi173 ceph-mon[32856]: pgmap v310: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 47 MiB/s wr, 1.74k op/s 2024-04-25T17:45:52.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:52 smithi098 ceph-mon[29192]: pgmap v310: 97 pgs: 97 active+clean; 3.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 47 MiB/s wr, 1.74k op/s 2024-04-25T17:45:54.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:54 smithi203 ceph-mon[32834]: pgmap v311: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 52 MiB/s wr, 1.96k op/s 2024-04-25T17:45:54.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:54 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:54.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:54 smithi173 ceph-mon[32856]: pgmap v311: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 52 MiB/s wr, 1.96k op/s 2024-04-25T17:45:54.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:54 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:54.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:54 smithi098 ceph-mon[29192]: pgmap v311: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 52 MiB/s wr, 1.96k op/s 2024-04-25T17:45:54.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:54 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:45:56.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:56 smithi173 ceph-mon[32856]: pgmap v312: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 42 MiB/s wr, 1.62k op/s 2024-04-25T17:45:56.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 565.0 s, 3012.0 tps, lat 33.247 ms stddev 37.730 2024-04-25T17:45:56.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:56 smithi098 ceph-mon[29192]: pgmap v312: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 42 MiB/s wr, 1.62k op/s 2024-04-25T17:45:56.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:56 smithi203 ceph-mon[32834]: pgmap v312: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 42 MiB/s wr, 1.62k op/s 2024-04-25T17:45:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.007+0000 7fac29d29640 1 -- 172.21.15.98:0/2708239735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac241021a0 msgr2=0x7fac24102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.007+0000 7fac29d29640 1 --2- 172.21.15.98:0/2708239735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac241021a0 0x7fac24102580 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fac14009900 tx=0x7fac1402a410 comp rx=0 tx=0).stop 2024-04-25T17:45:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.007+0000 7fac29d29640 1 -- 172.21.15.98:0/2708239735 shutdown_connections 2024-04-25T17:45:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.007+0000 7fac29d29640 1 --2- 172.21.15.98:0/2708239735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac24107270 0x7fac2410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.007+0000 7fac29d29640 1 --2- 172.21.15.98:0/2708239735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac24102b50 0x7fac24106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.007+0000 7fac29d29640 1 --2- 172.21.15.98:0/2708239735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac241021a0 0x7fac24102580 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.007+0000 7fac29d29640 1 -- 172.21.15.98:0/2708239735 >> 172.21.15.98:0/2708239735 conn(0x7fac240fda00 msgr2=0x7fac240ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.008+0000 7fac29d29640 1 -- 172.21.15.98:0/2708239735 shutdown_connections 2024-04-25T17:45:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.008+0000 7fac29d29640 1 -- 172.21.15.98:0/2708239735 wait complete. 2024-04-25T17:45:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.008+0000 7fac29d29640 1 Processor -- start 2024-04-25T17:45:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac29d29640 1 -- start start 2024-04-25T17:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac29d29640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac241021a0 0x7fac24198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac29d29640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac24107270 0x7fac24198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac29d29640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2419cfd0 0x7fac2419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac29d29640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac24112200 con 0x7fac241021a0 2024-04-25T17:45:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac29d29640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fac24112080 con 0x7fac24107270 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac241021a0 0x7fac24198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac241021a0 0x7fac24198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32898/0 (socket says 172.21.15.98:32898) 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac22ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac24107270 0x7fac24198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac237fe640 1 -- 172.21.15.98:0/551536410 learned_addr learned my addr 172.21.15.98:0/551536410 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac23fff640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2419cfd0 0x7fac2419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac29d29640 1 -- 172.21.15.98:0/551536410 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fac24112380 con 0x7fac2419cfd0 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac23fff640 1 -- 172.21.15.98:0/551536410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac24107270 msgr2=0x7fac24198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac23fff640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac24107270 0x7fac24198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac23fff640 1 -- 172.21.15.98:0/551536410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac241021a0 msgr2=0x7fac24198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac23fff640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac241021a0 0x7fac24198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac23fff640 1 -- 172.21.15.98:0/551536410 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2419dac0 con 0x7fac2419cfd0 2024-04-25T17:45:57.012 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.009+0000 7fac237fe640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac241021a0 0x7fac24198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:45:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.010+0000 7fac23fff640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2419cfd0 0x7fac2419d480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009fd0 tx=0x7fac1c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.010+0000 7fac20ff9640 1 -- 172.21.15.98:0/551536410 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c00ce20 con 0x7fac2419cfd0 2024-04-25T17:45:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.010+0000 7fac20ff9640 1 -- 172.21.15.98:0/551536410 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac1c005020 con 0x7fac2419cfd0 2024-04-25T17:45:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.010+0000 7fac20ff9640 1 -- 172.21.15.98:0/551536410 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c010690 con 0x7fac2419cfd0 2024-04-25T17:45:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.010+0000 7fac29d29640 1 -- 172.21.15.98:0/551536410 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2419ddb0 con 0x7fac2419cfd0 2024-04-25T17:45:57.013 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.010+0000 7fac29d29640 1 -- 172.21.15.98:0/551536410 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac241a54f0 con 0x7fac2419cfd0 2024-04-25T17:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.013+0000 7fac20ff9640 1 -- 172.21.15.98:0/551536410 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fac1c0054d0 con 0x7fac2419cfd0 2024-04-25T17:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.013+0000 7fac20ff9640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fac0c0a3e90 0x7fac0c0a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.013+0000 7fac2ca40740 1 Processor -- start 2024-04-25T17:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.013+0000 7fac2ca40740 1 -- start start 2024-04-25T17:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac20ff9640 1 -- 172.21.15.98:0/551536410 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7fac1c0d0df0 con 0x7fac2419cfd0 2024-04-25T17:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac237fe640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fac0c0a3e90 0x7fac0c0a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783620 0x5610ac783a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783f40 0x5610ac78e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 0x5610ac790bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac2ca40740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5610ac6e8970 con 0x5610ac78e800 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac2ca40740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5610ac6854a0 con 0x5610ac783620 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac2ca40740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5610ac6e4930 con 0x5610ac783f40 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac23fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 0x5610ac790bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac237fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783620 0x5610ac783a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac23fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 0x5610ac790bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32900/0 (socket says 172.21.15.98:32900) 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac23fff640 1 -- 172.21.15.98:0/939212197 learned_addr learned my addr 172.21.15.98:0/939212197 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac22ffd640 1 --2- 172.21.15.98:0/939212197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783f40 0x5610ac78e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac22ffd640 1 -- 172.21.15.98:0/939212197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783620 msgr2=0x5610ac783a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac22ffd640 1 --2- 172.21.15.98:0/939212197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783620 0x5610ac783a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac237fe640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fac0c0a3e90 0x7fac0c0a6350 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fac14004730 tx=0x7fac140052c0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac22ffd640 1 -- 172.21.15.98:0/939212197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 msgr2=0x5610ac790bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac22ffd640 1 --2- 172.21.15.98:0/939212197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 0x5610ac790bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac22ffd640 1 -- 172.21.15.98:0/939212197 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610ac3fb210 con 0x5610ac783f40 2024-04-25T17:45:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac23fff640 1 --2- 172.21.15.98:0/939212197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 0x5610ac790bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:45:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.014+0000 7fac22ffd640 1 --2- 172.21.15.98:0/939212197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783f40 0x5610ac78e2c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fac24069280 tx=0x7fac18011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a8de391c879e848 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.015+0000 7fac09ffb640 1 -- 172.21.15.98:0/939212197 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac18011e50 con 0x5610ac783f40 2024-04-25T17:45:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.015+0000 7fac09ffb640 1 -- 172.21.15.98:0/939212197 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac18005020 con 0x5610ac783f40 2024-04-25T17:45:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.015+0000 7fac09ffb640 1 -- 172.21.15.98:0/939212197 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac18004b20 con 0x5610ac783f40 2024-04-25T17:45:57.022 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.015+0000 7fac2ca40740 1 -- 172.21.15.98:0/939212197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783f40 msgr2=0x5610ac78e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:57.022 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.015+0000 7fac2ca40740 1 --2- 172.21.15.98:0/939212197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783f40 0x5610ac78e2c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fac24069280 tx=0x7fac18011440 comp rx=0 tx=0).stop 2024-04-25T17:45:57.022 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.016+0000 7fac2ca40740 1 -- 172.21.15.98:0/939212197 shutdown_connections 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.016+0000 7fac2ca40740 1 --2- 172.21.15.98:0/939212197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 0x5610ac790bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.016+0000 7fac2ca40740 1 --2- 172.21.15.98:0/939212197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783f40 0x5610ac78e2c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.016+0000 7fac2ca40740 1 --2- 172.21.15.98:0/939212197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783620 0x5610ac783a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.016+0000 7fac2ca40740 1 -- 172.21.15.98:0/939212197 >> 172.21.15.98:0/939212197 conn(0x5610ac77fb30 msgr2=0x5610ac77ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.017+0000 7fac2ca40740 1 -- 172.21.15.98:0/939212197 shutdown_connections 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.017+0000 7fac2ca40740 1 -- 172.21.15.98:0/939212197 wait complete. 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.017+0000 7fac2ca40740 1 -- start start 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac2ca40740 1 Processor -- start 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac78e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 0x5610ac78b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 0x5610ac78b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac2ca40740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5610ac795840 con 0x5610ac78e800 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac237fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac78e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac2ca40740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5610ac6830b0 con 0x5610ac783f40 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac2ca40740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5610ac7959c0 con 0x5610ac783620 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac237fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac78e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53130/0 (socket says 172.21.15.98:53130) 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac237fe640 1 -- 172.21.15.98:0/4226429711 learned_addr learned my addr 172.21.15.98:0/4226429711 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac22ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 0x5610ac78b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac23fff640 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 0x5610ac78b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac2ca40740 10 client.0 _ll_register_callbacks cb 0x5610ac670d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac237fe640 1 -- 172.21.15.98:0/4226429711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 msgr2=0x5610ac78b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac237fe640 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 0x5610ac78b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac237fe640 1 -- 172.21.15.98:0/4226429711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 msgr2=0x5610ac78b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac237fe640 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 0x5610ac78b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac237fe640 1 -- 172.21.15.98:0/4226429711 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610ac4c7620 con 0x5610ac783620 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.018+0000 7fac22ffd640 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 0x5610ac78b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.019+0000 7fac237fe640 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac78e130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fac14006010 tx=0x7fac14044000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e856f18fa6e45f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.019+0000 7fac02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.019+0000 7fac02ffd640 1 -- 172.21.15.98:0/4226429711 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14035040 con 0x5610ac783620 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.019+0000 7fac2ca40740 1 -- 172.21.15.98:0/4226429711 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac18005020 con 0x5610ac783620 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.019+0000 7fac02ffd640 1 -- 172.21.15.98:0/4226429711 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7fac14004b90 con 0x5610ac783620 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.019+0000 7fac02ffd640 1 -- 172.21.15.98:0/4226429711 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fac14004d90 con 0x5610ac783620 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.019+0000 7fac2ca40740 10 client.25888 fetch_fsmap learned FSMap version 44 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.019+0000 7fac2ca40740 1 -- 172.21.15.98:0/4226429711 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5610ac7a1760 con 0x5610ac783620 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.019+0000 7fac02ffd640 1 -- 172.21.15.98:0/4226429711 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac18005020 con 0x5610ac783620 2024-04-25T17:45:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.019+0000 7fac02ffd640 1 -- 172.21.15.98:0/4226429711 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fac14032720 con 0x5610ac783620 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.020+0000 7fac2ca40740 10 client.25888 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.020+0000 7fac2ca40740 10 client.25888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.020+0000 7fac2ca40740 20 client.25888 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.020+0000 7fac2ca40740 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5610ac7a2bd0 0x5610ac7a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.020+0000 7fac2ca40740 4 client.25888 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.020+0000 7fac2ca40740 1 -- 172.21.15.98:0/4226429711 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610ac6f6390 con 0x5610ac7a2bd0 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.020+0000 7fac22ffd640 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5610ac7a2bd0 0x5610ac7a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.021+0000 7fac22ffd640 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5610ac7a2bd0 0x5610ac7a4fe0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23ff6c659c4dee10 server_cookie=ab44666d02321f60 in_seq=0 out_seq=0 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.021+0000 7fac02ffd640 10 client.25888 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac02ffd640 1 -- 172.21.15.98:0/4226429711 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5610ac6f6390 con 0x5610ac7a2bd0 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac02ffd640 10 client.25888 handle_command_reply: tid=0 2024-04-25T17:45:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 client.25888 shutdown 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 -- 172.21.15.98:0/4226429711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 msgr2=0x5610ac78e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac78e130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fac14006010 tx=0x7fac14044000 comp rx=0 tx=0).stop 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 -- 172.21.15.98:0/4226429711 shutdown_connections 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5610ac7a2bd0 0x5610ac7a4fe0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac02ffd640 0 client.25888 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac78e800 0x5610ac78b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 0x5610ac78b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 --2- 172.21.15.98:0/4226429711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac78e130 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 -- 172.21.15.98:0/4226429711 >> 172.21.15.98:0/4226429711 conn(0x5610ac77fb30 msgr2=0x5610ac785860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 -- 172.21.15.98:0/4226429711 shutdown_connections 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 1 -- 172.21.15.98:0/4226429711 wait complete. 2024-04-25T17:45:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.022+0000 7fac2ca40740 20 client.25888 trim_cache size 0 max 16384 2024-04-25T17:45:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.043+0000 7fac2ca40740 1 Processor -- start 2024-04-25T17:45:57.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.043+0000 7fac2ca40740 1 -- start start 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac77fb30 0x5610ac79e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783620 0x5610ac77eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac783f40 0x5610ac7aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac2ca40740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac241ae2f0 con 0x5610ac783f40 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac2ca40740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fac241abb60 con 0x5610ac783620 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac2ca40740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fac241a99d0 con 0x5610ac77fb30 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac237fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac77fb30 0x5610ac79e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac237fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac77fb30 0x5610ac79e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53146/0 (socket says 172.21.15.98:53146) 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac237fe640 1 -- 172.21.15.98:0/2631910680 learned_addr learned my addr 172.21.15.98:0/2631910680 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac23fff640 1 --2- 172.21.15.98:0/2631910680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac783f40 0x5610ac7aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac22ffd640 1 --2- 172.21.15.98:0/2631910680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783620 0x5610ac77eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac237fe640 1 -- 172.21.15.98:0/2631910680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783620 msgr2=0x5610ac77eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac237fe640 1 --2- 172.21.15.98:0/2631910680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783620 0x5610ac77eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac237fe640 1 -- 172.21.15.98:0/2631910680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac783f40 msgr2=0x5610ac7aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac237fe640 1 --2- 172.21.15.98:0/2631910680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac783f40 0x5610ac7aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac237fe640 1 -- 172.21.15.98:0/2631910680 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610ac783d70 con 0x5610ac77fb30 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.044+0000 7fac237fe640 1 --2- 172.21.15.98:0/2631910680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac77fb30 0x5610ac79e700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fac14005c40 tx=0x7fac140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90bce56900016b5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.045+0000 7fac097fa640 1 -- 172.21.15.98:0/2631910680 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1405f070 con 0x5610ac77fb30 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.045+0000 7fac097fa640 1 -- 172.21.15.98:0/2631910680 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac1404e070 con 0x5610ac77fb30 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.045+0000 7fac097fa640 1 -- 172.21.15.98:0/2631910680 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14003830 con 0x5610ac77fb30 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.045+0000 7fac2ca40740 1 -- 172.21.15.98:0/2631910680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac77fb30 msgr2=0x5610ac79e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.045+0000 7fac2ca40740 1 --2- 172.21.15.98:0/2631910680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac77fb30 0x5610ac79e700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fac14005c40 tx=0x7fac140046c0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.045+0000 7fac2ca40740 1 -- 172.21.15.98:0/2631910680 shutdown_connections 2024-04-25T17:45:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.045+0000 7fac2ca40740 1 --2- 172.21.15.98:0/2631910680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac783f40 0x5610ac7aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.045+0000 7fac2ca40740 1 --2- 172.21.15.98:0/2631910680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783620 0x5610ac77eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.045+0000 7fac2ca40740 1 --2- 172.21.15.98:0/2631910680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac77fb30 0x5610ac79e700 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.045+0000 7fac2ca40740 1 -- 172.21.15.98:0/2631910680 >> 172.21.15.98:0/2631910680 conn(0x5610ac78bac0 msgr2=0x5610ac77e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.046+0000 7fac2ca40740 1 -- 172.21.15.98:0/2631910680 shutdown_connections 2024-04-25T17:45:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.046+0000 7fac2ca40740 1 -- 172.21.15.98:0/2631910680 wait complete. 2024-04-25T17:45:57.051 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.046+0000 7fac2ca40740 1 -- start start 2024-04-25T17:45:57.051 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.046+0000 7fac2ca40740 1 Processor -- start 2024-04-25T17:45:57.051 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac77fb30 0x5610ac793790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.051 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac793cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac2ca40740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 0x5610ac794210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac2ca40740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac241aa020 con 0x5610ac77fb30 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac2ca40740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fac241ab490 con 0x5610ac783f40 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac2ca40740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fac241aa6f0 con 0x5610ac783620 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac2ca40740 10 client.0 _ll_register_callbacks cb 0x7fac14005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac23fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 0x5610ac794210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac22ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac793cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac22ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac793cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53154/0 (socket says 172.21.15.98:53154) 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac22ffd640 1 -- 172.21.15.98:0/2277861118 learned_addr learned my addr 172.21.15.98:0/2277861118 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac237fe640 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac77fb30 0x5610ac793790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac22ffd640 1 -- 172.21.15.98:0/2277861118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 msgr2=0x5610ac794210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac22ffd640 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 0x5610ac794210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac22ffd640 1 -- 172.21.15.98:0/2277861118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac77fb30 msgr2=0x5610ac793790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac22ffd640 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac77fb30 0x5610ac793790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.047+0000 7fac22ffd640 1 -- 172.21.15.98:0/2277861118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610ac799eb0 con 0x5610ac783620 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.048+0000 7fac22ffd640 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac793cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fac18009730 tx=0x7fac18005670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd7fb1531eaad2cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.048+0000 7fac02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.048+0000 7fac02ffd640 1 -- 172.21.15.98:0/2277861118 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1801e040 con 0x5610ac783620 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.048+0000 7fac02ffd640 1 -- 172.21.15.98:0/2277861118 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8513+0+0 (secure 0 0 0) 0x7fac180037f0 con 0x5610ac783620 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.048+0000 7fac2ca40740 1 -- 172.21.15.98:0/2277861118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac1404e070 con 0x5610ac783620 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.048+0000 7fac02ffd640 1 -- 172.21.15.98:0/2277861118 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fac1803a050 con 0x5610ac783620 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.048+0000 7fac2ca40740 10 client.25894 fetch_fsmap learned FSMap version 44 2024-04-25T17:45:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.048+0000 7fac2ca40740 1 -- 172.21.15.98:0/2277861118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5610ac79b180 con 0x5610ac783620 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.048+0000 7fac02ffd640 1 -- 172.21.15.98:0/2277861118 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac1802e070 con 0x5610ac783620 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.048+0000 7fac02ffd640 1 -- 172.21.15.98:0/2277861118 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fac18018070 con 0x5610ac783620 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.049+0000 7fac2ca40740 10 client.25894 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.049+0000 7fac2ca40740 10 client.25894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.049+0000 7fac2ca40740 20 client.25894 populate_metadata read hostname 'smithi098' 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.049+0000 7fac2ca40740 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5610ac78e800 0x5610ac80f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.049+0000 7fac2ca40740 4 client.25894 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.049+0000 7fac2ca40740 1 -- 172.21.15.98:0/2277861118 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fac1404e070 con 0x5610ac78e800 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.049+0000 7fac237fe640 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5610ac78e800 0x5610ac80f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.049+0000 7fac02ffd640 10 client.25894 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.049+0000 7fac237fe640 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5610ac78e800 0x5610ac80f9f0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d2db430326bfba3 server_cookie=3dfed1add0f56930 in_seq=0 out_seq=0 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.057+0000 7fac02ffd640 1 -- 172.21.15.98:0/2277861118 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fac1404e070 con 0x5610ac78e800 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.057+0000 7fac02ffd640 10 client.25894 handle_command_reply: tid=0 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 client.25894 shutdown 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 -- 172.21.15.98:0/2277861118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 msgr2=0x5610ac793cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac793cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fac18009730 tx=0x7fac18005670 comp rx=0 tx=0).stop 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 -- 172.21.15.98:0/2277861118 shutdown_connections 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5610ac78e800 0x5610ac80f9f0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5610ac783f40 0x5610ac794210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac02ffd640 0 client.25894 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5610ac783620 0x5610ac793cd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 --2- 172.21.15.98:0/2277861118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5610ac77fb30 0x5610ac793790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 -- 172.21.15.98:0/2277861118 >> 172.21.15.98:0/2277861118 conn(0x5610ac78bac0 msgr2=0x5610ac77ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:57.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 -- 172.21.15.98:0/2277861118 shutdown_connections 2024-04-25T17:45:57.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.058+0000 7fac2ca40740 1 -- 172.21.15.98:0/2277861118 wait complete. 2024-04-25T17:45:57.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac2ca40740 20 client.25894 trim_cache size 0 max 16384 2024-04-25T17:45:57.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:45:57.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:45:57.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:45:57.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:45:57.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 -- 172.21.15.98:0/551536410 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fac0c0a3e90 msgr2=0x7fac0c0a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:57.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fac0c0a3e90 0x7fac0c0a6350 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fac14004730 tx=0x7fac140052c0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 -- 172.21.15.98:0/551536410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2419cfd0 msgr2=0x7fac2419d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:45:57.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2419cfd0 0x7fac2419d480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009fd0 tx=0x7fac1c00ef10 comp rx=0 tx=0).stop 2024-04-25T17:45:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 -- 172.21.15.98:0/551536410 shutdown_connections 2024-04-25T17:45:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fac0c0a3e90 0x7fac0c0a6350 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac2419cfd0 0x7fac2419d480 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac24107270 0x7fac24198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 --2- 172.21.15.98:0/551536410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac241021a0 0x7fac24198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:45:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 -- 172.21.15.98:0/551536410 >> 172.21.15.98:0/551536410 conn(0x7fac240fda00 msgr2=0x7fac240ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:45:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 -- 172.21.15.98:0/551536410 shutdown_connections 2024-04-25T17:45:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:45:57.059+0000 7fac29d29640 1 -- 172.21.15.98:0/551536410 wait complete. 2024-04-25T17:45:58.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:45:58 smithi173 ceph-mon[32856]: pgmap v313: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 MiB/s wr, 1.91k op/s 2024-04-25T17:45:58.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:45:58 smithi098 ceph-mon[29192]: pgmap v313: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 MiB/s wr, 1.91k op/s 2024-04-25T17:45:58.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:45:58 smithi203 ceph-mon[32834]: pgmap v313: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 MiB/s wr, 1.91k op/s 2024-04-25T17:46:00.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:00 smithi173 ceph-mon[32856]: pgmap v314: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 632 op/s 2024-04-25T17:46:00.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:00 smithi098 ceph-mon[29192]: pgmap v314: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 632 op/s 2024-04-25T17:46:00.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:00 smithi203 ceph-mon[32834]: pgmap v314: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 632 op/s 2024-04-25T17:46:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 570.0 s, 2781.8 tps, lat 35.438 ms stddev 67.858 2024-04-25T17:46:02.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:02 smithi173 ceph-mon[32856]: pgmap v315: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 632 op/s 2024-04-25T17:46:02.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:02 smithi098 ceph-mon[29192]: pgmap v315: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 632 op/s 2024-04-25T17:46:02.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:02 smithi203 ceph-mon[32834]: pgmap v315: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 632 op/s 2024-04-25T17:46:04.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:04 smithi173 ceph-mon[32856]: pgmap v316: 97 pgs: 97 active+clean; 4.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 32 MiB/s wr, 1.62k op/s 2024-04-25T17:46:04.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:04 smithi098 ceph-mon[29192]: pgmap v316: 97 pgs: 97 active+clean; 4.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 32 MiB/s wr, 1.62k op/s 2024-04-25T17:46:04.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:04 smithi203 ceph-mon[32834]: pgmap v316: 97 pgs: 97 active+clean; 4.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 32 MiB/s wr, 1.62k op/s 2024-04-25T17:46:06.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:06 smithi173 ceph-mon[32856]: pgmap v317: 97 pgs: 97 active+clean; 4.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 1.34k op/s 2024-04-25T17:46:06.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:06 smithi173 ceph-mon[32856]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T17:46:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 575.0 s, 2303.8 tps, lat 43.847 ms stddev 58.865 2024-04-25T17:46:06.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:06 smithi098 ceph-mon[29192]: pgmap v317: 97 pgs: 97 active+clean; 4.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 1.34k op/s 2024-04-25T17:46:06.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:06 smithi098 ceph-mon[29192]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T17:46:06.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:06 smithi203 ceph-mon[32834]: pgmap v317: 97 pgs: 97 active+clean; 4.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 1.34k op/s 2024-04-25T17:46:06.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:06 smithi203 ceph-mon[32834]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T17:46:08.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:08 smithi173 ceph-mon[32856]: pgmap v319: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 38 MiB/s wr, 2.38k op/s 2024-04-25T17:46:08.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:08 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:46:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:08 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:08 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:08 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:08 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:08 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:08 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:08 smithi098 ceph-mon[29192]: pgmap v319: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 38 MiB/s wr, 2.38k op/s 2024-04-25T17:46:08.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:08 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:46:08.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:08 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:08 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:08 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:08 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:08 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:08 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:08 smithi203 ceph-mon[32834]: pgmap v319: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 38 MiB/s wr, 2.38k op/s 2024-04-25T17:46:08.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:08 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:46:08.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:08 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:08 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:08 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:08 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:08 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:08.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:08 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.181+0000 7f1a45a42640 1 -- 172.21.15.98:0/1232697713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a401031e0 msgr2=0x7f1a401035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.181+0000 7f1a45a42640 1 --2- 172.21.15.98:0/1232697713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a401031e0 0x7f1a401035c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009900 tx=0x7f1a3002a510 comp rx=0 tx=0).stop 2024-04-25T17:46:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.181+0000 7f1a45a42640 1 -- 172.21.15.98:0/1232697713 shutdown_connections 2024-04-25T17:46:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.181+0000 7f1a45a42640 1 --2- 172.21.15.98:0/1232697713 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a4010c520 0x7f1a4010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.181+0000 7f1a45a42640 1 --2- 172.21.15.98:0/1232697713 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a40103b00 0x7f1a4010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.181+0000 7f1a45a42640 1 --2- 172.21.15.98:0/1232697713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a401031e0 0x7f1a401035c0 secure :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009900 tx=0x7f1a3002a510 comp rx=0 tx=0).stop 2024-04-25T17:46:10.182 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.181+0000 7f1a45a42640 1 -- 172.21.15.98:0/1232697713 >> 172.21.15.98:0/1232697713 conn(0x7f1a40077fd0 msgr2=0x7f1a400783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.182+0000 7f1a45a42640 1 -- 172.21.15.98:0/1232697713 shutdown_connections 2024-04-25T17:46:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.182+0000 7f1a45a42640 1 -- 172.21.15.98:0/1232697713 wait complete. 2024-04-25T17:46:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.183+0000 7f1a45a42640 1 Processor -- start 2024-04-25T17:46:10.184 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.183+0000 7f1a45a42640 1 -- start start 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.183+0000 7f1a45a42640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a40103b00 0x7f1a401a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.183+0000 7f1a45a42640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a4010c520 0x7f1a401a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.183+0000 7f1a45a42640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a4019a8e0 0x7f1a4019ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.183+0000 7f1a45a42640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1a401020c0 con 0x7f1a40103b00 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.183+0000 7f1a45a42640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1a40101f40 con 0x7f1a4010c520 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.183+0000 7f1a45a42640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1a40102240 con 0x7f1a4019a8e0 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a40103b00 0x7f1a401a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a40103b00 0x7f1a401a0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37250/0 (socket says 172.21.15.98:37250) 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a4019a8e0 0x7f1a4019ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3effd640 1 -- 172.21.15.98:0/3817316276 learned_addr learned my addr 172.21.15.98:0/3817316276 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3e7fc640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a4010c520 0x7f1a401a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3effd640 1 -- 172.21.15.98:0/3817316276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a4019a8e0 msgr2=0x7f1a4019ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3effd640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a4019a8e0 0x7f1a4019ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3effd640 1 -- 172.21.15.98:0/3817316276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a4010c520 msgr2=0x7f1a401a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3effd640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a4010c520 0x7f1a401a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3effd640 1 -- 172.21.15.98:0/3817316276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4019b400 con 0x7f1a40103b00 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3f7fe640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a4019a8e0 0x7f1a4019ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a3effd640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a40103b00 0x7f1a401a0730 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1a3002ab00 tx=0x7f1a30003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a44a40640 1 -- 172.21.15.98:0/3817316276 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3003f020 con 0x7f1a40103b00 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a45a42640 1 -- 172.21.15.98:0/3817316276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a4019b690 con 0x7f1a40103b00 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a45a42640 1 -- 172.21.15.98:0/3817316276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a401a7880 con 0x7f1a40103b00 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a44a40640 1 -- 172.21.15.98:0/3817316276 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a30031b60 con 0x7f1a40103b00 2024-04-25T17:46:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.184+0000 7f1a44a40640 1 -- 172.21.15.98:0/3817316276 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30035070 con 0x7f1a40103b00 2024-04-25T17:46:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a48831740 1 Processor -- start 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a48831740 1 -- start start 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a48831740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 0x5556156e9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a48831740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 0x5556156f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a48831740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x5556156f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a48831740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55561564e970 con 0x5556156e9f40 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a48831740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5556155eb4a0 con 0x5556156e9620 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a48831740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55561564a930 con 0x5556156f4800 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a44a40640 1 -- 172.21.15.98:0/3817316276 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1a30042020 con 0x7f1a40103b00 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 0x5556156e9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a3e7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x5556156f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 0x5556156f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.189 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a3e7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x5556156f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:33302/0 (socket says 172.21.15.98:33302) 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a3e7fc640 1 -- 172.21.15.98:0/3931237851 learned_addr learned my addr 172.21.15.98:0/3931237851 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 0x5556156e9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:58284/0 (socket says 172.21.15.98:58284) 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a3e7fc640 1 -- 172.21.15.98:0/3931237851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 msgr2=0x5556156e9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.188+0000 7f1a3e7fc640 1 --2- 172.21.15.98:0/3931237851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 0x5556156e9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.189+0000 7f1a3e7fc640 1 -- 172.21.15.98:0/3931237851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 msgr2=0x5556156f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.189+0000 7f1a3e7fc640 1 --2- 172.21.15.98:0/3931237851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 0x5556156f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.189+0000 7f1a3e7fc640 1 -- 172.21.15.98:0/3931237851 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555615361210 con 0x5556156f4800 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.189+0000 7f1a3e7fc640 1 --2- 172.21.15.98:0/3931237851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x5556156f6bf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009c60 tx=0x7f1a34011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=297644878141d37f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.189+0000 7f1a44a40640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1a280a3e40 0x7f1a280a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.189+0000 7f1a25ffb640 1 -- 172.21.15.98:0/3931237851 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34004230 con 0x5556156f4800 2024-04-25T17:46:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.189+0000 7f1a25ffb640 1 -- 172.21.15.98:0/3931237851 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a340043d0 con 0x5556156f4800 2024-04-25T17:46:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.189+0000 7f1a25ffb640 1 -- 172.21.15.98:0/3931237851 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34004ce0 con 0x5556156f4800 2024-04-25T17:46:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.189+0000 7f1a3f7fe640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1a280a3e40 0x7f1a280a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.189+0000 7f1a44a40640 1 -- 172.21.15.98:0/3817316276 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f1a300f08b0 con 0x7f1a40103b00 2024-04-25T17:46:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a3f7fe640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1a280a3e40 0x7f1a280a6300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009fd0 tx=0x7f1a38009500 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a48831740 1 -- 172.21.15.98:0/3931237851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 msgr2=0x5556156f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a48831740 1 --2- 172.21.15.98:0/3931237851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x5556156f6bf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009c60 tx=0x7f1a34011810 comp rx=0 tx=0).stop 2024-04-25T17:46:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a48831740 1 -- 172.21.15.98:0/3931237851 shutdown_connections 2024-04-25T17:46:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a48831740 1 --2- 172.21.15.98:0/3931237851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x5556156f6bf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a48831740 1 --2- 172.21.15.98:0/3931237851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 0x5556156f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a48831740 1 --2- 172.21.15.98:0/3931237851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 0x5556156e9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a48831740 1 -- 172.21.15.98:0/3931237851 >> 172.21.15.98:0/3931237851 conn(0x5556156e5b30 msgr2=0x5556156e5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a48831740 1 -- 172.21.15.98:0/3931237851 shutdown_connections 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a48831740 1 -- 172.21.15.98:0/3931237851 wait complete. 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.190+0000 7f1a48831740 1 -- start start 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.191+0000 7f1a48831740 1 Processor -- start 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a48831740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156f4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a48831740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156e9f40 0x5556156f13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a48831740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 0x5556156f1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a48831740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5556156f7430 con 0x5556156e9620 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a48831740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5556155e90b0 con 0x5556156f4800 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a48831740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5556156f75b0 con 0x5556156e9f40 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a48831740 10 client.0 _ll_register_callbacks cb 0x5556155d6d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156f4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 0x5556156f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 0x5556156f1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:58286/0 (socket says 172.21.15.98:58286) 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156f4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37270/0 (socket says 172.21.15.98:37270) 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a3f7fe640 1 -- 172.21.15.98:0/1114446052 learned_addr learned my addr 172.21.15.98:0/1114446052 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.192+0000 7f1a3e7fc640 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156e9f40 0x5556156f13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a3f7fe640 1 -- 172.21.15.98:0/1114446052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156e9f40 msgr2=0x5556156f13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a3f7fe640 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156e9f40 0x5556156f13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a3f7fe640 1 -- 172.21.15.98:0/1114446052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 msgr2=0x5556156f4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a3f7fe640 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156f4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a3f7fe640 1 -- 172.21.15.98:0/1114446052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55561542d620 con 0x5556156f4800 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a3f7fe640 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 0x5556156f1930 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1a38013010 tx=0x7f1a38004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=befb9eb876f71bec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a1effd640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a1effd640 1 -- 172.21.15.98:0/1114446052 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a380045c0 con 0x5556156f4800 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a1effd640 1 -- 172.21.15.98:0/1114446052 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f1a38004760 con 0x5556156f4800 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a48831740 1 -- 172.21.15.98:0/1114446052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a340043d0 con 0x5556156f4800 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a1effd640 1 -- 172.21.15.98:0/1114446052 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1a38012020 con 0x5556156f4800 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a48831740 10 client.25886 fetch_fsmap learned FSMap version 44 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.193+0000 7f1a48831740 1 -- 172.21.15.98:0/1114446052 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5556157033e0 con 0x5556156f4800 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.194+0000 7f1a1effd640 1 -- 172.21.15.98:0/1114446052 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a380121c0 con 0x5556156f4800 2024-04-25T17:46:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.194+0000 7f1a1effd640 1 -- 172.21.15.98:0/1114446052 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f1a38027070 con 0x5556156f4800 2024-04-25T17:46:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.194+0000 7f1a48831740 10 client.25886 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:46:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.194+0000 7f1a48831740 10 client.25886 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:46:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.194+0000 7f1a48831740 20 client.25886 populate_metadata read hostname 'smithi098' 2024-04-25T17:46:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.194+0000 7f1a48831740 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555615704960 0x555615706db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.194+0000 7f1a48831740 4 client.25886 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:46:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.194+0000 7f1a48831740 1 -- 172.21.15.98:0/1114446052 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1a340043d0 con 0x555615704960 2024-04-25T17:46:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.194+0000 7f1a3effd640 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555615704960 0x555615706db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.194+0000 7f1a1effd640 10 client.25886 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.195+0000 7f1a3effd640 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555615704960 0x555615706db0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e07b2f01ecd38aa server_cookie=74d2649e6e4ff164 in_seq=0 out_seq=0 2024-04-25T17:46:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.195+0000 7f1a1effd640 1 -- 172.21.15.98:0/1114446052 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f1a340043d0 con 0x555615704960 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.195+0000 7f1a1effd640 10 client.25886 handle_command_reply: tid=0 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.195+0000 7f1a48831740 1 client.25886 shutdown 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.195+0000 7f1a48831740 1 -- 172.21.15.98:0/1114446052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 msgr2=0x5556156f1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.195+0000 7f1a48831740 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 0x5556156f1930 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1a38013010 tx=0x7f1a38004ab0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.196+0000 7f1a48831740 1 -- 172.21.15.98:0/1114446052 shutdown_connections 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.196+0000 7f1a48831740 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555615704960 0x555615706db0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.196+0000 7f1a48831740 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 0x5556156f1930 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.196+0000 7f1a48831740 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156e9f40 0x5556156f13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.196+0000 7f1a1effd640 0 client.25886 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.196+0000 7f1a48831740 1 --2- 172.21.15.98:0/1114446052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156f4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.196+0000 7f1a48831740 1 -- 172.21.15.98:0/1114446052 >> 172.21.15.98:0/1114446052 conn(0x5556156e5b30 msgr2=0x5556156eb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.196+0000 7f1a48831740 1 -- 172.21.15.98:0/1114446052 shutdown_connections 2024-04-25T17:46:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.196+0000 7f1a48831740 1 -- 172.21.15.98:0/1114446052 wait complete. 2024-04-25T17:46:10.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.196+0000 7f1a48831740 20 client.25886 trim_cache size 0 max 16384 2024-04-25T17:46:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.211+0000 7f1a48831740 1 Processor -- start 2024-04-25T17:46:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.211+0000 7f1a48831740 1 -- start start 2024-04-25T17:46:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a48831740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 0x555615700350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a48831740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 0x5556156e4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a48831740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x55561570c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a48831740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c011080 con 0x5556156e9f40 2024-04-25T17:46:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a48831740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c00e8f0 con 0x5556156e9620 2024-04-25T17:46:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a48831740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c00c760 con 0x5556156f4800 2024-04-25T17:46:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 0x5556156e4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x55561570c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 0x5556156e4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37276/0 (socket says 172.21.15.98:37276) 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x55561570c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:33324/0 (socket says 172.21.15.98:33324) 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3e7fc640 1 -- 172.21.15.98:0/1458639863 learned_addr learned my addr 172.21.15.98:0/1458639863 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3effd640 1 --2- 172.21.15.98:0/1458639863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 0x555615700350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3f7fe640 1 -- 172.21.15.98:0/1458639863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 msgr2=0x555615700350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3f7fe640 1 --2- 172.21.15.98:0/1458639863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 0x555615700350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3f7fe640 1 -- 172.21.15.98:0/1458639863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 msgr2=0x5556156e4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3f7fe640 1 --2- 172.21.15.98:0/1458639863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 0x5556156e4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3f7fe640 1 -- 172.21.15.98:0/1458639863 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556156f1f30 con 0x5556156f4800 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3effd640 1 --2- 172.21.15.98:0/1458639863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 0x555615700350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3e7fc640 1 --2- 172.21.15.98:0/1458639863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 0x5556156e4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:10.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.212+0000 7f1a3f7fe640 1 --2- 172.21.15.98:0/1458639863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x55561570c6c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1a3800ffd0 tx=0x7f1a38005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29816b40394b6e5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a257fa640 1 -- 172.21.15.98:0/1458639863 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a38004b20 con 0x5556156f4800 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a257fa640 1 -- 172.21.15.98:0/1458639863 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a38004cc0 con 0x5556156f4800 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a257fa640 1 -- 172.21.15.98:0/1458639863 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a380058a0 con 0x5556156f4800 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a48831740 1 -- 172.21.15.98:0/1458639863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 msgr2=0x55561570c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a48831740 1 --2- 172.21.15.98:0/1458639863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x55561570c6c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1a3800ffd0 tx=0x7f1a38005bd0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a48831740 1 -- 172.21.15.98:0/1458639863 shutdown_connections 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a48831740 1 --2- 172.21.15.98:0/1458639863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156f4800 0x55561570c6c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a48831740 1 --2- 172.21.15.98:0/1458639863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9f40 0x5556156e4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a48831740 1 --2- 172.21.15.98:0/1458639863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156e9620 0x555615700350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a48831740 1 -- 172.21.15.98:0/1458639863 >> 172.21.15.98:0/1458639863 conn(0x5556156e5b30 msgr2=0x5556156e4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:10.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a48831740 1 -- 172.21.15.98:0/1458639863 shutdown_connections 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.213+0000 7f1a48831740 1 -- 172.21.15.98:0/1458639863 wait complete. 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a48831740 1 -- start start 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a48831740 1 Processor -- start 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a48831740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156fa500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a48831740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156e9f40 0x5556156faa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a48831740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 0x5556156faf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a48831740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c00cdb0 con 0x5556156e9620 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a48831740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c00e220 con 0x5556156f4800 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a48831740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c00d480 con 0x5556156e9f40 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a48831740 10 client.0 _ll_register_callbacks cb 0x7f1a38003e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156fa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a3effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156fa500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37278/0 (socket says 172.21.15.98:37278) 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a3effd640 1 -- 172.21.15.98:0/2822983850 learned_addr learned my addr 172.21.15.98:0/2822983850 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a3e7fc640 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156e9f40 0x5556156faa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a3f7fe640 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 0x5556156faf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a3effd640 1 -- 172.21.15.98:0/2822983850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156e9f40 msgr2=0x5556156faa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a3effd640 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156e9f40 0x5556156faa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.214+0000 7f1a3effd640 1 -- 172.21.15.98:0/2822983850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 msgr2=0x5556156faf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a3effd640 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 0x5556156faf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a3effd640 1 -- 172.21.15.98:0/2822983850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55561576a190 con 0x5556156e9620 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a3effd640 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156fa500 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1a300bc4f0 tx=0x7f1a3002a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b9bc3991042ce9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a1effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a1effd640 1 -- 172.21.15.98:0/2822983850 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a300bc150 con 0x5556156e9620 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a1effd640 1 -- 172.21.15.98:0/2822983850 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f1a3003b460 con 0x5556156e9620 2024-04-25T17:46:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a48831740 1 -- 172.21.15.98:0/2822983850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a38004cc0 con 0x5556156e9620 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a1effd640 1 -- 172.21.15.98:0/2822983850 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1a300bcef0 con 0x5556156e9620 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a48831740 10 client.16410 fetch_fsmap learned FSMap version 44 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a48831740 1 -- 172.21.15.98:0/2822983850 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55561576b460 con 0x5556156e9620 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a1effd640 1 -- 172.21.15.98:0/2822983850 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a38004cc0 con 0x5556156e9620 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a1effd640 1 -- 172.21.15.98:0/2822983850 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f1a300f0ce0 con 0x5556156e9620 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a48831740 10 client.16410 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a48831740 10 client.16410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a48831740 20 client.16410 populate_metadata read hostname 'smithi098' 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a48831740 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55561576bb30 0x55561576df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a48831740 4 client.16410 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.215+0000 7f1a48831740 1 -- 172.21.15.98:0/2822983850 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55561565c390 con 0x55561576bb30 2024-04-25T17:46:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.216+0000 7f1a3e7fc640 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55561576bb30 0x55561576df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.216+0000 7f1a1effd640 10 client.16410 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.216+0000 7f1a3e7fc640 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55561576bb30 0x55561576df20 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4fbf8bec12371a4 server_cookie=894fe67a940ba74d in_seq=0 out_seq=0 2024-04-25T17:46:10.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.323+0000 7f1a1effd640 1 -- 172.21.15.98:0/2822983850 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55561565c390 con 0x55561576bb30 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.323+0000 7f1a1effd640 10 client.16410 handle_command_reply: tid=0 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.323+0000 7f1a48831740 1 client.16410 shutdown 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.323+0000 7f1a48831740 1 -- 172.21.15.98:0/2822983850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 msgr2=0x5556156fa500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.323+0000 7f1a48831740 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156fa500 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1a300bc4f0 tx=0x7f1a3002a510 comp rx=0 tx=0).stop 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.324+0000 7f1a48831740 1 -- 172.21.15.98:0/2822983850 shutdown_connections 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.324+0000 7f1a48831740 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55561576bb30 0x55561576df20 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.324+0000 7f1a48831740 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5556156f4800 0x5556156faf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.324+0000 7f1a48831740 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5556156e9f40 0x5556156faa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.324+0000 7f1a48831740 1 --2- 172.21.15.98:0/2822983850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5556156e9620 0x5556156fa500 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.324+0000 7f1a48831740 1 -- 172.21.15.98:0/2822983850 >> 172.21.15.98:0/2822983850 conn(0x5556156e5b30 msgr2=0x5556156e4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.324+0000 7f1a1effd640 0 client.16410 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.324+0000 7f1a48831740 1 -- 172.21.15.98:0/2822983850 shutdown_connections 2024-04-25T17:46:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.324+0000 7f1a48831740 1 -- 172.21.15.98:0/2822983850 wait complete. 2024-04-25T17:46:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.324+0000 7f1a48831740 20 client.16410 trim_cache size 0 max 16384 2024-04-25T17:46:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:46:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:46:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:46:10.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:46:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.327+0000 7f1a45a42640 1 -- 172.21.15.98:0/3817316276 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1a280a3e40 msgr2=0x7f1a280a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.327+0000 7f1a45a42640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1a280a3e40 0x7f1a280a6300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009fd0 tx=0x7f1a38009500 comp rx=0 tx=0).stop 2024-04-25T17:46:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.327+0000 7f1a45a42640 1 -- 172.21.15.98:0/3817316276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a40103b00 msgr2=0x7f1a401a0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.327+0000 7f1a45a42640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a40103b00 0x7f1a401a0730 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1a3002ab00 tx=0x7f1a30003c10 comp rx=0 tx=0).stop 2024-04-25T17:46:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.327+0000 7f1a45a42640 1 -- 172.21.15.98:0/3817316276 shutdown_connections 2024-04-25T17:46:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.327+0000 7f1a45a42640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1a280a3e40 0x7f1a280a6300 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.327+0000 7f1a45a42640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a4019a8e0 0x7f1a4019ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.327+0000 7f1a45a42640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1a4010c520 0x7f1a401a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.327+0000 7f1a45a42640 1 --2- 172.21.15.98:0/3817316276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a40103b00 0x7f1a401a0730 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.327+0000 7f1a45a42640 1 -- 172.21.15.98:0/3817316276 >> 172.21.15.98:0/3817316276 conn(0x7f1a40077fd0 msgr2=0x7f1a40103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.328+0000 7f1a45a42640 1 -- 172.21.15.98:0/3817316276 shutdown_connections 2024-04-25T17:46:10.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:10.328+0000 7f1a45a42640 1 -- 172.21.15.98:0/3817316276 wait complete. 2024-04-25T17:46:10.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:10 smithi173 ceph-mon[32856]: pgmap v320: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 38 MiB/s wr, 2.38k op/s 2024-04-25T17:46:10.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:10 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:10 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:10 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:10 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:10 smithi173 ceph-mon[32856]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T17:46:10.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:10 smithi098 ceph-mon[29192]: pgmap v320: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 38 MiB/s wr, 2.38k op/s 2024-04-25T17:46:10.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:10 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:10 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:10 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:10 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:10 smithi098 ceph-mon[29192]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T17:46:10.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:10 smithi203 ceph-mon[32834]: pgmap v320: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 38 MiB/s wr, 2.38k op/s 2024-04-25T17:46:10.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:10 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:10 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:10 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:10 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:10.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:10 smithi203 ceph-mon[32834]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T17:46:11.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:11 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:11 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:11 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:11 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:11 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:11 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:11 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:11 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 580.0 s, 2752.6 tps, lat 36.278 ms stddev 43.613 2024-04-25T17:46:11.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:11 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:11 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:11 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:11 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:11 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:11 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:11 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:11 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:11 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:11 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:11 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:11 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:11 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:11 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:11 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:11.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:11 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:12 smithi173 ceph-mon[32856]: pgmap v322: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 19 MiB/s wr, 1.39k op/s 2024-04-25T17:46:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:12 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi203 to 176.2M 2024-04-25T17:46:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:12 smithi173 ceph-mon[32856]: Unable to set osd_memory_target on smithi203 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-25T17:46:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:12 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T17:46:12.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:12 smithi173 ceph-mon[32856]: Unable to set osd_memory_target on smithi173 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-04-25T17:46:12.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:12 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:12.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:12 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:12.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:12 smithi098 ceph-mon[29192]: pgmap v322: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 19 MiB/s wr, 1.39k op/s 2024-04-25T17:46:12.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:12 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi203 to 176.2M 2024-04-25T17:46:12.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:12 smithi098 ceph-mon[29192]: Unable to set osd_memory_target on smithi203 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-25T17:46:12.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:12 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T17:46:12.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:12 smithi098 ceph-mon[29192]: Unable to set osd_memory_target on smithi173 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-04-25T17:46:12.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:12 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:12.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:12 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:12.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:12 smithi203 ceph-mon[32834]: pgmap v322: 97 pgs: 97 active+clean; 4.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 19 MiB/s wr, 1.39k op/s 2024-04-25T17:46:12.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:12 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi203 to 176.2M 2024-04-25T17:46:12.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:12 smithi203 ceph-mon[32834]: Unable to set osd_memory_target on smithi203 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-25T17:46:12.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:12 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T17:46:12.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:12 smithi203 ceph-mon[32834]: Unable to set osd_memory_target on smithi173 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-04-25T17:46:12.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:12 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:12.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:12 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:13.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:13 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:13 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:13 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:13 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:13 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:46:13.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:13 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:46:13.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:13 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:13.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:13 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:13 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:13 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:13 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:13 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:46:13.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:13 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:46:13.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:13 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:13.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:13 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:13 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:13 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:13 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T17:46:13.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:13 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:46:13.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:13 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:46:13.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:13 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:46:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:14 smithi173 ceph-mon[32856]: pgmap v323: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 26 MiB/s wr, 1.85k op/s 2024-04-25T17:46:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:14 smithi173 ceph-mon[32856]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T17:46:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:14 smithi173 ceph-mon[32856]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T17:46:14.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:14 smithi098 ceph-mon[29192]: pgmap v323: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 26 MiB/s wr, 1.85k op/s 2024-04-25T17:46:14.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:14 smithi098 ceph-mon[29192]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T17:46:14.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:14 smithi098 ceph-mon[29192]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T17:46:14.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:14 smithi203 ceph-mon[32834]: pgmap v323: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 26 MiB/s wr, 1.85k op/s 2024-04-25T17:46:14.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:14 smithi203 ceph-mon[32834]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T17:46:14.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:14 smithi203 ceph-mon[32834]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T17:46:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:16 smithi173 ceph-mon[32856]: pgmap v324: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 22 MiB/s wr, 1.60k op/s 2024-04-25T17:46:16.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 585.0 s, 855.5 tps, lat 40.536 ms stddev 48.470 2024-04-25T17:46:16.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:16 smithi098 ceph-mon[29192]: pgmap v324: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 22 MiB/s wr, 1.60k op/s 2024-04-25T17:46:16.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:16 smithi203 ceph-mon[32834]: pgmap v324: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 22 MiB/s wr, 1.60k op/s 2024-04-25T17:46:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:18 smithi173 ceph-mon[32856]: pgmap v325: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 22 MiB/s wr, 1.78k op/s 2024-04-25T17:46:18.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:18 smithi098 ceph-mon[29192]: pgmap v325: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 22 MiB/s wr, 1.78k op/s 2024-04-25T17:46:18.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:18 smithi203 ceph-mon[32834]: pgmap v325: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 22 MiB/s wr, 1.78k op/s 2024-04-25T17:46:20.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:20 smithi173 ceph-mon[32856]: pgmap v326: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 22 MiB/s wr, 1.78k op/s 2024-04-25T17:46:20.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:20 smithi098 ceph-mon[29192]: pgmap v326: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 22 MiB/s wr, 1.78k op/s 2024-04-25T17:46:20.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:20 smithi203 ceph-mon[32834]: pgmap v326: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 22 MiB/s wr, 1.78k op/s 2024-04-25T17:46:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 590.0 s, 1045.6 tps, lat 158.130 ms stddev 916.950 2024-04-25T17:46:22.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:22 smithi173 ceph-mon[32856]: pgmap v327: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 21 MiB/s wr, 1.72k op/s 2024-04-25T17:46:22.527 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:22 smithi203 ceph-mon[32834]: pgmap v327: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 21 MiB/s wr, 1.72k op/s 2024-04-25T17:46:22.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:22 smithi098 ceph-mon[29192]: pgmap v327: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 21 MiB/s wr, 1.72k op/s 2024-04-25T17:46:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.458+0000 7fa26d34f640 1 -- 172.21.15.98:0/573681143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa268112df0 msgr2=0x7fa2681151e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:23.457 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.458+0000 7fa26d34f640 1 --2- 172.21.15.98:0/573681143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa268112df0 0x7fa2681151e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa258009890 tx=0x7fa25802a390 comp rx=0 tx=0).stop 2024-04-25T17:46:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.458+0000 7fa26d34f640 1 -- 172.21.15.98:0/573681143 shutdown_connections 2024-04-25T17:46:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.458+0000 7fa26d34f640 1 --2- 172.21.15.98:0/573681143 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa268112df0 0x7fa2681151e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.458+0000 7fa26d34f640 1 --2- 172.21.15.98:0/573681143 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa268076da0 0x7fa2681128b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.458+0000 7fa26d34f640 1 --2- 172.21.15.98:0/573681143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa268076480 0x7fa268076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.458+0000 7fa26d34f640 1 -- 172.21.15.98:0/573681143 >> 172.21.15.98:0/573681143 conn(0x7fa2680ffc80 msgr2=0x7fa2681020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.460+0000 7fa26d34f640 1 -- 172.21.15.98:0/573681143 shutdown_connections 2024-04-25T17:46:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.461+0000 7fa26d34f640 1 -- 172.21.15.98:0/573681143 wait complete. 2024-04-25T17:46:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.461+0000 7fa26d34f640 1 Processor -- start 2024-04-25T17:46:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa26d34f640 1 -- start start 2024-04-25T17:46:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa26d34f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa268076480 0x7fa26819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa26d34f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa268076da0 0x7fa26819f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa26d34f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa268112df0 0x7fa2681a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa26d34f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa2681187c0 con 0x7fa268076da0 2024-04-25T17:46:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa26d34f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa268118640 con 0x7fa268076480 2024-04-25T17:46:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa26d34f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa268118940 con 0x7fa268112df0 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa266ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa268076480 0x7fa26819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa2667fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa268076da0 0x7fa26819f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa266ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa268076480 0x7fa26819ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:42198/0 (socket says 172.21.15.98:42198) 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa266ffd640 1 -- 172.21.15.98:0/725155212 learned_addr learned my addr 172.21.15.98:0/725155212 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.462+0000 7fa2677fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa268112df0 0x7fa2681a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.463+0000 7fa266ffd640 1 -- 172.21.15.98:0/725155212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa268112df0 msgr2=0x7fa2681a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.463+0000 7fa266ffd640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa268112df0 0x7fa2681a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.463+0000 7fa266ffd640 1 -- 172.21.15.98:0/725155212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa268076da0 msgr2=0x7fa26819f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.463+0000 7fa266ffd640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa268076da0 0x7fa26819f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.463+0000 7fa266ffd640 1 -- 172.21.15.98:0/725155212 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2681a3d30 con 0x7fa268076480 2024-04-25T17:46:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.463+0000 7fa2677fe640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa268112df0 0x7fa2681a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.463+0000 7fa266ffd640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa268076480 0x7fa26819ed40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa26000c420 tx=0x7fa260009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.463+0000 7fa24ffff640 1 -- 172.21.15.98:0/725155212 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa260003970 con 0x7fa268076480 2024-04-25T17:46:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.463+0000 7fa26d34f640 1 -- 172.21.15.98:0/725155212 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2681a4020 con 0x7fa268076480 2024-04-25T17:46:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.464+0000 7fa26d34f640 1 -- 172.21.15.98:0/725155212 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2681ab8b0 con 0x7fa268076480 2024-04-25T17:46:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.464+0000 7fa24ffff640 1 -- 172.21.15.98:0/725155212 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2600097e0 con 0x7fa268076480 2024-04-25T17:46:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.464+0000 7fa24ffff640 1 -- 172.21.15.98:0/725155212 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26000b7d0 con 0x7fa268076480 2024-04-25T17:46:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.467+0000 7fa24ffff640 1 -- 172.21.15.98:0/725155212 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa26001d020 con 0x7fa268076480 2024-04-25T17:46:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.468+0000 7fa24ffff640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa2500a3f10 0x7fa2500a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.469+0000 7fa27006f740 1 Processor -- start 2024-04-25T17:46:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.469+0000 7fa24ffff640 1 -- 172.21.15.98:0/725155212 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fa2600d0520 con 0x7fa268076480 2024-04-25T17:46:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.469+0000 7fa2667fc640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa2500a3f10 0x7fa2500a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.470+0000 7fa2667fc640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa2500a3f10 0x7fa2500a63d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa2680794a0 tx=0x7fa25c00a2d0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.470+0000 7fa27006f740 1 -- start start 2024-04-25T17:46:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa27006f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844f620 0x5649f844fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa27006f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844ff40 0x5649f845a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa27006f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 0x5649f845cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa27006f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5649f83b4970 con 0x5649f844f620 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa27006f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5649f83514a0 con 0x5649f844ff40 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa27006f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5649f83b0930 con 0x5649f845a800 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa266ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844f620 0x5649f844fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa266ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844f620 0x5649f844fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56944/0 (socket says 172.21.15.98:56944) 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa266ffd640 1 -- 172.21.15.98:0/1758267845 learned_addr learned my addr 172.21.15.98:0/1758267845 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa2667fc640 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844ff40 0x5649f845a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa2677fe640 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 0x5649f845cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa2677fe640 1 -- 172.21.15.98:0/1758267845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844ff40 msgr2=0x5649f845a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.471+0000 7fa2677fe640 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844ff40 0x5649f845a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.472+0000 7fa2677fe640 1 -- 172.21.15.98:0/1758267845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844f620 msgr2=0x5649f844fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.472+0000 7fa2677fe640 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844f620 0x5649f844fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.472+0000 7fa2677fe640 1 -- 172.21.15.98:0/1758267845 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649f80c7210 con 0x5649f845a800 2024-04-25T17:46:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.472+0000 7fa266ffd640 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844f620 0x5649f844fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.472+0000 7fa2667fc640 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844ff40 0x5649f845a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.472+0000 7fa2677fe640 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 0x5649f845cbf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa258035040 tx=0x7fa258009460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bc0bb21c578ec5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.472+0000 7fa24d7fa640 1 -- 172.21.15.98:0/1758267845 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa258042020 con 0x5649f845a800 2024-04-25T17:46:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.472+0000 7fa24d7fa640 1 -- 172.21.15.98:0/1758267845 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa258038040 con 0x5649f845a800 2024-04-25T17:46:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.472+0000 7fa24d7fa640 1 -- 172.21.15.98:0/1758267845 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa258037040 con 0x5649f845a800 2024-04-25T17:46:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.473+0000 7fa27006f740 1 -- 172.21.15.98:0/1758267845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 msgr2=0x5649f845cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.473+0000 7fa27006f740 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 0x5649f845cbf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa258035040 tx=0x7fa258009460 comp rx=0 tx=0).stop 2024-04-25T17:46:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.473+0000 7fa27006f740 1 -- 172.21.15.98:0/1758267845 shutdown_connections 2024-04-25T17:46:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.473+0000 7fa27006f740 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 0x5649f845cbf0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.473+0000 7fa27006f740 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844ff40 0x5649f845a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.473+0000 7fa27006f740 1 --2- 172.21.15.98:0/1758267845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844f620 0x5649f844fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.473+0000 7fa27006f740 1 -- 172.21.15.98:0/1758267845 >> 172.21.15.98:0/1758267845 conn(0x5649f844bb30 msgr2=0x5649f844bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.473+0000 7fa27006f740 1 -- 172.21.15.98:0/1758267845 shutdown_connections 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.473+0000 7fa27006f740 1 -- 172.21.15.98:0/1758267845 wait complete. 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa27006f740 1 -- start start 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa27006f740 1 Processor -- start 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa27006f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f845a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa27006f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844ff40 0x5649f84573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa27006f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 0x5649f8457930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa27006f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5649f845d3d0 con 0x5649f844ff40 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa27006f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5649f834f0b0 con 0x5649f844f620 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa27006f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5649f845d550 con 0x5649f845a800 2024-04-25T17:46:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa27006f740 10 client.0 _ll_register_callbacks cb 0x5649f833cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa2667fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844ff40 0x5649f84573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa2667fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844ff40 0x5649f84573f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56948/0 (socket says 172.21.15.98:56948) 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa2667fc640 1 -- 172.21.15.98:0/32601165 learned_addr learned my addr 172.21.15.98:0/32601165 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa266ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f845a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.474+0000 7fa2677fe640 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 0x5649f8457930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa2667fc640 1 -- 172.21.15.98:0/32601165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 msgr2=0x5649f8457930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa2667fc640 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 0x5649f8457930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa2667fc640 1 -- 172.21.15.98:0/32601165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 msgr2=0x5649f845a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa2667fc640 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f845a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa2667fc640 1 -- 172.21.15.98:0/32601165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649f8193620 con 0x5649f844ff40 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa2677fe640 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 0x5649f8457930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa2667fc640 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844ff40 0x5649f84573f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa25c010e00 tx=0x7fa25c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf951770ac2c5ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa24aff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa24aff5640 1 -- 172.21.15.98:0/32601165 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c02a060 con 0x5649f844ff40 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa24aff5640 1 -- 172.21.15.98:0/32601165 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fa25c01c920 con 0x5649f844ff40 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa27006f740 1 -- 172.21.15.98:0/32601165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa258038040 con 0x5649f844ff40 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa24aff5640 1 -- 172.21.15.98:0/32601165 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa25c016070 con 0x5649f844ff40 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa27006f740 10 client.16419 fetch_fsmap learned FSMap version 44 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa27006f740 1 -- 172.21.15.98:0/32601165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5649f8469370 con 0x5649f844ff40 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.475+0000 7fa24aff5640 1 -- 172.21.15.98:0/32601165 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa258038040 con 0x5649f844ff40 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.476+0000 7fa24aff5640 1 -- 172.21.15.98:0/32601165 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fa25c025ca0 con 0x5649f844ff40 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.476+0000 7fa27006f740 10 client.16419 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.476+0000 7fa27006f740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.476+0000 7fa27006f740 20 client.16419 populate_metadata read hostname 'smithi098' 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.476+0000 7fa27006f740 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5649f846a920 0x5649f846cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.476+0000 7fa27006f740 4 client.16419 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:46:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.476+0000 7fa27006f740 1 -- 172.21.15.98:0/32601165 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649f83c2390 con 0x5649f846a920 2024-04-25T17:46:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.476+0000 7fa266ffd640 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5649f846a920 0x5649f846cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.476+0000 7fa24aff5640 10 client.16419 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.476+0000 7fa266ffd640 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5649f846a920 0x5649f846cd30 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56603a4cb5a7a075 server_cookie=379ce5dfdcfccf18 in_seq=0 out_seq=0 2024-04-25T17:46:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.477+0000 7fa24aff5640 1 -- 172.21.15.98:0/32601165 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649f83c2390 con 0x5649f846a920 2024-04-25T17:46:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.477+0000 7fa24aff5640 10 client.16419 handle_command_reply: tid=0 2024-04-25T17:46:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.477+0000 7fa27006f740 1 client.16419 shutdown 2024-04-25T17:46:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.478+0000 7fa27006f740 1 -- 172.21.15.98:0/32601165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844ff40 msgr2=0x5649f84573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.478+0000 7fa27006f740 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844ff40 0x5649f84573f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa25c010e00 tx=0x7fa25c004730 comp rx=0 tx=0).stop 2024-04-25T17:46:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.478+0000 7fa27006f740 1 -- 172.21.15.98:0/32601165 shutdown_connections 2024-04-25T17:46:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.478+0000 7fa27006f740 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5649f846a920 0x5649f846cd30 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.478+0000 7fa24aff5640 0 client.16419 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.478+0000 7fa27006f740 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f845a800 0x5649f8457930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.478+0000 7fa27006f740 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f844ff40 0x5649f84573f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.478+0000 7fa27006f740 1 --2- 172.21.15.98:0/32601165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f845a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.478+0000 7fa27006f740 1 -- 172.21.15.98:0/32601165 >> 172.21.15.98:0/32601165 conn(0x5649f844bb30 msgr2=0x5649f84518b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.478+0000 7fa27006f740 1 -- 172.21.15.98:0/32601165 shutdown_connections 2024-04-25T17:46:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.479+0000 7fa27006f740 1 -- 172.21.15.98:0/32601165 wait complete. 2024-04-25T17:46:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.479+0000 7fa27006f740 20 client.16419 trim_cache size 0 max 16384 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa27006f740 1 Processor -- start 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa27006f740 1 -- start start 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa27006f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f84662c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa27006f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 0x5649f8452970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa27006f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 0x5649f8459c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa27006f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa254011080 con 0x5649f845a800 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa27006f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa25400e8f0 con 0x5649f844f620 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa27006f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa25400c760 con 0x5649f844ff40 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa2677fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 0x5649f8459c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa266ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f84662c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa266ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f84662c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40940/0 (socket says 172.21.15.98:40940) 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa266ffd640 1 -- 172.21.15.98:0/2684626238 learned_addr learned my addr 172.21.15.98:0/2684626238 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.500+0000 7fa2667fc640 1 --2- 172.21.15.98:0/2684626238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 0x5649f8452970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.501+0000 7fa266ffd640 1 -- 172.21.15.98:0/2684626238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 msgr2=0x5649f8452970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.501+0000 7fa266ffd640 1 --2- 172.21.15.98:0/2684626238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 0x5649f8452970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.501+0000 7fa266ffd640 1 -- 172.21.15.98:0/2684626238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 msgr2=0x5649f8459c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.501+0000 7fa266ffd640 1 --2- 172.21.15.98:0/2684626238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 0x5649f8459c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.501+0000 7fa266ffd640 1 -- 172.21.15.98:0/2684626238 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649f844fd70 con 0x5649f844f620 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.501+0000 7fa266ffd640 1 --2- 172.21.15.98:0/2684626238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f84662c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa260096fd0 tx=0x7fa260000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d4d775bbc7ad05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.501+0000 7fa24cff9640 1 -- 172.21.15.98:0/2684626238 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26009a760 con 0x5649f844f620 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.501+0000 7fa24cff9640 1 -- 172.21.15.98:0/2684626238 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa260098ad0 con 0x5649f844f620 2024-04-25T17:46:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.501+0000 7fa24cff9640 1 -- 172.21.15.98:0/2684626238 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26009b8b0 con 0x5649f844f620 2024-04-25T17:46:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 -- 172.21.15.98:0/2684626238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 msgr2=0x5649f84662c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 --2- 172.21.15.98:0/2684626238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f84662c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa260096fd0 tx=0x7fa260000f40 comp rx=0 tx=0).stop 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 -- 172.21.15.98:0/2684626238 shutdown_connections 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 --2- 172.21.15.98:0/2684626238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 0x5649f8459c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 --2- 172.21.15.98:0/2684626238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 0x5649f8452970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 --2- 172.21.15.98:0/2684626238 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f84662c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 -- 172.21.15.98:0/2684626238 >> 172.21.15.98:0/2684626238 conn(0x5649f844bb30 msgr2=0x5649f8452260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 -- 172.21.15.98:0/2684626238 shutdown_connections 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 -- 172.21.15.98:0/2684626238 wait complete. 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 -- start start 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 Processor -- start 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f845e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.502+0000 7fa27006f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 0x5649f845e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa27006f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 0x5649f845ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa27006f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa25400cdb0 con 0x5649f845a800 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa27006f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa25400e220 con 0x5649f844f620 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa27006f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa25400d480 con 0x5649f844ff40 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa2677fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 0x5649f845ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa27006f740 10 client.0 _ll_register_callbacks cb 0x7fa25c010ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa2677fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 0x5649f845ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56964/0 (socket says 172.21.15.98:56964) 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa2677fe640 1 -- 172.21.15.98:0/2680043930 learned_addr learned my addr 172.21.15.98:0/2680043930 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa266ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f845e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa2667fc640 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 0x5649f845e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa266ffd640 1 -- 172.21.15.98:0/2680043930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 msgr2=0x5649f845e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa266ffd640 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 0x5649f845e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa266ffd640 1 -- 172.21.15.98:0/2680043930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 msgr2=0x5649f845ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa266ffd640 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 0x5649f845ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa266ffd640 1 -- 172.21.15.98:0/2680043930 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649f84cff60 con 0x5649f844f620 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa2677fe640 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 0x5649f845ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa2667fc640 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 0x5649f845e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa266ffd640 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f845e290 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa260004650 tx=0x7fa260009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb25861bac7c99c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa24aff5640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa24aff5640 1 -- 172.21.15.98:0/2680043930 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2600ce090 con 0x5649f844f620 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa27006f740 1 -- 172.21.15.98:0/2680043930 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa260098ad0 con 0x5649f844f620 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.503+0000 7fa24aff5640 1 -- 172.21.15.98:0/2680043930 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fa2600b5be0 con 0x5649f844f620 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.504+0000 7fa24aff5640 1 -- 172.21.15.98:0/2680043930 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa2600b5de0 con 0x5649f844f620 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.504+0000 7fa27006f740 10 client.25910 fetch_fsmap learned FSMap version 44 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.504+0000 7fa27006f740 1 -- 172.21.15.98:0/2680043930 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5649f84d1250 con 0x5649f844f620 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.505+0000 7fa24aff5640 1 -- 172.21.15.98:0/2680043930 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa260098ad0 con 0x5649f844f620 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.506+0000 7fa24aff5640 1 -- 172.21.15.98:0/2680043930 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fa26000b040 con 0x5649f844f620 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.506+0000 7fa27006f740 10 client.25910 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.506+0000 7fa27006f740 10 client.25910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:46:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.506+0000 7fa27006f740 20 client.25910 populate_metadata read hostname 'smithi098' 2024-04-25T17:46:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.506+0000 7fa27006f740 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5649f846a920 0x5649f84d3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.506+0000 7fa27006f740 4 client.25910 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:46:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.506+0000 7fa27006f740 1 -- 172.21.15.98:0/2680043930 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649f8457cc0 con 0x5649f846a920 2024-04-25T17:46:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.506+0000 7fa2667fc640 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5649f846a920 0x5649f84d3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.507+0000 7fa24aff5640 10 client.25910 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.507+0000 7fa2667fc640 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5649f846a920 0x5649f84d3830 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d7af66df1a7f2e0 server_cookie=ec4e81847afc0254 in_seq=0 out_seq=0 2024-04-25T17:46:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.530+0000 7fa24aff5640 1 -- 172.21.15.98:0/2680043930 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649f8457cc0 con 0x5649f846a920 2024-04-25T17:46:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.530+0000 7fa24aff5640 10 client.25910 handle_command_reply: tid=0 2024-04-25T17:46:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.530+0000 7fa27006f740 1 client.25910 shutdown 2024-04-25T17:46:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.530+0000 7fa27006f740 1 -- 172.21.15.98:0/2680043930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 msgr2=0x5649f845e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.530+0000 7fa27006f740 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f845e290 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa260004650 tx=0x7fa260009a20 comp rx=0 tx=0).stop 2024-04-25T17:46:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.531+0000 7fa27006f740 1 -- 172.21.15.98:0/2680043930 shutdown_connections 2024-04-25T17:46:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.531+0000 7fa27006f740 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5649f846a920 0x5649f84d3830 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.531+0000 7fa24aff5640 0 client.25910 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.531+0000 7fa27006f740 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649f845a800 0x5649f845ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.531+0000 7fa27006f740 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649f844ff40 0x5649f845e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.531+0000 7fa27006f740 1 --2- 172.21.15.98:0/2680043930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649f844f620 0x5649f845e290 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.531+0000 7fa27006f740 1 -- 172.21.15.98:0/2680043930 >> 172.21.15.98:0/2680043930 conn(0x5649f844bb30 msgr2=0x5649f844acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.531+0000 7fa27006f740 1 -- 172.21.15.98:0/2680043930 shutdown_connections 2024-04-25T17:46:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.531+0000 7fa27006f740 1 -- 172.21.15.98:0/2680043930 wait complete. 2024-04-25T17:46:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.532+0000 7fa27006f740 20 client.25910 trim_cache size 0 max 16384 2024-04-25T17:46:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:46:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:46:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:46:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:46:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 -- 172.21.15.98:0/725155212 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa2500a3f10 msgr2=0x7fa2500a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa2500a3f10 0x7fa2500a63d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa2680794a0 tx=0x7fa25c00a2d0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 -- 172.21.15.98:0/725155212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa268076480 msgr2=0x7fa26819ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa268076480 0x7fa26819ed40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa26000c420 tx=0x7fa260009f90 comp rx=0 tx=0).stop 2024-04-25T17:46:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 -- 172.21.15.98:0/725155212 shutdown_connections 2024-04-25T17:46:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa2500a3f10 0x7fa2500a63d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa268112df0 0x7fa2681a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa268076da0 0x7fa26819f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 --2- 172.21.15.98:0/725155212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa268076480 0x7fa26819ed40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 -- 172.21.15.98:0/725155212 >> 172.21.15.98:0/725155212 conn(0x7fa2680ffc80 msgr2=0x7fa268113a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 -- 172.21.15.98:0/725155212 shutdown_connections 2024-04-25T17:46:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:23.533+0000 7fa26d34f640 1 -- 172.21.15.98:0/725155212 wait complete. 2024-04-25T17:46:24.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:24 smithi098 ceph-mon[29192]: pgmap v328: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 20 MiB/s wr, 1.58k op/s 2024-04-25T17:46:24.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:24 smithi203 ceph-mon[32834]: pgmap v328: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 20 MiB/s wr, 1.58k op/s 2024-04-25T17:46:24.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:24 smithi173 ceph-mon[32856]: pgmap v328: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 20 MiB/s wr, 1.58k op/s 2024-04-25T17:46:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 595.0 s, 2554.6 tps, lat 39.208 ms stddev 43.616 2024-04-25T17:46:26.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:26 smithi098 ceph-mon[29192]: pgmap v329: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-25T17:46:26.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:26 smithi203 ceph-mon[32834]: pgmap v329: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-25T17:46:26.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:26 smithi173 ceph-mon[32856]: pgmap v329: 97 pgs: 97 active+clean; 4.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 15 MiB/s wr, 1.27k op/s 2024-04-25T17:46:28.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:28 smithi098 ceph-mon[29192]: pgmap v330: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 18 MiB/s wr, 1.55k op/s 2024-04-25T17:46:28.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:28 smithi203 ceph-mon[32834]: pgmap v330: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 18 MiB/s wr, 1.55k op/s 2024-04-25T17:46:28.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:28 smithi173 ceph-mon[32856]: pgmap v330: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 18 MiB/s wr, 1.55k op/s 2024-04-25T17:46:30.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:30 smithi098 ceph-mon[29192]: pgmap v331: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.7 MiB/s wr, 374 op/s 2024-04-25T17:46:30.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:30 smithi203 ceph-mon[32834]: pgmap v331: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.7 MiB/s wr, 374 op/s 2024-04-25T17:46:30.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:30 smithi173 ceph-mon[32856]: pgmap v331: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.7 MiB/s wr, 374 op/s 2024-04-25T17:46:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 600.0 s, 2214.0 tps, lat 45.074 ms stddev 54.048 2024-04-25T17:46:32.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:32 smithi098 ceph-mon[29192]: pgmap v332: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.7 MiB/s wr, 374 op/s 2024-04-25T17:46:32.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:32 smithi203 ceph-mon[32834]: pgmap v332: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.7 MiB/s wr, 374 op/s 2024-04-25T17:46:32.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:32 smithi173 ceph-mon[32856]: pgmap v332: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 4.7 MiB/s wr, 374 op/s 2024-04-25T17:46:34.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:34 smithi098 ceph-mon[29192]: pgmap v333: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 8.5 MiB/s wr, 660 op/s 2024-04-25T17:46:34.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:34 smithi203 ceph-mon[32834]: pgmap v333: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 8.5 MiB/s wr, 660 op/s 2024-04-25T17:46:34.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:34 smithi173 ceph-mon[32856]: pgmap v333: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 8.5 MiB/s wr, 660 op/s 2024-04-25T17:46:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 605.0 s, 2453.8 tps, lat 40.761 ms stddev 48.917 2024-04-25T17:46:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.661+0000 7fe2fede2640 1 -- 172.21.15.98:0/4151921347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f81005c0 msgr2=0x7fe2f8100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.661+0000 7fe2fede2640 1 --2- 172.21.15.98:0/4151921347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f81005c0 0x7fe2f8100a40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000a9a0 tx=0x7fe2e002b690 comp rx=0 tx=0).stop 2024-04-25T17:46:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.661+0000 7fe2fede2640 1 -- 172.21.15.98:0/4151921347 shutdown_connections 2024-04-25T17:46:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.661+0000 7fe2fede2640 1 --2- 172.21.15.98:0/4151921347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2f8100f80 0x7fe2f810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.661+0000 7fe2fede2640 1 --2- 172.21.15.98:0/4151921347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f81005c0 0x7fe2f8100a40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.661+0000 7fe2fede2640 1 --2- 172.21.15.98:0/4151921347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2f8106680 0x7fe2f8106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.661+0000 7fe2fede2640 1 -- 172.21.15.98:0/4151921347 >> 172.21.15.98:0/4151921347 conn(0x7fe2f80773e0 msgr2=0x7fe2f80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.662+0000 7fe2fede2640 1 -- 172.21.15.98:0/4151921347 shutdown_connections 2024-04-25T17:46:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.662+0000 7fe2fede2640 1 -- 172.21.15.98:0/4151921347 wait complete. 2024-04-25T17:46:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.662+0000 7fe2fede2640 1 Processor -- start 2024-04-25T17:46:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fede2640 1 -- start start 2024-04-25T17:46:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fede2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2f81005c0 0x7fe2f81a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fede2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f8100f80 0x7fe2f81a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fede2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2f8106680 0x7fe2f819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fede2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f810fb60 con 0x7fe2f81005c0 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fede2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f810f9e0 con 0x7fe2f8106680 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fede2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f810fce0 con 0x7fe2f8100f80 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fcb7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2f81005c0 0x7fe2f81a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fcb7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2f81005c0 0x7fe2f81a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34860/0 (socket says 172.21.15.98:34860) 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/83077992 learned_addr learned my addr 172.21.15.98:0/83077992 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f8100f80 0x7fe2f81a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2fd37f640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2f8106680 0x7fe2f819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2f7fff640 1 -- 172.21.15.98:0/83077992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2f8106680 msgr2=0x7fe2f819aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2f8106680 0x7fe2f819aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2f7fff640 1 -- 172.21.15.98:0/83077992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2f81005c0 msgr2=0x7fe2f81a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2f81005c0 0x7fe2f81a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.663+0000 7fe2f7fff640 1 -- 172.21.15.98:0/83077992 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f819b370 con 0x7fe2f8100f80 2024-04-25T17:46:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.664+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f8100f80 0x7fe2f81a0ab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000a9a0 tx=0x7fe2e0032a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.664+0000 7fe2f5ffb640 1 -- 172.21.15.98:0/83077992 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00051c0 con 0x7fe2f8100f80 2024-04-25T17:46:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.664+0000 7fe2f5ffb640 1 -- 172.21.15.98:0/83077992 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e00309c0 con 0x7fe2f8100f80 2024-04-25T17:46:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.664+0000 7fe2fede2640 1 -- 172.21.15.98:0/83077992 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f819b600 con 0x7fe2f8100f80 2024-04-25T17:46:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.664+0000 7fe2f5ffb640 1 -- 172.21.15.98:0/83077992 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0004a20 con 0x7fe2f8100f80 2024-04-25T17:46:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.664+0000 7fe2fede2640 1 -- 172.21.15.98:0/83077992 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f81a74f0 con 0x7fe2f8100f80 2024-04-25T17:46:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.666+0000 7fe2f5ffb640 1 -- 172.21.15.98:0/83077992 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe2e0030b70 con 0x7fe2f8100f80 2024-04-25T17:46:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.666+0000 7fe301bd1740 1 Processor -- start 2024-04-25T17:46:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.666+0000 7fe2f5ffb640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe2e40a3dc0 0x7fe2e40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.666+0000 7fe301bd1740 1 -- start start 2024-04-25T17:46:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2fd37f640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe2e40a3dc0 0x7fe2e40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2f5ffb640 1 -- 172.21.15.98:0/83077992 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fe2e00f19d0 con 0x7fe2f8100f80 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1eea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1fbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe301bd1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55588c153970 con 0x55588c1eef40 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe301bd1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55588c0f04a0 con 0x55588c1f9800 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe301bd1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55588c14f930 con 0x55588c1ee620 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2fd37f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1fbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2fcb7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34870/0 (socket says 172.21.15.98:34870) 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2fd37f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1fbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:59388/0 (socket says 172.21.15.98:59388) 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2f7fff640 1 -- 172.21.15.98:0/2091361689 learned_addr learned my addr 172.21.15.98:0/2091361689 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2f7fff640 1 -- 172.21.15.98:0/2091361689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 msgr2=0x55588c1eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/2091361689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1eea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2f7fff640 1 -- 172.21.15.98:0/2091361689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 msgr2=0x55588c1fbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/2091361689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1fbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.667+0000 7fe2f7fff640 1 -- 172.21.15.98:0/2091361689 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55588be66210 con 0x55588c1eef40 2024-04-25T17:46:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.668+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/2091361689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f92c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00ba2e0 tx=0x7fe2e00b9ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f72e4b5627704a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.668+0000 7fe2fd37f640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe2e40a3dc0 0x7fe2e40a6280 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8005fd0 tx=0x7fe2e8011bc0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.668+0000 7fe2eeffd640 1 -- 172.21.15.98:0/2091361689 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00c1cc0 con 0x55588c1eef40 2024-04-25T17:46:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.668+0000 7fe2eeffd640 1 -- 172.21.15.98:0/2091361689 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e00a6e50 con 0x55588c1eef40 2024-04-25T17:46:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.668+0000 7fe2eeffd640 1 -- 172.21.15.98:0/2091361689 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00a80c0 con 0x55588c1eef40 2024-04-25T17:46:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.669+0000 7fe301bd1740 1 -- 172.21.15.98:0/2091361689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 msgr2=0x55588c1f92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.669+0000 7fe301bd1740 1 --2- 172.21.15.98:0/2091361689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f92c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00ba2e0 tx=0x7fe2e00b9ce0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 -- 172.21.15.98:0/2091361689 shutdown_connections 2024-04-25T17:46:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 --2- 172.21.15.98:0/2091361689 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 --2- 172.21.15.98:0/2091361689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f92c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 --2- 172.21.15.98:0/2091361689 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 -- 172.21.15.98:0/2091361689 >> 172.21.15.98:0/2091361689 conn(0x55588c1eab30 msgr2=0x55588c1eaf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 -- 172.21.15.98:0/2091361689 shutdown_connections 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 -- 172.21.15.98:0/2091361689 wait complete. 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 -- start start 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 Processor -- start 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.670+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2fcb7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:34872/0 (socket says 172.21.15.98:34872) 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2fcb7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:51806/0 (socket says 172.21.15.98:51806) 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2f7fff640 1 -- 172.21.15.98:0/1093314086 learned_addr learned my addr 172.21.15.98:0/1093314086 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe301bd1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55588c1fc1f0 con 0x55588c1eef40 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe301bd1740 1 -- 172.21.15.98:0/1093314086 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55588c0ee0b0 con 0x55588c1f9800 2024-04-25T17:46:36.676 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe301bd1740 1 -- 172.21.15.98:0/1093314086 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55588c1fc370 con 0x55588c1ee620 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2fd37f640 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe301bd1740 10 client.0 _ll_register_callbacks cb 0x55588c0dbd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/1093314086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 msgr2=0x55588c1f69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2fcb7e640 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1f69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/1093314086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 msgr2=0x55588c1f6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2fcb7e640 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/1093314086 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55588bf32620 con 0x55588c1ee620 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2fcb7e640 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f9180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0012740 tx=0x7fe2f0012c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46aabdbe4c746201 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2d7fff640 1 -- 172.21.15.98:0/1093314086 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0004030 con 0x55588c1ee620 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.671+0000 7fe2d7fff640 1 -- 172.21.15.98:0/1093314086 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fe2f00041d0 con 0x55588c1ee620 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe301bd1740 1 -- 172.21.15.98:0/1093314086 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2e00a6e50 con 0x55588c1ee620 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe2d7fff640 1 -- 172.21.15.98:0/1093314086 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe2f0004410 con 0x55588c1ee620 2024-04-25T17:46:36.677 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe301bd1740 10 client.25942 fetch_fsmap learned FSMap version 44 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe301bd1740 1 -- 172.21.15.98:0/1093314086 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55588c207f00 con 0x55588c1ee620 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe2d7fff640 1 -- 172.21.15.98:0/1093314086 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e00a6e50 con 0x55588c1ee620 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe2d7fff640 1 -- 172.21.15.98:0/1093314086 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fe2f0003bc0 con 0x55588c1ee620 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe301bd1740 10 client.25942 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe301bd1740 10 client.25942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe301bd1740 20 client.25942 populate_metadata read hostname 'smithi098' 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe301bd1740 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55588c209370 0x55588c20b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe301bd1740 4 client.25942 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe301bd1740 1 -- 172.21.15.98:0/1093314086 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55588c161390 con 0x55588c209370 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.672+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55588c209370 0x55588c20b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.673+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55588c209370 0x55588c20b7a0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=394943520ca129c3 server_cookie=19c87e798cad17b8 in_seq=0 out_seq=0 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.673+0000 7fe2d7fff640 10 client.25942 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.674+0000 7fe2d7fff640 1 -- 172.21.15.98:0/1093314086 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55588c161390 con 0x55588c209370 2024-04-25T17:46:36.678 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.674+0000 7fe2d7fff640 10 client.25942 handle_command_reply: tid=0 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.674+0000 7fe301bd1740 1 client.25942 shutdown 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 1 -- 172.21.15.98:0/1093314086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 msgr2=0x55588c1f9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f9180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0012740 tx=0x7fe2f0012c00 comp rx=0 tx=0).stop 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 1 -- 172.21.15.98:0/1093314086 shutdown_connections 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55588c209370 0x55588c20b7a0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe2d7fff640 0 client.25942 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c1f6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 1 --2- 172.21.15.98:0/1093314086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f9180 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 1 -- 172.21.15.98:0/1093314086 >> 172.21.15.98:0/1093314086 conn(0x55588c1eab30 msgr2=0x55588c1e9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 1 -- 172.21.15.98:0/1093314086 shutdown_connections 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 1 -- 172.21.15.98:0/1093314086 wait complete. 2024-04-25T17:46:36.679 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.675+0000 7fe301bd1740 20 client.25942 trim_cache size 0 max 16384 2024-04-25T17:46:36.703 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.703+0000 7fe301bd1740 1 Processor -- start 2024-04-25T17:46:36.703 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.704+0000 7fe301bd1740 1 -- start start 2024-04-25T17:46:36.704 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.704+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.704 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c205020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.704 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c2110b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.704 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe301bd1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81b25c0 con 0x55588c1eef40 2024-04-25T17:46:36.705 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe301bd1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81afe30 con 0x55588c1f9800 2024-04-25T17:46:36.705 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe301bd1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81adca0 con 0x55588c1ee620 2024-04-25T17:46:36.705 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2fcb7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.705 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2fcb7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:51820/0 (socket says 172.21.15.98:51820) 2024-04-25T17:46:36.705 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/3128439028 learned_addr learned my addr 172.21.15.98:0/3128439028 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:36.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:36 smithi098 ceph-mon[29192]: pgmap v334: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 7.3 MiB/s wr, 570 op/s 2024-04-25T17:46:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/3128439028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c205020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.706 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2fd37f640 1 --2- 172.21.15.98:0/3128439028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c2110b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/3128439028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 msgr2=0x55588c2110b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2fcb7e640 1 --2- 172.21.15.98:0/3128439028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c2110b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/3128439028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 msgr2=0x55588c205020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2fcb7e640 1 --2- 172.21.15.98:0/3128439028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c205020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/3128439028 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55588c205b90 con 0x55588c1ee620 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/3128439028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c205020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.705+0000 7fe2fcb7e640 1 --2- 172.21.15.98:0/3128439028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f6410 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009340 tx=0x7fe2f0005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcf2e04526fbd7b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.706+0000 7fe2ee7fc640 1 -- 172.21.15.98:0/3128439028 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f00105a0 con 0x55588c1ee620 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.706+0000 7fe2ee7fc640 1 -- 172.21.15.98:0/3128439028 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2f0012e50 con 0x55588c1ee620 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.706+0000 7fe2ee7fc640 1 -- 172.21.15.98:0/3128439028 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f000f9f0 con 0x55588c1ee620 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.706+0000 7fe301bd1740 1 -- 172.21.15.98:0/3128439028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 msgr2=0x55588c1f6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.706+0000 7fe301bd1740 1 --2- 172.21.15.98:0/3128439028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f6410 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009340 tx=0x7fe2f0005970 comp rx=0 tx=0).stop 2024-04-25T17:46:36.707 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.707+0000 7fe301bd1740 1 -- 172.21.15.98:0/3128439028 shutdown_connections 2024-04-25T17:46:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.707+0000 7fe301bd1740 1 --2- 172.21.15.98:0/3128439028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c2110b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.707+0000 7fe301bd1740 1 --2- 172.21.15.98:0/3128439028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c1eef40 0x55588c205020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.707+0000 7fe301bd1740 1 --2- 172.21.15.98:0/3128439028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1ee620 0x55588c1f6410 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009340 tx=0x7fe2f0005970 comp rx=0 tx=0).stop 2024-04-25T17:46:36.708 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.707+0000 7fe301bd1740 1 -- 172.21.15.98:0/3128439028 >> 172.21.15.98:0/3128439028 conn(0x55588c1eab30 msgr2=0x55588c1f5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:36.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.708+0000 7fe301bd1740 1 -- 172.21.15.98:0/3128439028 shutdown_connections 2024-04-25T17:46:36.709 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.708+0000 7fe301bd1740 1 -- 172.21.15.98:0/3128439028 wait complete. 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.708+0000 7fe301bd1740 1 -- start start 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.708+0000 7fe301bd1740 1 Processor -- start 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1eef40 0x55588c201930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1fec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe301bd1740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c209370 0x55588c1ff1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe301bd1740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81ae2f0 con 0x55588c209370 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe301bd1740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81af760 con 0x55588c1f9800 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe2fcb7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1eef40 0x55588c201930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe301bd1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f81ae9c0 con 0x55588c1eef40 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe2f7fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1fec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe2fcb7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1eef40 0x55588c201930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:51828/0 (socket says 172.21.15.98:51828) 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/972038311 learned_addr learned my addr 172.21.15.98:0/972038311 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe301bd1740 10 client.0 _ll_register_callbacks cb 0x7fe2f0009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:46:36.710 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe2fd37f640 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c209370 0x55588c1ff1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/972038311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 msgr2=0x55588c1fec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe2fcb7e640 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1fec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/972038311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c209370 msgr2=0x55588c1ff1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe2fcb7e640 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c209370 0x55588c1ff1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.709+0000 7fe2fcb7e640 1 -- 172.21.15.98:0/972038311 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55588c1ff9e0 con 0x55588c1eef40 2024-04-25T17:46:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.710+0000 7fe2fcb7e640 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1eef40 0x55588c201930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00093d0 tx=0x7fe2f0010b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=428032e0d318e134 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.710+0000 7fe2d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:46:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.710+0000 7fe2d7fff640 1 -- 172.21.15.98:0/972038311 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0003f90 con 0x55588c1eef40 2024-04-25T17:46:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.710+0000 7fe2d7fff640 1 -- 172.21.15.98:0/972038311 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fe2f000f790 con 0x55588c1eef40 2024-04-25T17:46:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.710+0000 7fe301bd1740 1 -- 172.21.15.98:0/972038311 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2f0012e50 con 0x55588c1eef40 2024-04-25T17:46:36.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:36 smithi203 ceph-mon[32834]: pgmap v334: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 7.3 MiB/s wr, 570 op/s 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.710+0000 7fe2d7fff640 1 -- 172.21.15.98:0/972038311 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fe2f000f990 con 0x55588c1eef40 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.710+0000 7fe301bd1740 10 client.25948 fetch_fsmap learned FSMap version 44 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.711+0000 7fe301bd1740 1 -- 172.21.15.98:0/972038311 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55588c26f980 con 0x55588c1eef40 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.711+0000 7fe2d7fff640 1 -- 172.21.15.98:0/972038311 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2f0012e50 con 0x55588c1eef40 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.711+0000 7fe2d7fff640 1 -- 172.21.15.98:0/972038311 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fe2f0039440 con 0x55588c1eef40 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.711+0000 7fe301bd1740 10 client.25948 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.711+0000 7fe301bd1740 10 client.25948 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.711+0000 7fe301bd1740 20 client.25948 populate_metadata read hostname 'smithi098' 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.711+0000 7fe301bd1740 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55588c26ff80 0x55588c272370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.711+0000 7fe301bd1740 4 client.25948 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.711+0000 7fe301bd1740 1 -- 172.21.15.98:0/972038311 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55588c15b200 con 0x55588c26ff80 2024-04-25T17:46:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.711+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55588c26ff80 0x55588c272370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.712+0000 7fe2d7fff640 10 client.25948 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.712+0000 7fe2f7fff640 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55588c26ff80 0x55588c272370 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=190e5eeb18ba0c48 server_cookie=4c9e8bb9475563aa in_seq=0 out_seq=0 2024-04-25T17:46:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.727+0000 7fe2d7fff640 1 -- 172.21.15.98:0/972038311 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55588c15b200 con 0x55588c26ff80 2024-04-25T17:46:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.727+0000 7fe2d7fff640 10 client.25948 handle_command_reply: tid=0 2024-04-25T17:46:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.727+0000 7fe301bd1740 1 client.25948 shutdown 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.727+0000 7fe301bd1740 1 -- 172.21.15.98:0/972038311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1eef40 msgr2=0x55588c201930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.727+0000 7fe301bd1740 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1eef40 0x55588c201930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00093d0 tx=0x7fe2f0010b70 comp rx=0 tx=0).stop 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.728+0000 7fe301bd1740 1 -- 172.21.15.98:0/972038311 shutdown_connections 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.728+0000 7fe301bd1740 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55588c26ff80 0x55588c272370 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.728+0000 7fe301bd1740 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55588c209370 0x55588c1ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.728+0000 7fe301bd1740 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55588c1f9800 0x55588c1fec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.728+0000 7fe301bd1740 1 --2- 172.21.15.98:0/972038311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55588c1eef40 0x55588c201930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.728+0000 7fe301bd1740 1 -- 172.21.15.98:0/972038311 >> 172.21.15.98:0/972038311 conn(0x55588c1eab30 msgr2=0x55588c1e9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.728+0000 7fe2d7fff640 0 client.25948 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.728+0000 7fe301bd1740 1 -- 172.21.15.98:0/972038311 shutdown_connections 2024-04-25T17:46:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.728+0000 7fe301bd1740 1 -- 172.21.15.98:0/972038311 wait complete. 2024-04-25T17:46:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.728+0000 7fe301bd1740 20 client.25948 trim_cache size 0 max 16384 2024-04-25T17:46:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:46:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:46:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:46:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.729+0000 7fe2fede2640 1 -- 172.21.15.98:0/83077992 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe2e40a3dc0 msgr2=0x7fe2e40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.729+0000 7fe2fede2640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe2e40a3dc0 0x7fe2e40a6280 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8005fd0 tx=0x7fe2e8011bc0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.729+0000 7fe2fede2640 1 -- 172.21.15.98:0/83077992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f8100f80 msgr2=0x7fe2f81a0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.729+0000 7fe2fede2640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f8100f80 0x7fe2f81a0ab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000a9a0 tx=0x7fe2e0032a20 comp rx=0 tx=0).stop 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.729+0000 7fe2fede2640 1 -- 172.21.15.98:0/83077992 shutdown_connections 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.729+0000 7fe2fede2640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe2e40a3dc0 0x7fe2e40a6280 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.729+0000 7fe2fede2640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe2f8106680 0x7fe2f819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.729+0000 7fe2fede2640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f8100f80 0x7fe2f81a0ab0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.729+0000 7fe2fede2640 1 --2- 172.21.15.98:0/83077992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2f81005c0 0x7fe2f81a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.729+0000 7fe2fede2640 1 -- 172.21.15.98:0/83077992 >> 172.21.15.98:0/83077992 conn(0x7fe2f80773e0 msgr2=0x7fe2f80fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.730+0000 7fe2fede2640 1 -- 172.21.15.98:0/83077992 shutdown_connections 2024-04-25T17:46:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:36.730+0000 7fe2fede2640 1 -- 172.21.15.98:0/83077992 wait complete. 2024-04-25T17:46:36.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:36 smithi173 ceph-mon[32856]: pgmap v334: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 7.3 MiB/s wr, 570 op/s 2024-04-25T17:46:37.471 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.3 2024-04-25T17:46:37.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:46:37.816 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:46:38.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:38 smithi098 ceph-mon[29192]: pgmap v335: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 12 MiB/s wr, 933 op/s 2024-04-25T17:46:38.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:38 smithi203 ceph-mon[32834]: pgmap v335: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 12 MiB/s wr, 933 op/s 2024-04-25T17:46:38.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:38 smithi173 ceph-mon[32856]: pgmap v335: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 12 MiB/s wr, 933 op/s 2024-04-25T17:46:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.946+0000 7fe91d4e1700 1 -- 172.21.15.98:0/2919757095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9181026e0 msgr2=0x7fe91810e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.946+0000 7fe91d4e1700 1 --2- 172.21.15.98:0/2919757095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9181026e0 0x7fe91810e770 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe91000b0a0 tx=0x7fe91000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:46:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.947+0000 7fe91d4e1700 1 -- 172.21.15.98:0/2919757095 shutdown_connections 2024-04-25T17:46:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.947+0000 7fe91d4e1700 1 --2- 172.21.15.98:0/2919757095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe918106990 0x7fe918106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.947+0000 7fe91d4e1700 1 --2- 172.21.15.98:0/2919757095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9181026e0 0x7fe91810e770 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.947+0000 7fe91d4e1700 1 --2- 172.21.15.98:0/2919757095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe918101d20 0x7fe9181021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.947+0000 7fe91d4e1700 1 -- 172.21.15.98:0/2919757095 >> 172.21.15.98:0/2919757095 conn(0x7fe9180788f0 msgr2=0x7fe918078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.947+0000 7fe91d4e1700 1 -- 172.21.15.98:0/2919757095 shutdown_connections 2024-04-25T17:46:39.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.947+0000 7fe91d4e1700 1 -- 172.21.15.98:0/2919757095 wait complete. 2024-04-25T17:46:39.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.948+0000 7fe91d4e1700 1 Processor -- start 2024-04-25T17:46:39.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe91d4e1700 1 -- start start 2024-04-25T17:46:39.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe91d4e1700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe918101d20 0x7fe9181a42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:39.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe91d4e1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9181026e0 0x7fe9181a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:39.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe91d4e1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918106990 0x7fe91819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe9167fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9181026e0 0x7fe9181a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe9167fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9181026e0 0x7fe9181a47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:59422/0 (socket says 172.21.15.98:59422) 2024-04-25T17:46:39.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe9167fc700 1 -- 172.21.15.98:0/926521885 learned_addr learned my addr 172.21.15.98:0/926521885 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe918111ae0 con 0x7fe918106990 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe918111960 con 0x7fe9181026e0 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe918111c60 con 0x7fe918101d20 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe916ffd700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe918101d20 0x7fe9181a42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.949+0000 7fe9177fe700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918106990 0x7fe91819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe9167fc700 1 -- 172.21.15.98:0/926521885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe918101d20 msgr2=0x7fe9181a42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe9167fc700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe918101d20 0x7fe9181a42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe9167fc700 1 -- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918106990 msgr2=0x7fe91819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe9167fc700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918106990 0x7fe91819f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe9167fc700 1 -- 172.21.15.98:0/926521885 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91819ff50 con 0x7fe9181026e0 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe9177fe700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918106990 0x7fe91819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe9167fc700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9181026e0 0x7fe9181a47e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00d980 tx=0x7fe90c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe8fffff700 1 -- 172.21.15.98:0/926521885 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c016070 con 0x7fe9181026e0 2024-04-25T17:46:39.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9181a0240 con 0x7fe9181026e0 2024-04-25T17:46:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe8fffff700 1 -- 172.21.15.98:0/926521885 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe90c004830 con 0x7fe9181026e0 2024-04-25T17:46:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe8fffff700 1 -- 172.21.15.98:0/926521885 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c005540 con 0x7fe9181026e0 2024-04-25T17:46:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.950+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9181ac1d0 con 0x7fe9181026e0 2024-04-25T17:46:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.951+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe918103080 con 0x7fe9181026e0 2024-04-25T17:46:39.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.953+0000 7fe8fffff700 1 -- 172.21.15.98:0/926521885 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe90c004220 con 0x7fe9181026e0 2024-04-25T17:46:39.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.954+0000 7fe8fffff700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe9000a3e30 0x7fe9000a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:39.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.954+0000 7fe8fffff700 1 -- 172.21.15.98:0/926521885 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fe90c0d0f90 con 0x7fe9181026e0 2024-04-25T17:46:39.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.955+0000 7fe916ffd700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe9000a3e30 0x7fe9000a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:39.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.956+0000 7fe916ffd700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe9000a3e30 0x7fe9000a62f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe918078f30 tx=0x7fe90800b3c0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:39.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:39.958+0000 7fe8fffff700 1 -- 172.21.15.98:0/926521885 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe90c01f020 con 0x7fe9181026e0 2024-04-25T17:46:40.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.381+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe918069730 con 0x7fe9181026e0 2024-04-25T17:46:40.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.383+0000 7fe8fffff700 1 -- 172.21.15.98:0/926521885 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11883 (secure 0 0 0) 0x7fe918069730 con 0x7fe9181026e0 2024-04-25T17:46:40.383 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:46:40.383 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:45:52.737355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe9000a3e30 msgr2=0x7fe9000a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe9000a3e30 0x7fe9000a62f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe918078f30 tx=0x7fe90800b3c0 comp rx=0 tx=0).stop 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9181026e0 msgr2=0x7fe9181a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9181026e0 0x7fe9181a47e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe90c00d980 tx=0x7fe90c00dc90 comp rx=0 tx=0).stop 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 shutdown_connections 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe9000a3e30 0x7fe9000a62f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe918101d20 0x7fe9181a42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe918106990 0x7fe91819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 --2- 172.21.15.98:0/926521885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9181026e0 0x7fe9181a47e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 >> 172.21.15.98:0/926521885 conn(0x7fe9180788f0 msgr2=0x7fe91806b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 shutdown_connections 2024-04-25T17:46:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:40.386+0000 7fe91d4e1700 1 -- 172.21.15.98:0/926521885 wait complete. 2024-04-25T17:46:40.387 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 44 2024-04-25T17:46:40.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:40 smithi098 ceph-mon[29192]: pgmap v336: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.7 MiB/s wr, 648 op/s 2024-04-25T17:46:40.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:40 smithi203 ceph-mon[32834]: pgmap v336: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.7 MiB/s wr, 648 op/s 2024-04-25T17:46:40.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:40 smithi173 ceph-mon[32856]: pgmap v336: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.7 MiB/s wr, 648 op/s 2024-04-25T17:46:41.436 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:41 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/926521885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:46:41.479 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 78.3 --await-for 42.0' 2024-04-25T17:46:41.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 78.3 --await-for 42.0 2024-04-25T17:46:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 610.0 s, 3090.2 tps, lat 32.413 ms stddev 36.956 2024-04-25T17:46:41.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:41 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/926521885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:46:41.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:41 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/926521885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:46:41.863 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:46:42.640 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:42 smithi098 ceph-mon[29192]: pgmap v337: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.7 MiB/s wr, 648 op/s 2024-04-25T17:46:42.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:42 smithi203 ceph-mon[32834]: pgmap v337: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.7 MiB/s wr, 648 op/s 2024-04-25T17:46:42.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:42 smithi173 ceph-mon[32856]: pgmap v337: 97 pgs: 97 active+clean; 4.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.7 MiB/s wr, 648 op/s 2024-04-25T17:46:43.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.965+0000 7f85f05fd700 1 -- 172.21.15.98:0/804256058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85e8105fb0 msgr2=0x7f85e810db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:43.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.965+0000 7f85f05fd700 1 --2- 172.21.15.98:0/804256058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85e8105fb0 0x7f85e810db80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f85e000b0d0 tx=0x7f85e000b3e0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.965+0000 7f85f05fd700 1 -- 172.21.15.98:0/804256058 shutdown_connections 2024-04-25T17:46:43.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.965+0000 7f85f05fd700 1 --2- 172.21.15.98:0/804256058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85e8105fb0 0x7f85e810db80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.965+0000 7f85f05fd700 1 --2- 172.21.15.98:0/804256058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85e8100ed0 0x7f85e81012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.965+0000 7f85f05fd700 1 --2- 172.21.15.98:0/804256058 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e8101880 0x7f85e81058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.965+0000 7f85f05fd700 1 -- 172.21.15.98:0/804256058 >> 172.21.15.98:0/804256058 conn(0x7f85e8077b70 msgr2=0x7f85e80feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:43.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.966+0000 7f85f05fd700 1 -- 172.21.15.98:0/804256058 shutdown_connections 2024-04-25T17:46:43.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.966+0000 7f85f05fd700 1 -- 172.21.15.98:0/804256058 wait complete. 2024-04-25T17:46:43.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.967+0000 7f85f05fd700 1 Processor -- start 2024-04-25T17:46:43.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.967+0000 7f85f05fd700 1 -- start start 2024-04-25T17:46:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.968+0000 7f85f05fd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e8100ed0 0x7f85e8197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.968+0000 7f85f05fd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85e8101880 0x7f85e81982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.968+0000 7f85ee399700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e8100ed0 0x7f85e8197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.968+0000 7f85ee399700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e8100ed0 0x7f85e8197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53736/0 (socket says 172.21.15.98:53736) 2024-04-25T17:46:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.968+0000 7f85ee399700 1 -- 172.21.15.98:0/431972306 learned_addr learned my addr 172.21.15.98:0/431972306 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.968+0000 7f85f05fd700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85e8105fb0 0x7f85e819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.968+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85e8111b70 con 0x7f85e8100ed0 2024-04-25T17:46:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f85e81119f0 con 0x7f85e8101880 2024-04-25T17:46:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85edb98700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85e8101880 0x7f85e81982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85e8111cf0 con 0x7f85e8105fb0 2024-04-25T17:46:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85e8105fb0 0x7f85e819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85eeb9a700 1 -- 172.21.15.98:0/431972306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85e8101880 msgr2=0x7f85e81982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:43.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85e8101880 0x7f85e81982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85eeb9a700 1 -- 172.21.15.98:0/431972306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e8100ed0 msgr2=0x7f85e8197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:43.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e8100ed0 0x7f85e8197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85eeb9a700 1 -- 172.21.15.98:0/431972306 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e819cd70 con 0x7f85e8105fb0 2024-04-25T17:46:43.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85edb98700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85e8101880 0x7f85e81982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:43.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85ee399700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e8100ed0 0x7f85e8197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:43.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85e8105fb0 0x7f85e819c630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f85e0015000 tx=0x7f85e0005a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:43.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.969+0000 7f85df7fe700 1 -- 172.21.15.98:0/431972306 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e0003c00 con 0x7f85e8105fb0 2024-04-25T17:46:43.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.970+0000 7f85df7fe700 1 -- 172.21.15.98:0/431972306 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85e001f070 con 0x7f85e8105fb0 2024-04-25T17:46:43.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.970+0000 7f85df7fe700 1 -- 172.21.15.98:0/431972306 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e001b790 con 0x7f85e8105fb0 2024-04-25T17:46:43.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.970+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e819d000 con 0x7f85e8105fb0 2024-04-25T17:46:43.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.970+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e810eb20 con 0x7f85e8105fb0 2024-04-25T17:46:43.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.974+0000 7f85df7fe700 1 -- 172.21.15.98:0/431972306 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f85e00050d0 con 0x7f85e8105fb0 2024-04-25T17:46:43.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.976+0000 7f85df7fe700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85d40a3ed0 0x7f85d40a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.976+0000 7f85fae86b80 1 Processor -- start 2024-04-25T17:46:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.976+0000 7f85df7fe700 1 -- 172.21.15.98:0/431972306 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f85e00d2b20 con 0x7f85e8105fb0 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.976+0000 7f85fae86b80 1 -- start start 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af819b2b0 0x563af81996e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af8199c20 0x563af819a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af81ad5a0 0x563af81af990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85fae86b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563af80ece20 con 0x563af81ad5a0 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85fae86b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563af8093ac0 con 0x563af8199c20 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85fae86b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563af80e8de0 con 0x563af819b2b0 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85eeb9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af81ad5a0 0x563af81af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85ee399700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85d40a3ed0 0x7f85d40a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85eeb9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af81ad5a0 0x563af81af990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53742/0 (socket says 172.21.15.98:53742) 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85eeb9a700 1 -- 172.21.15.98:0/3103750428 learned_addr learned my addr 172.21.15.98:0/3103750428 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85edb98700 1 --2- 172.21.15.98:0/3103750428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af8199c20 0x563af819a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.977+0000 7f85ee399700 1 --2- 172.21.15.98:0/3103750428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af819b2b0 0x563af81996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85edb98700 1 -- 172.21.15.98:0/3103750428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af819b2b0 msgr2=0x563af81996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:43.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85edb98700 1 --2- 172.21.15.98:0/3103750428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af819b2b0 0x563af81996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85edb98700 1 -- 172.21.15.98:0/3103750428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af81ad5a0 msgr2=0x563af81af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:43.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85edb98700 1 --2- 172.21.15.98:0/3103750428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af81ad5a0 0x563af81af990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85edb98700 1 -- 172.21.15.98:0/3103750428 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563af80578d0 con 0x563af8199c20 2024-04-25T17:46:43.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/3103750428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af81ad5a0 0x563af81af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:46:43.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85edb98700 1 --2- 172.21.15.98:0/3103750428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af8199c20 0x563af819a0a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009fb0 tx=0x7f85d8011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b526dddfc0c889c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85ee399700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85d40a3ed0 0x7f85d40a6390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f85e4013e90 tx=0x7f85e4009cd0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85dcff9700 1 -- 172.21.15.98:0/3103750428 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8004230 con 0x563af8199c20 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85dcff9700 1 -- 172.21.15.98:0/3103750428 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85d8004ce0 con 0x563af8199c20 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.978+0000 7f85dcff9700 1 -- 172.21.15.98:0/3103750428 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8003680 con 0x563af8199c20 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.979+0000 7f85fae86b80 1 -- 172.21.15.98:0/3103750428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af8199c20 msgr2=0x563af819a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.979+0000 7f85fae86b80 1 --2- 172.21.15.98:0/3103750428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af8199c20 0x563af819a0a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009fb0 tx=0x7f85d8011720 comp rx=0 tx=0).stop 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.979+0000 7f85fae86b80 1 -- 172.21.15.98:0/3103750428 shutdown_connections 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.979+0000 7f85fae86b80 1 --2- 172.21.15.98:0/3103750428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af819b2b0 0x563af81996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.979+0000 7f85fae86b80 1 --2- 172.21.15.98:0/3103750428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af81ad5a0 0x563af81af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.979+0000 7f85fae86b80 1 --2- 172.21.15.98:0/3103750428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af8199c20 0x563af819a0a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.979+0000 7f85fae86b80 1 -- 172.21.15.98:0/3103750428 >> 172.21.15.98:0/3103750428 conn(0x563af80c6a20 msgr2=0x563af8195ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.979+0000 7f85fae86b80 1 -- 172.21.15.98:0/3103750428 shutdown_connections 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.979+0000 7f85fae86b80 1 -- 172.21.15.98:0/3103750428 wait complete. 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.980+0000 7f85fae86b80 1 -- start start 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.980+0000 7f85fae86b80 1 Processor -- start 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.980+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af8199c20 0x563af81a0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af819b2b0 0x563af819d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85ee399700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af8199c20 0x563af81a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85ee399700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af8199c20 0x563af81a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53744/0 (socket says 172.21.15.98:53744) 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85ee399700 1 -- 172.21.15.98:0/1349221770 learned_addr learned my addr 172.21.15.98:0/1349221770 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563af81b02d0 con 0x563af8199c20 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563af80a6740 con 0x563af819b2b0 2024-04-25T17:46:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563af81b0450 con 0x563af81ad5a0 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85edb98700 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af819b2b0 0x563af819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.981+0000 7f85fae86b80 10 client.0 _ll_register_callbacks cb 0x563af8085ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85eeb9a700 1 -- 172.21.15.98:0/1349221770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af819b2b0 msgr2=0x563af819d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af819b2b0 0x563af819d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85eeb9a700 1 -- 172.21.15.98:0/1349221770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af8199c20 msgr2=0x563af81a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af8199c20 0x563af81a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85eeb9a700 1 -- 172.21.15.98:0/1349221770 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563af803aa60 con 0x563af81ad5a0 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85ee399700 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af8199c20 0x563af81a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819d980 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f85e0009850 tx=0x7f85e009e670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c16e512ddd71168d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:43.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85d1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85d1ffb700 1 -- 172.21.15.98:0/1349221770 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e009ec90 con 0x563af81ad5a0 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85d8004ce0 con 0x563af81ad5a0 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85d1ffb700 1 -- 172.21.15.98:0/1349221770 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f85e00a5040 con 0x563af81ad5a0 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85d1ffb700 1 -- 172.21.15.98:0/1349221770 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f85e00a3040 con 0x563af81ad5a0 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85fae86b80 10 client.25966 fetch_fsmap learned FSMap version 44 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.982+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x563af81b0330 con 0x563af81ad5a0 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.983+0000 7f85d1ffb700 1 -- 172.21.15.98:0/1349221770 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85d8004ce0 con 0x563af81ad5a0 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.983+0000 7f85d1ffb700 1 -- 172.21.15.98:0/1349221770 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f85e0011070 con 0x563af81ad5a0 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.983+0000 7f85fae86b80 10 client.25966 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.983+0000 7f85fae86b80 10 client.25966 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.983+0000 7f85fae86b80 20 client.25966 populate_metadata read hostname 'smithi098' 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.983+0000 7f85fae86b80 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563af81bd0a0 0x563af81bf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.983+0000 7f85fae86b80 4 client.25966 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:46:43.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.983+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563af80fa610 con 0x563af81bd0a0 2024-04-25T17:46:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.983+0000 7f85ee399700 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563af81bd0a0 0x563af81bf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.984+0000 7f85d1ffb700 10 client.25966 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.984+0000 7f85ee399700 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563af81bd0a0 0x563af81bf490 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc23238a7d5b876b server_cookie=74d7f06ec6760d2f in_seq=0 out_seq=0 2024-04-25T17:46:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.984+0000 7f85d1ffb700 1 -- 172.21.15.98:0/1349221770 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563af80fa610 con 0x563af81bd0a0 2024-04-25T17:46:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.984+0000 7f85d1ffb700 10 client.25966 handle_command_reply: tid=0 2024-04-25T17:46:43.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.984+0000 7f85fae86b80 1 client.25966 shutdown 2024-04-25T17:46:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.985+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 msgr2=0x563af819d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.985+0000 7f85fae86b80 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819d980 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f85e0009850 tx=0x7f85e009e670 comp rx=0 tx=0).stop 2024-04-25T17:46:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.985+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 shutdown_connections 2024-04-25T17:46:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.985+0000 7f85fae86b80 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563af81bd0a0 0x563af81bf490 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.985+0000 7f85fae86b80 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819d980 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.985+0000 7f85fae86b80 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af8199c20 0x563af81a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.985+0000 7f85fae86b80 1 --2- 172.21.15.98:0/1349221770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af819b2b0 0x563af819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:43.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.985+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 >> 172.21.15.98:0/1349221770 conn(0x563af80c6a20 msgr2=0x563af819bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.985+0000 7f85d1ffb700 0 client.25966 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.986+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 shutdown_connections 2024-04-25T17:46:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.986+0000 7f85fae86b80 1 -- 172.21.15.98:0/1349221770 wait complete. 2024-04-25T17:46:43.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:43.986+0000 7f85fae86b80 20 client.25966 trim_cache size 0 max 16384 2024-04-25T17:46:44.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.018+0000 7f85fae86b80 1 Processor -- start 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.018+0000 7f85fae86b80 1 -- start start 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.018+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af8199c20 0x563af8195d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.018+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af819b2b0 0x563af81b91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.018+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.018+0000 7f85fae86b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85e81ad5d0 con 0x563af819b2b0 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.019+0000 7f85fae86b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f85e81aae00 con 0x563af8199c20 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.019+0000 7f85eeb9a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.019+0000 7f85eeb9a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53956/0 (socket says 172.21.15.98:53956) 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.019+0000 7f85eeb9a700 1 -- 172.21.15.98:0/4223213540 learned_addr learned my addr 172.21.15.98:0/4223213540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.019+0000 7f85fae86b80 1 -- 172.21.15.98:0/4223213540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85e81a8c70 con 0x563af81ad5a0 2024-04-25T17:46:44.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.019+0000 7f85edb98700 1 --2- 172.21.15.98:0/4223213540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af819b2b0 0x563af81b91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:44.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.020+0000 7f85eeb9a700 1 -- 172.21.15.98:0/4223213540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af8199c20 msgr2=0x563af8195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:44.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.020+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/4223213540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af8199c20 0x563af8195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.020+0000 7f85eeb9a700 1 -- 172.21.15.98:0/4223213540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af819b2b0 msgr2=0x563af81b91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:44.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.020+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/4223213540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af819b2b0 0x563af81b91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.020+0000 7f85eeb9a700 1 -- 172.21.15.98:0/4223213540 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563af8196770 con 0x563af81ad5a0 2024-04-25T17:46:44.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.020+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/4223213540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819fc80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f85e009b660 tx=0x7f85e0009d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da19a17fa9109f35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.025+0000 7f85d3fff700 1 -- 172.21.15.98:0/4223213540 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e00a7870 con 0x563af81ad5a0 2024-04-25T17:46:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.025+0000 7f85d3fff700 1 -- 172.21.15.98:0/4223213540 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85e00b8120 con 0x563af81ad5a0 2024-04-25T17:46:44.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.025+0000 7f85d3fff700 1 -- 172.21.15.98:0/4223213540 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e009b480 con 0x563af81ad5a0 2024-04-25T17:46:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.026+0000 7f85fae86b80 1 -- 172.21.15.98:0/4223213540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 msgr2=0x563af819fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:44.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.026+0000 7f85fae86b80 1 --2- 172.21.15.98:0/4223213540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819fc80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f85e009b660 tx=0x7f85e0009d00 comp rx=0 tx=0).stop 2024-04-25T17:46:44.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.030+0000 7f85fae86b80 1 -- 172.21.15.98:0/4223213540 shutdown_connections 2024-04-25T17:46:44.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.030+0000 7f85fae86b80 1 --2- 172.21.15.98:0/4223213540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af81ad5a0 0x563af819fc80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.030+0000 7f85fae86b80 1 --2- 172.21.15.98:0/4223213540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af8199c20 0x563af8195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.030+0000 7f85fae86b80 1 --2- 172.21.15.98:0/4223213540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af819b2b0 0x563af81b91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.030+0000 7f85fae86b80 1 -- 172.21.15.98:0/4223213540 >> 172.21.15.98:0/4223213540 conn(0x563af80c6a20 msgr2=0x563af8195640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.036+0000 7f85fae86b80 1 -- 172.21.15.98:0/4223213540 shutdown_connections 2024-04-25T17:46:44.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.036+0000 7f85fae86b80 1 -- 172.21.15.98:0/4223213540 wait complete. 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.036+0000 7f85fae86b80 1 -- start start 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85fae86b80 1 Processor -- start 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af8199c20 0x563af81b58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af819b2b0 0x563af81b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85fae86b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af81ad5a0 0x563af81b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85ee399700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af8199c20 0x563af81b58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85ee399700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af8199c20 0x563af81b58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53972/0 (socket says 172.21.15.98:53972) 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85ee399700 1 -- 172.21.15.98:0/598006134 learned_addr learned my addr 172.21.15.98:0/598006134 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85e81a92c0 con 0x563af819b2b0 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f85e81aa730 con 0x563af81ad5a0 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85e81a9990 con 0x563af8199c20 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.037+0000 7f85eeb9a700 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af81ad5a0 0x563af81b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:44.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.038+0000 7f85fae86b80 10 client.0 _ll_register_callbacks cb 0x563af8085ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:46:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.040+0000 7f85ee399700 1 -- 172.21.15.98:0/598006134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af81ad5a0 msgr2=0x563af81b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.040+0000 7f85ee399700 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af81ad5a0 0x563af81b0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.040+0000 7f85ee399700 1 -- 172.21.15.98:0/598006134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af819b2b0 msgr2=0x563af81b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.040+0000 7f85ee399700 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af819b2b0 0x563af81b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.040+0000 7f85ee399700 1 -- 172.21.15.98:0/598006134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563af81b1120 con 0x563af8199c20 2024-04-25T17:46:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.041+0000 7f85ee399700 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af8199c20 0x563af81b58d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f85e4013a90 tx=0x7f85e4013480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bcb996f3860d415 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.041+0000 7f85d1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:46:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.041+0000 7f85d1ffb700 1 -- 172.21.15.98:0/598006134 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4013c30 con 0x563af8199c20 2024-04-25T17:46:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.041+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85e00b8120 con 0x563af8199c20 2024-04-25T17:46:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.041+0000 7f85d1ffb700 1 -- 172.21.15.98:0/598006134 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f85e4004830 con 0x563af8199c20 2024-04-25T17:46:44.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.041+0000 7f85d1ffb700 1 -- 172.21.15.98:0/598006134 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f85e4004a30 con 0x563af8199c20 2024-04-25T17:46:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.044+0000 7f85fae86b80 10 client.25972 fetch_fsmap learned FSMap version 44 2024-04-25T17:46:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.044+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x563af821e2f0 con 0x563af8199c20 2024-04-25T17:46:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.045+0000 7f85d1ffb700 1 -- 172.21.15.98:0/598006134 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85e40054f0 con 0x563af8199c20 2024-04-25T17:46:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.045+0000 7f85d1ffb700 1 -- 172.21.15.98:0/598006134 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f85e400f040 con 0x563af8199c20 2024-04-25T17:46:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.047+0000 7f85fae86b80 10 client.25972 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:46:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.047+0000 7f85fae86b80 10 client.25972 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:46:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.047+0000 7f85fae86b80 20 client.25972 populate_metadata read hostname 'smithi098' 2024-04-25T17:46:44.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.047+0000 7f85fae86b80 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563af821ea20 0x563af8220e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.047+0000 7f85fae86b80 4 client.25972 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.3, "await_for": 42.0}] 2024-04-25T17:46:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.047+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 78.3, "await_for": 42.0}) -- 0x7f85e00b8120 con 0x563af821ea20 2024-04-25T17:46:44.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.048+0000 7f85edb98700 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563af821ea20 0x563af8220e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.057+0000 7f85edb98700 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563af821ea20 0x563af8220e40 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe633666cd8845d9 server_cookie=6a03310a4705ffe0 in_seq=0 out_seq=0 2024-04-25T17:46:44.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.057+0000 7f85d1ffb700 10 client.25972 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:44.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85d1ffb700 1 -- 172.21.15.98:0/598006134 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x7f85e00b8120 con 0x563af821ea20 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85d1ffb700 10 client.25972 handle_command_reply: tid=0 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 client.25972 shutdown 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af8199c20 msgr2=0x563af81b58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af8199c20 0x563af81b58d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f85e4013a90 tx=0x7f85e4013480 comp rx=0 tx=0).stop 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 shutdown_connections 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x563af821ea20 0x563af8220e40 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563af8199c20 0x563af81b58d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563af81ad5a0 0x563af81b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85d1ffb700 0 client.25972 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 --2- 172.21.15.98:0/598006134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563af819b2b0 0x563af81b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 >> 172.21.15.98:0/598006134 conn(0x563af80c6a20 msgr2=0x563af8194440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:44.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 shutdown_connections 2024-04-25T17:46:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.279+0000 7f85fae86b80 1 -- 172.21.15.98:0/598006134 wait complete. 2024-04-25T17:46:44.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85fae86b80 20 client.25972 trim_cache size 0 max 16384 2024-04-25T17:46:44.281 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:46:44.281 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 44, 2024-04-25T17:46:44.281 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:46:44.281 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 30, 2024-04-25T17:46:44.281 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: "3b78fc9f": { 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 30, 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 78.3, 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:46:44.282 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:46:44.283 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:46:44.283 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:46:44.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:46:44.283 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:46:44.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:46:44.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:46:44.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:46:44.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:46:44.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:46:44.283 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:46:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85d40a3ed0 msgr2=0x7f85d40a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85d40a3ed0 0x7f85d40a6390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f85e4013e90 tx=0x7f85e4009cd0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85e8105fb0 msgr2=0x7f85e819c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85e8105fb0 0x7f85e819c630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f85e0015000 tx=0x7f85e0005a80 comp rx=0 tx=0).stop 2024-04-25T17:46:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 shutdown_connections 2024-04-25T17:46:44.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f85d40a3ed0 0x7f85d40a6390 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85e8105fb0 0x7f85e819c630 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85e8100ed0 0x7f85e8197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 --2- 172.21.15.98:0/431972306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85e8101880 0x7f85e81982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 >> 172.21.15.98:0/431972306 conn(0x7f85e8077b70 msgr2=0x7f85e8103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 shutdown_connections 2024-04-25T17:46:44.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:46:44.280+0000 7f85f05fd700 1 -- 172.21.15.98:0/431972306 wait complete. 2024-04-25T17:46:44.637 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:44 smithi098 ceph-mon[29192]: pgmap v338: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 13 MiB/s wr, 953 op/s 2024-04-25T17:46:44.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:44 smithi203 ceph-mon[32834]: pgmap v338: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 13 MiB/s wr, 953 op/s 2024-04-25T17:46:44.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:44 smithi173 ceph-mon[32856]: pgmap v338: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 13 MiB/s wr, 953 op/s 2024-04-25T17:46:45.002 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3b78fc9f', quiesce took 7.5 seconds. Will release after: 18.3 2024-04-25T17:46:46.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 615.0 s, 1605.0 tps, lat 31.728 ms stddev 37.052 2024-04-25T17:46:46.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:46 smithi098 ceph-mon[29192]: pgmap v339: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 9.0 MiB/s wr, 667 op/s 2024-04-25T17:46:46.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:46 smithi203 ceph-mon[32834]: pgmap v339: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 9.0 MiB/s wr, 667 op/s 2024-04-25T17:46:46.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:46 smithi173 ceph-mon[32856]: pgmap v339: 97 pgs: 97 active+clean; 4.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 9.0 MiB/s wr, 667 op/s 2024-04-25T17:46:47.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:47 smithi098 ceph-mon[29192]: pgmap v340: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-25T17:46:47.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:47 smithi203 ceph-mon[32834]: pgmap v340: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-25T17:46:47.773 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:47 smithi173 ceph-mon[32856]: pgmap v340: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-04-25T17:46:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.829+0000 7fb3deea7640 1 -- 172.21.15.98:0/198812363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d8101cc0 msgr2=0x7fb3d8105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.829+0000 7fb3deea7640 1 --2- 172.21.15.98:0/198812363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d8101cc0 0x7fb3d8105ab0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009900 tx=0x7fb3c402a410 comp rx=0 tx=0).stop 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.829+0000 7fb3deea7640 1 -- 172.21.15.98:0/198812363 shutdown_connections 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.829+0000 7fb3deea7640 1 --2- 172.21.15.98:0/198812363 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d8106190 0x7fb3d810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.829+0000 7fb3deea7640 1 --2- 172.21.15.98:0/198812363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d8101cc0 0x7fb3d8105ab0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.829+0000 7fb3deea7640 1 --2- 172.21.15.98:0/198812363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d8101310 0x7fb3d81016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.829+0000 7fb3deea7640 1 -- 172.21.15.98:0/198812363 >> 172.21.15.98:0/198812363 conn(0x7fb3d8077cc0 msgr2=0x7fb3d80780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.829+0000 7fb3deea7640 1 -- 172.21.15.98:0/198812363 shutdown_connections 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.829+0000 7fb3deea7640 1 -- 172.21.15.98:0/198812363 wait complete. 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3deea7640 1 Processor -- start 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3deea7640 1 -- start start 2024-04-25T17:46:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3deea7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d8101310 0x7fb3d81a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3deea7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d8106190 0x7fb3d81a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3deea7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d819a7f0 0x7fb3d819aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3deea7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d8111170 con 0x7fb3d8101310 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3deea7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d8110ff0 con 0x7fb3d8106190 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3deea7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81112f0 con 0x7fb3d819a7f0 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3dcc43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d8101310 0x7fb3d81a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3dcc43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d8101310 0x7fb3d81a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53770/0 (socket says 172.21.15.98:53770) 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3dcc43640 1 -- 172.21.15.98:0/1719110072 learned_addr learned my addr 172.21.15.98:0/1719110072 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3dd444640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d819a7f0 0x7fb3d819aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.830+0000 7fb3d7fff640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d8106190 0x7fb3d81a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3dcc43640 1 -- 172.21.15.98:0/1719110072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d819a7f0 msgr2=0x7fb3d819aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d819a7f0 0x7fb3d819aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3dcc43640 1 -- 172.21.15.98:0/1719110072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d8106190 msgr2=0x7fb3d81a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d8106190 0x7fb3d81a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3dcc43640 1 -- 172.21.15.98:0/1719110072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d819b380 con 0x7fb3d8101310 2024-04-25T17:46:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3dd444640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d819a7f0 0x7fb3d819aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:46:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d8101310 0x7fb3d81a0680 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00d770 tx=0x7fb3cc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3d5ffb640 1 -- 172.21.15.98:0/1719110072 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc017070 con 0x7fb3d8101310 2024-04-25T17:46:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3deea7640 1 -- 172.21.15.98:0/1719110072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d819b670 con 0x7fb3d8101310 2024-04-25T17:46:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3d5ffb640 1 -- 172.21.15.98:0/1719110072 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3cc0054d0 con 0x7fb3d8101310 2024-04-25T17:46:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3d5ffb640 1 -- 172.21.15.98:0/1719110072 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc004c30 con 0x7fb3d8101310 2024-04-25T17:46:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.831+0000 7fb3deea7640 1 -- 172.21.15.98:0/1719110072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d81a77d0 con 0x7fb3d8101310 2024-04-25T17:46:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.833+0000 7fb3d5ffb640 1 -- 172.21.15.98:0/1719110072 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb3cc004e10 con 0x7fb3d8101310 2024-04-25T17:46:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3e1bbe740 1 Processor -- start 2024-04-25T17:46:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3e1bbe740 1 -- start start 2024-04-25T17:46:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3d5ffb640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb3c80a3dc0 0x7fb3c80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a7ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7cf40 0x555759a872c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a87800 0x555759a89bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5557599e1970 con 0x555759a7cf40 2024-04-25T17:46:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55575997e4a0 con 0x555759a87800 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5557599dd930 con 0x555759a7c620 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3dd444640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb3c80a3dc0 0x7fb3c80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3d5ffb640 1 -- 172.21.15.98:0/1719110072 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fb3cc0d03e0 con 0x7fb3d8101310 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7cf40 0x555759a872c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3d7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7cf40 0x555759a872c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53786/0 (socket says 172.21.15.98:53786) 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.834+0000 7fb3d7fff640 1 -- 172.21.15.98:0/3441496287 learned_addr learned my addr 172.21.15.98:0/3441496287 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/3441496287 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a7ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3dd444640 1 --2- 172.21.15.98:0/3441496287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a87800 0x555759a89bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3d7fff640 1 -- 172.21.15.98:0/3441496287 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 msgr2=0x555759a7ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3d7fff640 1 --2- 172.21.15.98:0/3441496287 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a7ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3d7fff640 1 -- 172.21.15.98:0/3441496287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a87800 msgr2=0x555759a89bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3d7fff640 1 --2- 172.21.15.98:0/3441496287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a87800 0x555759a89bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3d7fff640 1 -- 172.21.15.98:0/3441496287 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557596f4210 con 0x555759a7cf40 2024-04-25T17:46:49.843 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3dd444640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb3c80a3dc0 0x7fb3c80a6280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000c8a0 tx=0x7fb3d000cca0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3d7fff640 1 --2- 172.21.15.98:0/3441496287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7cf40 0x555759a872c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009900 tx=0x7fb3c4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ccf157babdbbd95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3c2ffd640 1 -- 172.21.15.98:0/3441496287 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c403f070 con 0x555759a7cf40 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3c2ffd640 1 -- 172.21.15.98:0/3441496287 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c40316a0 con 0x555759a7cf40 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.835+0000 7fb3c2ffd640 1 -- 172.21.15.98:0/3441496287 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4035070 con 0x555759a7cf40 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3441496287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7cf40 msgr2=0x555759a872c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3441496287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7cf40 0x555759a872c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009900 tx=0x7fb3c4032ce0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3441496287 shutdown_connections 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3441496287 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a87800 0x555759a89bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3441496287 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7cf40 0x555759a872c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3441496287 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a7ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3441496287 >> 172.21.15.98:0/3441496287 conn(0x555759a78b30 msgr2=0x555759a78f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3441496287 shutdown_connections 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3441496287 wait complete. 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 -- start start 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.836+0000 7fb3e1bbe740 1 Processor -- start 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a87190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a7cf40 0x555759a84480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a84a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x555759a8a3d0 con 0x555759a87800 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55575997c0b0 con 0x555759a7cf40 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555759a8a550 con 0x555759a7c620 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3e1bbe740 10 client.0 _ll_register_callbacks cb 0x555759969d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dd444640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a84a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dd444640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a84a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53802/0 (socket says 172.21.15.98:53802) 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dd444640 1 -- 172.21.15.98:0/3231596906 learned_addr learned my addr 172.21.15.98:0/3231596906 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a87190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3d7fff640 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a7cf40 0x555759a84480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dd444640 1 -- 172.21.15.98:0/3231596906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 msgr2=0x555759a87190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dd444640 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a87190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dd444640 1 -- 172.21.15.98:0/3231596906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a7cf40 msgr2=0x555759a84480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dd444640 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a7cf40 0x555759a84480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dd444640 1 -- 172.21.15.98:0/3231596906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557597c0620 con 0x555759a87800 2024-04-25T17:46:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a87190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:46:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3dd444640 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a84a00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb3d001c6b0 tx=0x7fb3d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908ff24ff29d5af4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:46:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3231596906 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0009d00 con 0x555759a87800 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3231596906 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fb3d001a410 con 0x555759a87800 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.837+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3231596906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3c40316a0 con 0x555759a87800 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3231596906 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb3d001a610 con 0x555759a87800 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3e1bbe740 10 client.16476 fetch_fsmap learned FSMap version 44 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3231596906 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x555759a96370 con 0x555759a87800 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3231596906 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3c40316a0 con 0x555759a87800 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3231596906 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fb3d002e7d0 con 0x555759a87800 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3e1bbe740 10 client.16476 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3e1bbe740 10 client.16476 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3e1bbe740 20 client.16476 populate_metadata read hostname 'smithi098' 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555759a97830 0x555759a99c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3e1bbe740 4 client.16476 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3231596906 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557599ef390 con 0x555759a97830 2024-04-25T17:46:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.838+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555759a97830 0x555759a99c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.839+0000 7fb3b7fff640 10 client.16476 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.839+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555759a97830 0x555759a99c60 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a82a157da52c2876 server_cookie=91b2fe1635c5155a in_seq=0 out_seq=0 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.839+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3231596906 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557599ef390 con 0x555759a97830 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.839+0000 7fb3b7fff640 10 client.16476 handle_command_reply: tid=0 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.839+0000 7fb3e1bbe740 1 client.16476 shutdown 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3231596906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 msgr2=0x555759a84a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a84a00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb3d001c6b0 tx=0x7fb3d0004a40 comp rx=0 tx=0).stop 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3231596906 shutdown_connections 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555759a97830 0x555759a99c60 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a84a00 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3b7fff640 0 client.16476 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a7cf40 0x555759a84480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3231596906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a87190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3231596906 >> 172.21.15.98:0/3231596906 conn(0x555759a78b30 msgr2=0x555759a7e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3231596906 shutdown_connections 2024-04-25T17:46:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3231596906 wait complete. 2024-04-25T17:46:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.840+0000 7fb3e1bbe740 20 client.16476 trim_cache size 0 max 16384 2024-04-25T17:46:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3e1bbe740 1 Processor -- start 2024-04-25T17:46:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3e1bbe740 1 -- start start 2024-04-25T17:46:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7c620 0x555759a93290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7cf40 0x555759a783c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a87800 0x555759a833e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81b25f0 con 0x555759a7c620 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81afe60 con 0x555759a87800 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81adcd0 con 0x555759a7cf40 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3dcc43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7c620 0x555759a93290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3dcc43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7c620 0x555759a93290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53808/0 (socket says 172.21.15.98:53808) 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3dcc43640 1 -- 172.21.15.98:0/205700039 learned_addr learned my addr 172.21.15.98:0/205700039 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3dd444640 1 --2- 172.21.15.98:0/205700039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a87800 0x555759a833e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3dcc43640 1 -- 172.21.15.98:0/205700039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7cf40 msgr2=0x555759a783c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/205700039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7cf40 0x555759a783c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3dcc43640 1 -- 172.21.15.98:0/205700039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a87800 msgr2=0x555759a833e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/205700039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a87800 0x555759a833e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3dcc43640 1 -- 172.21.15.98:0/205700039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555759a829f0 con 0x555759a7c620 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/205700039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7c620 0x555759a93290 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc005c20 tx=0x7fb3cc0986f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470c4a45d3905810 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3c27fc640 1 -- 172.21.15.98:0/205700039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc020040 con 0x555759a7c620 2024-04-25T17:46:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3c27fc640 1 -- 172.21.15.98:0/205700039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3cc0a4f90 con 0x555759a7c620 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.855+0000 7fb3c27fc640 1 -- 172.21.15.98:0/205700039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc098b20 con 0x555759a7c620 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/205700039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7c620 msgr2=0x555759a93290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/205700039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7c620 0x555759a93290 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc005c20 tx=0x7fb3cc0986f0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/205700039 shutdown_connections 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/205700039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a87800 0x555759a833e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/205700039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7cf40 0x555759a783c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/205700039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a7c620 0x555759a93290 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/205700039 >> 172.21.15.98:0/205700039 conn(0x555759a78b30 msgr2=0x555759a77cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/205700039 shutdown_connections 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/205700039 wait complete. 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 -- start start 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.856+0000 7fb3e1bbe740 1 Processor -- start 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a8d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a7cf40 0x555759a8d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3e1bbe740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a8dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81ae320 con 0x555759a87800 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81af790 con 0x555759a7cf40 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3e1bbe740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81ae9f0 con 0x555759a7c620 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3e1bbe740 10 client.0 _ll_register_callbacks cb 0x7fb3d0007f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dd444640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a8dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dd444640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a8dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53812/0 (socket says 172.21.15.98:53812) 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dd444640 1 -- 172.21.15.98:0/3052125562 learned_addr learned my addr 172.21.15.98:0/3052125562 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3d7fff640 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a7cf40 0x555759a8d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a8d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dd444640 1 -- 172.21.15.98:0/3052125562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 msgr2=0x555759a8d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dd444640 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a8d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dd444640 1 -- 172.21.15.98:0/3052125562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a7cf40 msgr2=0x555759a8d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dd444640 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a7cf40 0x555759a8d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dd444640 1 -- 172.21.15.98:0/3052125562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555759afcea0 con 0x555759a87800 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a8d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3dd444640 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a8dda0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb3d002ebc0 tx=0x7fb3d001aa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46ca70d40e7ff42 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.857+0000 7fb3b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3052125562 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d003b060 con 0x555759a87800 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3052125562 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7fb3d001a410 con 0x555759a87800 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3052125562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3cc0a4f90 con 0x555759a87800 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3052125562 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb3d002e990 con 0x555759a87800 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3e1bbe740 10 client.16482 fetch_fsmap learned FSMap version 44 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3052125562 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x555759afe190 con 0x555759a87800 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3052125562 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3cc0a4f90 con 0x555759a87800 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3052125562 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fb3d0036430 con 0x555759a87800 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3e1bbe740 10 client.16482 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:46:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3e1bbe740 10 client.16482 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:46:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3e1bbe740 20 client.16482 populate_metadata read hostname 'smithi098' 2024-04-25T17:46:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555759a97830 0x555759b00870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:46:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3e1bbe740 4 client.16482 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:46:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3052125562 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5557599e9200 con 0x555759a97830 2024-04-25T17:46:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.858+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555759a97830 0x555759b00870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:46:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.859+0000 7fb3b7fff640 10 client.16482 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.859+0000 7fb3dcc43640 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555759a97830 0x555759b00870 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a1eabd3becb83ba server_cookie=25c456d76331c92f in_seq=0 out_seq=0 2024-04-25T17:46:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3b7fff640 1 -- 172.21.15.98:0/3052125562 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5557599e9200 con 0x555759a97830 2024-04-25T17:46:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3b7fff640 10 client.16482 handle_command_reply: tid=0 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 client.16482 shutdown 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3052125562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 msgr2=0x555759a8dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a8dda0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb3d002ebc0 tx=0x7fb3d001aa60 comp rx=0 tx=0).stop 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3052125562 shutdown_connections 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x555759a97830 0x555759b00870 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x555759a87800 0x555759a8dda0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3b7fff640 0 client.16482 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x555759a7cf40 0x555759a8d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 --2- 172.21.15.98:0/3052125562 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555759a7c620 0x555759a8d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3052125562 >> 172.21.15.98:0/3052125562 conn(0x555759a78b30 msgr2=0x555759a92280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3052125562 shutdown_connections 2024-04-25T17:46:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.872+0000 7fb3e1bbe740 1 -- 172.21.15.98:0/3052125562 wait complete. 2024-04-25T17:46:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3e1bbe740 20 client.16482 trim_cache size 0 max 16384 2024-04-25T17:46:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:46:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:46:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:46:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:46:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 -- 172.21.15.98:0/1719110072 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb3c80a3dc0 msgr2=0x7fb3c80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb3c80a3dc0 0x7fb3c80a6280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000c8a0 tx=0x7fb3d000cca0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 -- 172.21.15.98:0/1719110072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d8101310 msgr2=0x7fb3d81a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:46:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d8101310 0x7fb3d81a0680 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00d770 tx=0x7fb3cc00dc30 comp rx=0 tx=0).stop 2024-04-25T17:46:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 -- 172.21.15.98:0/1719110072 shutdown_connections 2024-04-25T17:46:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb3c80a3dc0 0x7fb3c80a6280 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3d819a7f0 0x7fb3d819aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3d8106190 0x7fb3d81a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 --2- 172.21.15.98:0/1719110072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb3d8101310 0x7fb3d81a0680 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:46:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 -- 172.21.15.98:0/1719110072 >> 172.21.15.98:0/1719110072 conn(0x7fb3d8077cc0 msgr2=0x7fb3d8107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:46:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 -- 172.21.15.98:0/1719110072 shutdown_connections 2024-04-25T17:46:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:46:49.873+0000 7fb3deea7640 1 -- 172.21.15.98:0/1719110072 wait complete. 2024-04-25T17:46:49.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:49 smithi098 ceph-mon[29192]: pgmap v341: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 7.1 MiB/s wr, 507 op/s 2024-04-25T17:46:49.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:49 smithi203 ceph-mon[32834]: pgmap v341: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 7.1 MiB/s wr, 507 op/s 2024-04-25T17:46:50.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:49 smithi173 ceph-mon[32856]: pgmap v341: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 7.1 MiB/s wr, 507 op/s 2024-04-25T17:46:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 620.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:46:51.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:51 smithi098 ceph-mon[29192]: pgmap v342: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 7.1 MiB/s wr, 507 op/s 2024-04-25T17:46:51.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:51 smithi203 ceph-mon[32834]: pgmap v342: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 7.1 MiB/s wr, 507 op/s 2024-04-25T17:46:52.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:51 smithi173 ceph-mon[32856]: pgmap v342: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 7.1 MiB/s wr, 507 op/s 2024-04-25T17:46:53.824 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:53 smithi173 ceph-mon[32856]: pgmap v343: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 7.1 MiB/s wr, 514 op/s 2024-04-25T17:46:53.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:53 smithi098 ceph-mon[29192]: pgmap v343: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 7.1 MiB/s wr, 514 op/s 2024-04-25T17:46:53.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:53 smithi203 ceph-mon[32834]: pgmap v343: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 7.1 MiB/s wr, 514 op/s 2024-04-25T17:46:55.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:55 smithi098 ceph-mon[29192]: pgmap v344: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.0 MiB/s wr, 209 op/s 2024-04-25T17:46:55.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:55 smithi203 ceph-mon[32834]: pgmap v344: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.0 MiB/s wr, 209 op/s 2024-04-25T17:46:56.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:55 smithi173 ceph-mon[32856]: pgmap v344: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 3.0 MiB/s wr, 209 op/s 2024-04-25T17:46:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 625.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:46:57.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:57 smithi098 ceph-mon[29192]: pgmap v345: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 3.0 MiB/s wr, 212 op/s 2024-04-25T17:46:57.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:57 smithi203 ceph-mon[32834]: pgmap v345: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 3.0 MiB/s wr, 212 op/s 2024-04-25T17:46:58.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:57 smithi173 ceph-mon[32856]: pgmap v345: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 3.0 MiB/s wr, 212 op/s 2024-04-25T17:46:59.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:46:59 smithi098 ceph-mon[29192]: pgmap v346: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T17:46:59.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:46:59 smithi203 ceph-mon[32834]: pgmap v346: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T17:47:00.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:46:59 smithi173 ceph-mon[32856]: pgmap v346: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T17:47:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 630.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:47:01.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:01 smithi098 ceph-mon[29192]: pgmap v347: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T17:47:01.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:01 smithi203 ceph-mon[32834]: pgmap v347: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T17:47:02.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:01 smithi173 ceph-mon[32856]: pgmap v347: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T17:47:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.969+0000 7f6b40f36640 1 -- 172.21.15.98:0/3709395779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c06b350 msgr2=0x7f6b3c106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.969+0000 7f6b40f36640 1 --2- 172.21.15.98:0/3709395779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c06b350 0x7f6b3c106ba0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009900 tx=0x7f6b2402a410 comp rx=0 tx=0).stop 2024-04-25T17:47:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.969+0000 7f6b40f36640 1 -- 172.21.15.98:0/3709395779 shutdown_connections 2024-04-25T17:47:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.969+0000 7f6b40f36640 1 --2- 172.21.15.98:0/3709395779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c107280 0x7f6b3c10ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.969+0000 7f6b40f36640 1 --2- 172.21.15.98:0/3709395779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c06b350 0x7f6b3c106ba0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.969+0000 7f6b40f36640 1 --2- 172.21.15.98:0/3709395779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b3c06a9a0 0x7f6b3c06ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.969+0000 7f6b40f36640 1 -- 172.21.15.98:0/3709395779 >> 172.21.15.98:0/3709395779 conn(0x7f6b3c077c30 msgr2=0x7f6b3c078040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.970+0000 7f6b40f36640 1 -- 172.21.15.98:0/3709395779 shutdown_connections 2024-04-25T17:47:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.970+0000 7f6b40f36640 1 -- 172.21.15.98:0/3709395779 wait complete. 2024-04-25T17:47:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.970+0000 7f6b40f36640 1 Processor -- start 2024-04-25T17:47:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.970+0000 7f6b40f36640 1 -- start start 2024-04-25T17:47:02.972 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b40f36640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c06a9a0 0x7f6b3c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b40f36640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c107280 0x7f6b3c198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b40f36640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b3c19d010 0x7f6b3c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b40f36640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c112240 con 0x7f6b3c107280 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b40f36640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1120c0 con 0x7f6b3c19d010 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b40f36640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1123c0 con 0x7f6b3c06a9a0 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c107280 0x7f6b3c198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c107280 0x7f6b3c198ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42476/0 (socket says 172.21.15.98:42476) 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b33fff640 1 -- 172.21.15.98:0/3115900021 learned_addr learned my addr 172.21.15.98:0/3115900021 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b3a59c640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c06a9a0 0x7f6b3c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b3ad9d640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b3c19d010 0x7f6b3c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b33fff640 1 -- 172.21.15.98:0/3115900021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c06a9a0 msgr2=0x7f6b3c1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b33fff640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c06a9a0 0x7f6b3c1987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.973 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b33fff640 1 -- 172.21.15.98:0/3115900021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b3c19d010 msgr2=0x7f6b3c19d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b33fff640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b3c19d010 0x7f6b3c19d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b33fff640 1 -- 172.21.15.98:0/3115900021 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3c19db00 con 0x7f6b3c107280 2024-04-25T17:47:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b33fff640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c107280 0x7f6b3c198ce0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6b24032fd0 tx=0x7f6b24004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b337fe640 1 -- 172.21.15.98:0/3115900021 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2403f030 con 0x7f6b3c107280 2024-04-25T17:47:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b40f36640 1 -- 172.21.15.98:0/3115900021 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b3c19dd90 con 0x7f6b3c107280 2024-04-25T17:47:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b337fe640 1 -- 172.21.15.98:0/3115900021 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b24004490 con 0x7f6b3c107280 2024-04-25T17:47:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b337fe640 1 -- 172.21.15.98:0/3115900021 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b24035070 con 0x7f6b3c107280 2024-04-25T17:47:02.974 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.971+0000 7f6b40f36640 1 -- 172.21.15.98:0/3115900021 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b3c1a5550 con 0x7f6b3c107280 2024-04-25T17:47:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.974+0000 7f6b337fe640 1 -- 172.21.15.98:0/3115900021 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6b24042030 con 0x7f6b3c107280 2024-04-25T17:47:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.974+0000 7f6b43c54740 1 Processor -- start 2024-04-25T17:47:02.980 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.974+0000 7f6b43c54740 1 -- start start 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.974+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e4118a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.974+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4118f40 0x55a3e41232c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.974+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4123800 0x55a3e4125bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.974+0000 7f6b43c54740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a3e407d970 con 0x55a3e4118620 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.974+0000 7f6b43c54740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a3e401a4a0 con 0x55a3e4118f40 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.974+0000 7f6b43c54740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a3e4079930 con 0x55a3e4123800 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.974+0000 7f6b337fe640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6b280a3dc0 0x7f6b280a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e4118a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e4118a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42492/0 (socket says 172.21.15.98:42492) 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 -- 172.21.15.98:0/1229152105 learned_addr learned my addr 172.21.15.98:0/1229152105 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6b280a3dc0 0x7f6b280a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3a59c640 1 --2- 172.21.15.98:0/1229152105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4118f40 0x55a3e41232c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b33fff640 1 --2- 172.21.15.98:0/1229152105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4123800 0x55a3e4125bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b337fe640 1 -- 172.21.15.98:0/3115900021 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f6b240f0cb0 con 0x7f6b3c107280 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 -- 172.21.15.98:0/1229152105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4123800 msgr2=0x55a3e4125bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 --2- 172.21.15.98:0/1229152105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4123800 0x55a3e4125bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 -- 172.21.15.98:0/1229152105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4118f40 msgr2=0x55a3e41232c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 --2- 172.21.15.98:0/1229152105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4118f40 0x55a3e41232c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 -- 172.21.15.98:0/1229152105 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3e3d90210 con 0x55a3e4118620 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 --2- 172.21.15.98:0/1229152105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e4118a00 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6b34012d00 tx=0x7f6b34010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6110a43f92dda4aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b3ad9d640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6b280a3dc0 0x7f6b280a6280 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009fd0 tx=0x7f6b34017000 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b30ff9640 1 -- 172.21.15.98:0/1229152105 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34022070 con 0x55a3e4118620 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b30ff9640 1 -- 172.21.15.98:0/1229152105 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b340124f0 con 0x55a3e4118620 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.975+0000 7f6b30ff9640 1 -- 172.21.15.98:0/1229152105 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3401e660 con 0x55a3e4118620 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.976+0000 7f6b43c54740 1 -- 172.21.15.98:0/1229152105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 msgr2=0x55a3e4118a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.976+0000 7f6b43c54740 1 --2- 172.21.15.98:0/1229152105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e4118a00 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6b34012d00 tx=0x7f6b34010740 comp rx=0 tx=0).stop 2024-04-25T17:47:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.976+0000 7f6b43c54740 1 -- 172.21.15.98:0/1229152105 shutdown_connections 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.976+0000 7f6b43c54740 1 --2- 172.21.15.98:0/1229152105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4123800 0x55a3e4125bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.976+0000 7f6b43c54740 1 --2- 172.21.15.98:0/1229152105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4118f40 0x55a3e41232c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.976+0000 7f6b43c54740 1 --2- 172.21.15.98:0/1229152105 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e4118a00 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.976+0000 7f6b43c54740 1 -- 172.21.15.98:0/1229152105 >> 172.21.15.98:0/1229152105 conn(0x55a3e4114b30 msgr2=0x55a3e4114f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.976+0000 7f6b43c54740 1 -- 172.21.15.98:0/1229152105 shutdown_connections 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.976+0000 7f6b43c54740 1 -- 172.21.15.98:0/1229152105 wait complete. 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b43c54740 1 -- start start 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b43c54740 1 Processor -- start 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 0x55a3e41231b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e41204a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e4120a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b43c54740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a3e41263d0 con 0x55a3e4118f40 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b43c54740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a3e40180b0 con 0x55a3e4123800 2024-04-25T17:47:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b43c54740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a3e4126550 con 0x55a3e4118620 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b43c54740 10 client.0 _ll_register_callbacks cb 0x55a3e4005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e41204a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e41204a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42498/0 (socket says 172.21.15.98:42498) 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b33fff640 1 -- 172.21.15.98:0/3322590362 learned_addr learned my addr 172.21.15.98:0/3322590362 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b3ad9d640 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e4120a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b3a59c640 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 0x55a3e41231b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b33fff640 1 -- 172.21.15.98:0/3322590362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 msgr2=0x55a3e41231b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b33fff640 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 0x55a3e41231b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b33fff640 1 -- 172.21.15.98:0/3322590362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 msgr2=0x55a3e4120a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b33fff640 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e4120a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.977+0000 7f6b33fff640 1 -- 172.21.15.98:0/3322590362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3e3e5c620 con 0x55a3e4118f40 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b33fff640 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e41204a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6b24031440 tx=0x7f6b240b9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8502ce63e26ee73d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b21ffb640 1 -- 172.21.15.98:0/3322590362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b24049040 con 0x55a3e4118f40 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b21ffb640 1 -- 172.21.15.98:0/3322590362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f6b240ee680 con 0x55a3e4118f40 2024-04-25T17:47:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b43c54740 1 -- 172.21.15.98:0/3322590362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b340124f0 con 0x55a3e4118f40 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b21ffb640 1 -- 172.21.15.98:0/3322590362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f6b240c2190 con 0x55a3e4118f40 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b43c54740 10 client.16494 fetch_fsmap learned FSMap version 44 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b43c54740 1 -- 172.21.15.98:0/3322590362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a3e4132340 con 0x55a3e4118f40 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b21ffb640 1 -- 172.21.15.98:0/3322590362 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b340124f0 con 0x55a3e4118f40 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b21ffb640 1 -- 172.21.15.98:0/3322590362 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f6b240b92c0 con 0x55a3e4118f40 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b43c54740 10 client.16494 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b43c54740 10 client.16494 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b43c54740 20 client.16494 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b43c54740 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a3e4133840 0x55a3e4135c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b43c54740 4 client.16494 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.978+0000 7f6b43c54740 1 -- 172.21.15.98:0/3322590362 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3e408b390 con 0x55a3e4133840 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.979+0000 7f6b3a59c640 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a3e4133840 0x55a3e4135c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.979+0000 7f6b21ffb640 10 client.16494 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.979+0000 7f6b3a59c640 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a3e4133840 0x55a3e4135c30 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60fcce35e69aa7b1 server_cookie=deb82fe721c4c45d in_seq=0 out_seq=0 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b21ffb640 1 -- 172.21.15.98:0/3322590362 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a3e408b390 con 0x55a3e4133840 2024-04-25T17:47:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b21ffb640 10 client.16494 handle_command_reply: tid=0 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 client.16494 shutdown 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 -- 172.21.15.98:0/3322590362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 msgr2=0x55a3e41204a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e41204a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6b24031440 tx=0x7f6b240b9000 comp rx=0 tx=0).stop 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 -- 172.21.15.98:0/3322590362 shutdown_connections 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a3e4133840 0x55a3e4135c30 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e4120a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e41204a0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 --2- 172.21.15.98:0/3322590362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 0x55a3e41231b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 -- 172.21.15.98:0/3322590362 >> 172.21.15.98:0/3322590362 conn(0x55a3e4114b30 msgr2=0x55a3e411a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b21ffb640 0 client.16494 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 -- 172.21.15.98:0/3322590362 shutdown_connections 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 1 -- 172.21.15.98:0/3322590362 wait complete. 2024-04-25T17:47:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.980+0000 7f6b43c54740 20 client.16494 trim_cache size 0 max 16384 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b43c54740 1 Processor -- start 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b43c54740 1 -- start start 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 0x55a3e412f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e410fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e411b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b43c54740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1b0400 con 0x55a3e4118f40 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b43c54740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1adc70 con 0x55a3e4123800 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b43c54740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1abae0 con 0x55a3e4118620 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e410fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b33fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e410fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42508/0 (socket says 172.21.15.98:42508) 2024-04-25T17:47:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b33fff640 1 -- 172.21.15.98:0/3628095340 learned_addr learned my addr 172.21.15.98:0/3628095340 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b3a59c640 1 --2- 172.21.15.98:0/3628095340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 0x55a3e412f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b33fff640 1 -- 172.21.15.98:0/3628095340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 msgr2=0x55a3e412f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b33fff640 1 --2- 172.21.15.98:0/3628095340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 0x55a3e412f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b33fff640 1 -- 172.21.15.98:0/3628095340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 msgr2=0x55a3e411b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b33fff640 1 --2- 172.21.15.98:0/3628095340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e411b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b33fff640 1 -- 172.21.15.98:0/3628095340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3e4117e20 con 0x55a3e4118f40 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b3a59c640 1 --2- 172.21.15.98:0/3628095340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 0x55a3e412f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.995+0000 7f6b33fff640 1 --2- 172.21.15.98:0/3628095340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e410fb10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6b240b9620 tx=0x7f6b240bb160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57eb3126c4595eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b23fff640 1 -- 172.21.15.98:0/3628095340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b24049070 con 0x55a3e4118f40 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b23fff640 1 -- 172.21.15.98:0/3628095340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b240c58c0 con 0x55a3e4118f40 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b23fff640 1 -- 172.21.15.98:0/3628095340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b240b92c0 con 0x55a3e4118f40 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b43c54740 1 -- 172.21.15.98:0/3628095340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 msgr2=0x55a3e410fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b43c54740 1 --2- 172.21.15.98:0/3628095340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e410fb10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6b240b9620 tx=0x7f6b240bb160 comp rx=0 tx=0).stop 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b43c54740 1 -- 172.21.15.98:0/3628095340 shutdown_connections 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b43c54740 1 --2- 172.21.15.98:0/3628095340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e411b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b43c54740 1 --2- 172.21.15.98:0/3628095340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118f40 0x55a3e410fb10 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b43c54740 1 --2- 172.21.15.98:0/3628095340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118620 0x55a3e412f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b43c54740 1 -- 172.21.15.98:0/3628095340 >> 172.21.15.98:0/3628095340 conn(0x55a3e4114b30 msgr2=0x55a3e411b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b43c54740 1 -- 172.21.15.98:0/3628095340 shutdown_connections 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b43c54740 1 -- 172.21.15.98:0/3628095340 wait complete. 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.996+0000 7f6b43c54740 1 -- start start 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b43c54740 1 Processor -- start 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e412bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118f40 0x55a3e411cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b43c54740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e411d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b43c54740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1ac130 con 0x55a3e4118620 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b43c54740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1ad5a0 con 0x55a3e4123800 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b43c54740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1ac800 con 0x55a3e4118f40 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b43c54740 10 client.0 _ll_register_callbacks cb 0x7f6b240bca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b3a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e412bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b3a59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e412bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42512/0 (socket says 172.21.15.98:42512) 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b3a59c640 1 -- 172.21.15.98:0/1624623174 learned_addr learned my addr 172.21.15.98:0/1624623174 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b3ad9d640 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e411d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b33fff640 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118f40 0x55a3e411cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b3a59c640 1 -- 172.21.15.98:0/1624623174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118f40 msgr2=0x55a3e411cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b3a59c640 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118f40 0x55a3e411cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b3a59c640 1 -- 172.21.15.98:0/1624623174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 msgr2=0x55a3e411d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b3a59c640 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e411d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b3a59c640 1 -- 172.21.15.98:0/1624623174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3e4198db0 con 0x55a3e4118620 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b3a59c640 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e412bcf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009f40 tx=0x7f6b2c01df20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a94a7b780d141f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.997+0000 7f6b21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b21ffb640 1 -- 172.21.15.98:0/1624623174 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c016ca0 con 0x55a3e4118620 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b21ffb640 1 -- 172.21.15.98:0/1624623174 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8513+0+0 (secure 0 0 0) 0x7f6b2c01d1b0 con 0x55a3e4118620 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b43c54740 1 -- 172.21.15.98:0/1624623174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b240c58c0 con 0x55a3e4118620 2024-04-25T17:47:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b21ffb640 1 -- 172.21.15.98:0/1624623174 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f6b2c016e40 con 0x55a3e4118620 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b43c54740 10 client.16500 fetch_fsmap learned FSMap version 44 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b43c54740 1 -- 172.21.15.98:0/1624623174 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a3e4199f40 con 0x55a3e4118620 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b21ffb640 1 -- 172.21.15.98:0/1624623174 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b240c58c0 con 0x55a3e4118620 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b21ffb640 1 -- 172.21.15.98:0/1624623174 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f6b2c0259e0 con 0x55a3e4118620 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b43c54740 10 client.16500 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b43c54740 10 client.16500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b43c54740 20 client.16500 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b43c54740 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a3e419a670 0x55a3e419ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b43c54740 4 client.16500 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b43c54740 1 -- 172.21.15.98:0/1624623174 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a3e4085200 con 0x55a3e419a670 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.998+0000 7f6b33fff640 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a3e419a670 0x55a3e419ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.999+0000 7f6b33fff640 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a3e419a670 0x55a3e419ca90 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3c0850bf9bf65a2 server_cookie=a4acc17b80f4d64e in_seq=0 out_seq=0 2024-04-25T17:47:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:02.999+0000 7f6b21ffb640 10 client.16500 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:03.303 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:47:03.549 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:47:03.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:03 smithi098 ceph-mon[29192]: pgmap v348: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 KiB/s wr, 31 op/s 2024-04-25T17:47:03.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:03 smithi203 ceph-mon[32834]: pgmap v348: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 KiB/s wr, 31 op/s 2024-04-25T17:47:04.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:03 smithi173 ceph-mon[32856]: pgmap v348: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 68 KiB/s wr, 31 op/s 2024-04-25T17:47:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b21ffb640 1 -- 172.21.15.98:0/1624623174 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a3e4085200 con 0x55a3e419a670 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b21ffb640 10 client.16500 handle_command_reply: tid=0 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 client.16500 shutdown 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 -- 172.21.15.98:0/1624623174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 msgr2=0x55a3e412bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e412bcf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009f40 tx=0x7f6b2c01df20 comp rx=0 tx=0).stop 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 -- 172.21.15.98:0/1624623174 shutdown_connections 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a3e419a670 0x55a3e419ca90 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a3e4123800 0x55a3e411d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a3e4118f40 0x55a3e411cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b21ffb640 0 client.16500 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 --2- 172.21.15.98:0/1624623174 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a3e4118620 0x55a3e412bcf0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 -- 172.21.15.98:0/1624623174 >> 172.21.15.98:0/1624623174 conn(0x55a3e4114b30 msgr2=0x55a3e411a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 -- 172.21.15.98:0/1624623174 shutdown_connections 2024-04-25T17:47:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 1 -- 172.21.15.98:0/1624623174 wait complete. 2024-04-25T17:47:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.369+0000 7f6b43c54740 20 client.16500 trim_cache size 0 max 16384 2024-04-25T17:47:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:47:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:47:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:47:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:47:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 -- 172.21.15.98:0/3115900021 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6b280a3dc0 msgr2=0x7f6b280a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6b280a3dc0 0x7f6b280a6280 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009fd0 tx=0x7f6b34017000 comp rx=0 tx=0).stop 2024-04-25T17:47:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 -- 172.21.15.98:0/3115900021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c107280 msgr2=0x7f6b3c198ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c107280 0x7f6b3c198ce0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6b24032fd0 tx=0x7f6b24004270 comp rx=0 tx=0).stop 2024-04-25T17:47:04.372 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 -- 172.21.15.98:0/3115900021 shutdown_connections 2024-04-25T17:47:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6b280a3dc0 0x7f6b280a6280 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b3c19d010 0x7f6b3c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b3c107280 0x7f6b3c198ce0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 --2- 172.21.15.98:0/3115900021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b3c06a9a0 0x7f6b3c1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 -- 172.21.15.98:0/3115900021 >> 172.21.15.98:0/3115900021 conn(0x7f6b3c077c30 msgr2=0x7f6b3c0ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 -- 172.21.15.98:0/3115900021 shutdown_connections 2024-04-25T17:47:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:04.370+0000 7f6b40f36640 1 -- 172.21.15.98:0/3115900021 wait complete. 2024-04-25T17:47:05.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: *** Caught signal (Segmentation fault) ** 2024-04-25T17:47:05.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: in thread 7fe8b0683700 thread_name:md_log_replay 2024-04-25T17:47:05.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T17:47:05.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8c1a44d20] 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2: /usr/bin/ceph-mds(+0x549372) [0x556323cf7372] 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x556323cfab23] 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 4: (MDLog::_replay_thread()+0x617) [0x556323d55647] 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 5: (MDLog::ReplayThread::entry()+0x11) [0x556323d5ed31] 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fe8c1a3a1ca] 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 7: clone() 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2024-04-25T17:47:04.925+0000 7fe8b0683700 -1 *** Caught signal (Segmentation fault) ** 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: in thread 7fe8b0683700 thread_name:md_log_replay 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8c1a44d20] 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2: /usr/bin/ceph-mds(+0x549372) [0x556323cf7372] 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x556323cfab23] 2024-04-25T17:47:05.213 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 4: (MDLog::_replay_thread()+0x617) [0x556323d55647] 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 5: (MDLog::ReplayThread::entry()+0x11) [0x556323d5ed31] 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fe8c1a3a1ca] 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 7: clone() 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 0> 2024-04-25T17:47:04.925+0000 7fe8b0683700 -1 *** Caught signal (Segmentation fault) ** 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: in thread 7fe8b0683700 thread_name:md_log_replay 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8c1a44d20] 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2: /usr/bin/ceph-mds(+0x549372) [0x556323cf7372] 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x556323cfab23] 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 4: (MDLog::_replay_thread()+0x617) [0x556323d55647] 2024-04-25T17:47:05.214 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 5: (MDLog::ReplayThread::entry()+0x11) [0x556323d5ed31] 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fe8c1a3a1ca] 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 7: clone() 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 0> 2024-04-25T17:47:04.925+0000 7fe8b0683700 -1 *** Caught signal (Segmentation fault) ** 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: in thread 7fe8b0683700 thread_name:md_log_replay 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe8c1a44d20] 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2: /usr/bin/ceph-mds(+0x549372) [0x556323cf7372] 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x556323cfab23] 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 4: (MDLog::_replay_thread()+0x617) [0x556323d55647] 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 5: (MDLog::ReplayThread::entry()+0x11) [0x556323d5ed31] 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7fe8c1a3a1ca] 2024-04-25T17:47:05.215 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 7: clone() 2024-04-25T17:47:05.216 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T17:47:05.216 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[68058]: 2024-04-25T17:47:05.216 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:04 smithi203 podman[69767]: 2024-04-25 17:47:04.998592365 +0000 UTC m=+0.023781125 container died 905b117a93741a29c8d62f9ed84da14c1a550e4c42da7e2706c829a3c8aca1f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T17:47:05.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:05 smithi203 ceph-mon[32834]: pgmap v349: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 23 op/s 2024-04-25T17:47:05.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:05 smithi203 ceph-mon[32834]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T17:47:05.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:05 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3282872807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:47:05.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.227+0000 7f70f53de700 1 -- 172.21.15.98:0/1422386045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70f0101930 msgr2=0x7f70f0101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:05.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.227+0000 7f70f53de700 1 --2- 172.21.15.98:0/1422386045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70f0101930 0x7f70f0101db0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009990 tx=0x7f70e4009ca0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.227+0000 7f70f53de700 1 -- 172.21.15.98:0/1422386045 shutdown_connections 2024-04-25T17:47:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.227+0000 7f70f53de700 1 --2- 172.21.15.98:0/1422386045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70f01022f0 0x7f70f010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.227+0000 7f70f53de700 1 --2- 172.21.15.98:0/1422386045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70f0107980 0x7f70f0107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.227+0000 7f70f53de700 1 --2- 172.21.15.98:0/1422386045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70f0101930 0x7f70f0101db0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.227+0000 7f70f53de700 1 -- 172.21.15.98:0/1422386045 >> 172.21.15.98:0/1422386045 conn(0x7f70f00788b0 msgr2=0x7f70f0078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.227+0000 7f70f53de700 1 -- 172.21.15.98:0/1422386045 shutdown_connections 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.227+0000 7f70f53de700 1 -- 172.21.15.98:0/1422386045 wait complete. 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70f53de700 1 Processor -- start 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70f53de700 1 -- start start 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70f53de700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70f0101930 0x7f70f0197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70f53de700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70f01022f0 0x7f70f0198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70f53de700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70f0107980 0x7f70f01987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70f53de700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f70f0111ad0 con 0x7f70f0101930 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70f53de700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f70f0111950 con 0x7f70f0107980 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70f53de700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f70f0111c50 con 0x7f70f01022f0 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70eeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70f0101930 0x7f70f0197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70eeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70f0101930 0x7f70f0197d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48088/0 (socket says 172.21.15.98:48088) 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70eeffd700 1 -- 172.21.15.98:0/3282872807 learned_addr learned my addr 172.21.15.98:0/3282872807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:05.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70ee7fc700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70f01022f0 0x7f70f0198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.228+0000 7f70ef7fe700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70f0107980 0x7f70f01987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70eeffd700 1 -- 172.21.15.98:0/3282872807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70f01022f0 msgr2=0x7f70f0198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70eeffd700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70f01022f0 0x7f70f0198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70eeffd700 1 -- 172.21.15.98:0/3282872807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70f0107980 msgr2=0x7f70f01987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70eeffd700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70f0107980 0x7f70f01987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70eeffd700 1 -- 172.21.15.98:0/3282872807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f01a75b0 con 0x7f70f0101930 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70eeffd700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70f0101930 0x7f70f0197d40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f70e000c9b0 tx=0x7f70e000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70d7fff700 1 -- 172.21.15.98:0/3282872807 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e0007b00 con 0x7f70f0101930 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70d7fff700 1 -- 172.21.15.98:0/3282872807 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70e0007ca0 con 0x7f70f0101930 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70f53de700 1 -- 172.21.15.98:0/3282872807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70f01a78a0 con 0x7f70f0101930 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70d7fff700 1 -- 172.21.15.98:0/3282872807 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e0018660 con 0x7f70f0101930 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.229+0000 7f70f53de700 1 -- 172.21.15.98:0/3282872807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70f01a7db0 con 0x7f70f0101930 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.230+0000 7f70f53de700 1 -- 172.21.15.98:0/3282872807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70f0069730 con 0x7f70f0101930 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.231+0000 7f70d7fff700 1 -- 172.21.15.98:0/3282872807 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f70e000f450 con 0x7f70f0101930 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.231+0000 7f70d7fff700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f70d80a3db0 0x7f70d80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:05.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.232+0000 7f70ee7fc700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f70d80a3db0 0x7f70d80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:05.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.232+0000 7f70d7fff700 1 -- 172.21.15.98:0/3282872807 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8557+0+0 (secure 0 0 0) 0x7f70e00d0a30 con 0x7f70f0101930 2024-04-25T17:47:05.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.232+0000 7f70ee7fc700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f70d80a3db0 0x7f70d80a6270 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f70e40098e0 tx=0x7f70e4005960 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:05.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.235+0000 7f70d7fff700 1 -- 172.21.15.98:0/3282872807 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f70e0014030 con 0x7f70f0101930 2024-04-25T17:47:05.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.483+0000 7f70f53de700 1 -- 172.21.15.98:0/3282872807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70f002df80 con 0x7f70f0101930 2024-04-25T17:47:05.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.483+0000 7f70d7fff700 1 -- 172.21.15.98:0/3282872807 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11883 (secure 0 0 0) 0x7f70e009d140 con 0x7f70f0101930 2024-04-25T17:47:05.746 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:47:05.746 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:45:52.737355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25195":{"gid":25195,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/1401608926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":1401608926},{"type":"v1","addr":"172.21.15.203:6837","nonce":1401608926}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:47:05.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 -- 172.21.15.98:0/3282872807 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f70d80a3db0 msgr2=0x7f70d80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:05.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f70d80a3db0 0x7f70d80a6270 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f70e40098e0 tx=0x7f70e4005960 comp rx=0 tx=0).stop 2024-04-25T17:47:05.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 -- 172.21.15.98:0/3282872807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70f0101930 msgr2=0x7f70f0197d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70f0101930 0x7f70f0197d40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f70e000c9b0 tx=0x7f70e000ccc0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 -- 172.21.15.98:0/3282872807 shutdown_connections 2024-04-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f70d80a3db0 0x7f70d80a6270 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70f0107980 0x7f70f01987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70f0101930 0x7f70f0197d40 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 --2- 172.21.15.98:0/3282872807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70f01022f0 0x7f70f0198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 -- 172.21.15.98:0/3282872807 >> 172.21.15.98:0/3282872807 conn(0x7f70f00788b0 msgr2=0x7f70f00ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 -- 172.21.15.98:0/3282872807 shutdown_connections 2024-04-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:05.486+0000 7f70f53de700 1 -- 172.21.15.98:0/3282872807 wait complete. 2024-04-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 44 2024-04-25T17:47:05.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:05 smithi098 ceph-mon[29192]: pgmap v349: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 23 op/s 2024-04-25T17:47:05.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:05 smithi098 ceph-mon[29192]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T17:47:05.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:05 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3282872807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:47:06.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:05 smithi173 ceph-mon[32856]: pgmap v349: 97 pgs: 97 active+clean; 4.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 KiB/s wr, 23 op/s 2024-04-25T17:47:06.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:05 smithi173 ceph-mon[32856]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T17:47:06.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:05 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3282872807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:47:06.211 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:05 smithi203 podman[69767]: 2024-04-25 17:47:05.859695625 +0000 UTC m=+0.884884392 container remove 905b117a93741a29c8d62f9ed84da14c1a550e4c42da7e2706c829a3c8aca1f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T17:47:06.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:05 smithi203 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-04-25T17:47:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 635.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:47:06.711 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:06 smithi203 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-25T17:47:06.711 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:06 smithi203 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.753s CPU time. 2024-04-25T17:47:07.225 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id 3b78fc9f --release --await' 2024-04-25T17:47:07.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id 3b78fc9f --release --await 2024-04-25T17:47:07.468 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:47:07.854 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:07 smithi098 ceph-mon[29192]: pgmap v351: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 477 op/s 2024-04-25T17:47:07.962 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:07 smithi203 ceph-mon[32834]: pgmap v351: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 477 op/s 2024-04-25T17:47:08.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:07 smithi173 ceph-mon[32856]: pgmap v351: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 477 op/s 2024-04-25T17:47:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.494+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3896540484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f006a750 msgr2=0x7fb6f006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.494+0000 7fb6f4eba700 1 --2- 172.21.15.98:0/3896540484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f006a750 0x7fb6f006abb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4009990 tx=0x7fb6e4009ca0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.494+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3896540484 shutdown_connections 2024-04-25T17:47:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.494+0000 7fb6f4eba700 1 --2- 172.21.15.98:0/3896540484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f006b0f0 0x7fb6f010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.494+0000 7fb6f4eba700 1 --2- 172.21.15.98:0/3896540484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6f0108a30 0x7fb6f0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.494+0000 7fb6f4eba700 1 --2- 172.21.15.98:0/3896540484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f006a750 0x7fb6f006abb0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.494+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3896540484 >> 172.21.15.98:0/3896540484 conn(0x7fb6f00788f0 msgr2=0x7fb6f0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.494+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3896540484 shutdown_connections 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.494+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3896540484 wait complete. 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6f4eba700 1 Processor -- start 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6f4eba700 1 -- start start 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6f4eba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f006a750 0x7fb6f0199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6f4eba700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6f006b0f0 0x7fb6f0199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6f4eba700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f0108a30 0x7fb6f019d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6f4eba700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f0112bd0 con 0x7fb6f006a750 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6f4eba700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f0112a50 con 0x7fb6f006b0f0 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6f4eba700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f0112d50 con 0x7fb6f0108a30 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6ee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f006a750 0x7fb6f0199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6ee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f006a750 0x7fb6f0199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48102/0 (socket says 172.21.15.98:48102) 2024-04-25T17:47:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6ee59c700 1 -- 172.21.15.98:0/3370633345 learned_addr learned my addr 172.21.15.98:0/3370633345 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f0108a30 0x7fb6f019d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6edd9b700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6f006b0f0 0x7fb6f0199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6ee59c700 1 -- 172.21.15.98:0/3370633345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f0108a30 msgr2=0x7fb6f019d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f0108a30 0x7fb6f019d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6ee59c700 1 -- 172.21.15.98:0/3370633345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6f006b0f0 msgr2=0x7fb6f0199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.495+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6f006b0f0 0x7fb6f0199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.496+0000 7fb6ee59c700 1 -- 172.21.15.98:0/3370633345 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f019e0e0 con 0x7fb6f006a750 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.496+0000 7fb6edd9b700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6f006b0f0 0x7fb6f0199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.496+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f0108a30 0x7fb6f019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.496+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f006a750 0x7fb6f0199120 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000c920 tx=0x7fb6e000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.496+0000 7fb6df7fe700 1 -- 172.21.15.98:0/3370633345 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0007b00 con 0x7fb6f006a750 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.496+0000 7fb6df7fe700 1 -- 172.21.15.98:0/3370633345 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6e0004ce0 con 0x7fb6f006a750 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.496+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3370633345 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6f019e3d0 con 0x7fb6f006a750 2024-04-25T17:47:09.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.496+0000 7fb6df7fe700 1 -- 172.21.15.98:0/3370633345 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0005650 con 0x7fb6f006a750 2024-04-25T17:47:09.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.496+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3370633345 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6f01a5c50 con 0x7fb6f006a750 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.498+0000 7fb6df7fe700 1 -- 172.21.15.98:0/3370633345 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb6e0004750 con 0x7fb6f006a750 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.498+0000 7fb6df7fe700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6d80a3db0 0x7fb6d80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.499+0000 7fb6edd9b700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6d80a3db0 0x7fb6d80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.499+0000 7fb6df7fe700 1 -- 172.21.15.98:0/3370633345 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8557+0+0 (secure 0 0 0) 0x7fb6e00d0420 con 0x7fb6f006a750 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.499+0000 7fb6edd9b700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6d80a3db0 0x7fb6d80a6270 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400b5c0 tx=0x7fb6e4005fb0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.499+0000 7fb6ff743b80 1 Processor -- start 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6ff743b80 1 -- start start 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8650 0x55695f6a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8f70 0x55695f6b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6ba670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55695f5fbd90 con 0x55695f6b8280 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55695f5a2a30 con 0x55695f6a8f70 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55695f5f7d50 con 0x55695f6a8650 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6eed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6ba670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6eed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6ba670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48114/0 (socket says 172.21.15.98:48114) 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6eed9d700 1 -- 172.21.15.98:0/590104567 learned_addr learned my addr 172.21.15.98:0/590104567 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6edd9b700 1 --2- 172.21.15.98:0/590104567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8f70 0x55695f6b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/590104567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8650 0x55695f6a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6eed9d700 1 -- 172.21.15.98:0/590104567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8650 msgr2=0x55695f6a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/590104567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8650 0x55695f6a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6eed9d700 1 -- 172.21.15.98:0/590104567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8f70 msgr2=0x55695f6b7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/590104567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8f70 0x55695f6b7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.500+0000 7fb6eed9d700 1 -- 172.21.15.98:0/590104567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55695f566840 con 0x55695f6b8280 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.501+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/590104567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6ba670 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009e60 tx=0x7fb6e80077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c247586cacf814 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.501+0000 7fb6dcff9700 1 -- 172.21.15.98:0/590104567 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8014610 con 0x55695f6b8280 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.501+0000 7fb6dcff9700 1 -- 172.21.15.98:0/590104567 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6e8019070 con 0x55695f6b8280 2024-04-25T17:47:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.501+0000 7fb6dcff9700 1 -- 172.21.15.98:0/590104567 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8013a30 con 0x55695f6b8280 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.501+0000 7fb6ff743b80 1 -- 172.21.15.98:0/590104567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 msgr2=0x55695f6ba670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.501+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/590104567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6ba670 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8009e60 tx=0x7fb6e80077e0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.501+0000 7fb6ff743b80 1 -- 172.21.15.98:0/590104567 shutdown_connections 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.501+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/590104567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8650 0x55695f6a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.501+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/590104567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6ba670 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/590104567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8f70 0x55695f6b7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 -- 172.21.15.98:0/590104567 >> 172.21.15.98:0/590104567 conn(0x55695f5d5990 msgr2=0x55695f6a4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 -- 172.21.15.98:0/590104567 shutdown_connections 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 -- 172.21.15.98:0/590104567 wait complete. 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 -- start start 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 Processor -- start 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8650 0x55695f6b79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6a8f70 0x55695f6b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6b8280 0x55695f6b5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55695f6bad10 con 0x55695f6a8f70 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55695f5b56b0 con 0x55695f6b8280 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55695f6bae90 con 0x55695f6a8650 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6ff743b80 10 client.0 _ll_register_callbacks cb 0x55695f594a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6edd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6a8f70 0x55695f6b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6edd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6a8f70 0x55695f6b4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48126/0 (socket says 172.21.15.98:48126) 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.502+0000 7fb6edd9b700 1 -- 172.21.15.98:0/1548744271 learned_addr learned my addr 172.21.15.98:0/1548744271 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6b8280 0x55695f6b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8650 0x55695f6b79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6edd9b700 1 -- 172.21.15.98:0/1548744271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8650 msgr2=0x55695f6b79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6edd9b700 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8650 0x55695f6b79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6edd9b700 1 -- 172.21.15.98:0/1548744271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6b8280 msgr2=0x55695f6b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6edd9b700 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6b8280 0x55695f6b5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6edd9b700 1 -- 172.21.15.98:0/1548744271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55695f5499d0 con 0x55695f6a8f70 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6edd9b700 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6a8f70 0x55695f6b4cc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400bfd0 tx=0x7fb6e4019050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c34bd7b844e67f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6d5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1548744271 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4034060 con 0x55695f6a8f70 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1548744271 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8557+0+0 (secure 0 0 0) 0x7fb6e400fdf0 con 0x55695f6a8f70 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1548744271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6e8019070 con 0x55695f6a8f70 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1548744271 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb6e4003860 con 0x55695f6a8f70 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6ff743b80 10 client.16518 fetch_fsmap learned FSMap version 44 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1548744271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55695f6c5580 con 0x55695f6a8f70 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.503+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1548744271 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6e8019070 con 0x55695f6a8f70 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.504+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1548744271 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fb6e402f460 con 0x55695f6a8f70 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.504+0000 7fb6ff743b80 10 client.16518 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.504+0000 7fb6ff743b80 10 client.16518 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.504+0000 7fb6ff743b80 20 client.16518 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.504+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55695f6c7bc0 0x55695f6c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.504+0000 7fb6ff743b80 4 client.16518 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:47:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.504+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1548744271 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55695f609580 con 0x55695f6c7bc0 2024-04-25T17:47:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.504+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55695f6c7bc0 0x55695f6c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.504+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55695f6c7bc0 0x55695f6c9fb0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15bb0bf27f2ee38a server_cookie=9d2fa43ed0d423fd in_seq=0 out_seq=0 2024-04-25T17:47:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.504+0000 7fb6d5ffb700 10 client.16518 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1548744271 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55695f609580 con 0x55695f6c7bc0 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6d5ffb700 10 client.16518 handle_command_reply: tid=0 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6ff743b80 1 client.16518 shutdown 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1548744271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6a8f70 msgr2=0x55695f6b4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6a8f70 0x55695f6b4cc0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400bfd0 tx=0x7fb6e4019050 comp rx=0 tx=0).stop 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1548744271 shutdown_connections 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55695f6c7bc0 0x55695f6c9fb0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8650 0x55695f6b79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6b8280 0x55695f6b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1548744271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6a8f70 0x55695f6b4cc0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1548744271 >> 172.21.15.98:0/1548744271 conn(0x55695f5d5990 msgr2=0x55695f6aaff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6d5ffb700 0 client.16518 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.505+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1548744271 shutdown_connections 2024-04-25T17:47:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.506+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1548744271 wait complete. 2024-04-25T17:47:09.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.506+0000 7fb6ff743b80 20 client.16518 trim_cache size 0 max 16384 2024-04-25T17:47:09.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.524+0000 7fb6ff743b80 1 Processor -- start 2024-04-25T17:47:09.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.524+0000 7fb6ff743b80 1 -- start start 2024-04-25T17:47:09.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8650 0x55695f6b6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8f70 0x55695f6c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01adb60 con 0x55695f6b8280 2024-04-25T17:47:09.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01ab390 con 0x55695f6a8650 2024-04-25T17:47:09.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01a9200 con 0x55695f6a8f70 2024-04-25T17:47:09.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6eed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6eed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48138/0 (socket says 172.21.15.98:48138) 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6eed9d700 1 -- 172.21.15.98:0/3478603300 learned_addr learned my addr 172.21.15.98:0/3478603300 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/3478603300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8650 0x55695f6b6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6eed9d700 1 -- 172.21.15.98:0/3478603300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8f70 msgr2=0x55695f6c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/3478603300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8f70 0x55695f6c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6eed9d700 1 -- 172.21.15.98:0/3478603300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8650 msgr2=0x55695f6b6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/3478603300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8650 0x55695f6b6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6eed9d700 1 -- 172.21.15.98:0/3478603300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55695f6b3230 con 0x55695f6b8280 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/3478603300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6cf8d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8018000 tx=0x7fb6e8009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd8f9b89e829c31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6d7fff700 1 -- 172.21.15.98:0/3478603300 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8004c90 con 0x55695f6b8280 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6d7fff700 1 -- 172.21.15.98:0/3478603300 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6e8019040 con 0x55695f6b8280 2024-04-25T17:47:09.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.525+0000 7fb6d7fff700 1 -- 172.21.15.98:0/3478603300 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e80286d0 con 0x55695f6b8280 2024-04-25T17:47:09.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 -- 172.21.15.98:0/3478603300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 msgr2=0x55695f6cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:09.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/3478603300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6cf8d0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8018000 tx=0x7fb6e8009a00 comp rx=0 tx=0).stop 2024-04-25T17:47:09.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 -- 172.21.15.98:0/3478603300 shutdown_connections 2024-04-25T17:47:09.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/3478603300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6cf8d0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/3478603300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8650 0x55695f6b6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/3478603300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8f70 0x55695f6c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 -- 172.21.15.98:0/3478603300 >> 172.21.15.98:0/3478603300 conn(0x55695f5d5990 msgr2=0x55695f6b0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:09.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 -- 172.21.15.98:0/3478603300 shutdown_connections 2024-04-25T17:47:09.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 -- 172.21.15.98:0/3478603300 wait complete. 2024-04-25T17:47:09.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 -- start start 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.526+0000 7fb6ff743b80 1 Processor -- start 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8650 0x55695f6c0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8f70 0x55695f6bb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6ff743b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6bb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01a9850 con 0x55695f6b8280 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01aacc0 con 0x55695f6a8650 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6ff743b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f01a9f20 con 0x55695f6a8f70 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6ff743b80 10 client.0 _ll_register_callbacks cb 0x55695f594a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6eed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6bb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6eed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6bb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48140/0 (socket says 172.21.15.98:48140) 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6eed9d700 1 -- 172.21.15.98:0/1613992655 learned_addr learned my addr 172.21.15.98:0/1613992655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8650 0x55695f6c0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6edd9b700 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8f70 0x55695f6bb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6eed9d700 1 -- 172.21.15.98:0/1613992655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8f70 msgr2=0x55695f6bb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8f70 0x55695f6bb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6eed9d700 1 -- 172.21.15.98:0/1613992655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8650 msgr2=0x55695f6c0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8650 0x55695f6c0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6eed9d700 1 -- 172.21.15.98:0/1613992655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55695f6bbc60 con 0x55695f6b8280 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6edd9b700 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8f70 0x55695f6bb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6eed9d700 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6bb620 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8007820 tx=0x7fb6e80097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dda361aaac4899d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6d5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1613992655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e802e050 con 0x55695f6b8280 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1613992655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6e8019040 con 0x55695f6b8280 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1613992655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8557+0+0 (secure 0 0 0) 0x7fb6e80138f0 con 0x55695f6b8280 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.527+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1613992655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb6e8013af0 con 0x55695f6b8280 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6ff743b80 10 client.16524 fetch_fsmap learned FSMap version 44 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1613992655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55695f72eff0 con 0x55695f6b8280 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1613992655 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6e8019040 con 0x55695f6b8280 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1613992655 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7fb6e803a430 con 0x55695f6b8280 2024-04-25T17:47:09.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6ff743b80 10 client.16524 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T17:47:09.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6ff743b80 10 client.16524 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:47:09.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6ff743b80 20 client.16524 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:09.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55695f72f2b0 0x55695f7316a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:09.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6ff743b80 4 client.16524 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3b78fc9f", "release": true, "await": true}] 2024-04-25T17:47:09.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1613992655 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3b78fc9f", "release": true, "await": true}) -- 0x55695f603620 con 0x55695f72f2b0 2024-04-25T17:47:09.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55695f72f2b0 0x55695f7316a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:09.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6d5ffb700 10 client.16524 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:09.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.528+0000 7fb6ee59c700 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55695f72f2b0 0x55695f7316a0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2c25858ff65c8de server_cookie=520db3edbb0a2790 in_seq=0 out_seq=0 2024-04-25T17:47:09.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.534+0000 7fb6d5ffb700 1 -- 172.21.15.98:0/1613992655 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55695f603620 con 0x55695f72f2b0 2024-04-25T17:47:09.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.534+0000 7fb6d5ffb700 10 client.16524 handle_command_reply: tid=0 2024-04-25T17:47:09.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.534+0000 7fb6ff743b80 1 client.16524 shutdown 2024-04-25T17:47:09.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1613992655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 msgr2=0x55695f6bb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:09.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6bb620 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8007820 tx=0x7fb6e80097e0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1613992655 shutdown_connections 2024-04-25T17:47:09.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55695f72f2b0 0x55695f7316a0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6d5ffb700 0 client.16524 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55695f6b8280 0x55695f6bb620 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55695f6a8650 0x55695f6c0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 1 --2- 172.21.15.98:0/1613992655 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55695f6a8f70 0x55695f6bb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1613992655 >> 172.21.15.98:0/1613992655 conn(0x55695f5d5990 msgr2=0x55695f6a3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1613992655 shutdown_connections 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 1 -- 172.21.15.98:0/1613992655 wait complete. 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6ff743b80 20 client.16524 trim_cache size 0 max 16384 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 44, 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:47:09.537 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 32, 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "3b78fc9f": { 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 32, 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 78.3, 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:47:09.538 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:47:09.539 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:47:09.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:47:09.539 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:47:09.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:47:09.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:47:09.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:47:09.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:47:09.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:47:09.539 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:47:09.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3370633345 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6d80a3db0 msgr2=0x7fb6d80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:09.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6f4eba700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6d80a3db0 0x7fb6d80a6270 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400b5c0 tx=0x7fb6e4005fb0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.535+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3370633345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f006a750 msgr2=0x7fb6f0199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:09.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.536+0000 7fb6f4eba700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f006a750 0x7fb6f0199120 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000c920 tx=0x7fb6e000cce0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.536+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3370633345 shutdown_connections 2024-04-25T17:47:09.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.536+0000 7fb6f4eba700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb6d80a3db0 0x7fb6d80a6270 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.536+0000 7fb6f4eba700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f0108a30 0x7fb6f019d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.536+0000 7fb6f4eba700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb6f006a750 0x7fb6f0199120 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.536+0000 7fb6f4eba700 1 --2- 172.21.15.98:0/3370633345 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6f006b0f0 0x7fb6f0199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:09.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.536+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3370633345 >> 172.21.15.98:0/3370633345 conn(0x7fb6f00788f0 msgr2=0x7fb6f00ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:09.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.536+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3370633345 shutdown_connections 2024-04-25T17:47:09.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:09.536+0000 7fb6f4eba700 1 -- 172.21.15.98:0/3370633345 wait complete. 2024-04-25T17:47:09.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:09 smithi098 ceph-mon[29192]: pgmap v352: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 477 op/s 2024-04-25T17:47:09.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:09 smithi203 ceph-mon[32834]: pgmap v352: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 477 op/s 2024-04-25T17:47:10.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:09 smithi173 ceph-mon[32856]: pgmap v352: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 477 op/s 2024-04-25T17:47:10.283 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3b78fc9f', seconds elapsed: 32.8 2024-04-25T17:47:11.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:11 smithi098 ceph-mon[29192]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T17:47:11.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:11 smithi203 ceph-mon[32834]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T17:47:11.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:11 smithi173 ceph-mon[32856]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T17:47:11.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 640.0 s, 199.2 tps, lat 2721.873 ms stddev 7600.299 2024-04-25T17:47:12.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:12 smithi098 ceph-mon[29192]: pgmap v354: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 MiB/s wr, 566 op/s 2024-04-25T17:47:12.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:12 smithi203 ceph-mon[32834]: pgmap v354: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 MiB/s wr, 566 op/s 2024-04-25T17:47:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:12 smithi173 ceph-mon[32856]: pgmap v354: 97 pgs: 97 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 MiB/s wr, 566 op/s 2024-04-25T17:47:13.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:13 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:47:13.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:13 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:47:13.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:13 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:47:14.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:14 smithi098 ceph-mon[29192]: pgmap v355: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 MiB/s wr, 921 op/s 2024-04-25T17:47:14.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:14 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:47:14.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:14 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:47:14.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:14 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:47:14.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:14 smithi203 ceph-mon[32834]: pgmap v355: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 MiB/s wr, 921 op/s 2024-04-25T17:47:14.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:14 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:47:14.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:14 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:47:14.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:14 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:47:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:14 smithi173 ceph-mon[32856]: pgmap v355: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 MiB/s wr, 921 op/s 2024-04-25T17:47:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:14 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:47:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:14 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:47:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:14 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:47:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:16 smithi098 ceph-mon[29192]: pgmap v356: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 MiB/s wr, 742 op/s 2024-04-25T17:47:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:16 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:47:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:16 smithi203 ceph-mon[32834]: pgmap v356: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 MiB/s wr, 742 op/s 2024-04-25T17:47:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:16 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:47:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:16 smithi173 ceph-mon[32856]: pgmap v356: 97 pgs: 97 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 MiB/s wr, 742 op/s 2024-04-25T17:47:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:16 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:47:16.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 645.0 s, 2562.9 tps, lat 41.913 ms stddev 65.994 2024-04-25T17:47:16.717 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:16 smithi203 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-04-25T17:47:16.717 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:16 smithi203 systemd[1]: Stopped Ceph mds.c for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:47:16.717 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:16 smithi203 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service: Consumed 1.753s CPU time. 2024-04-25T17:47:16.717 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:16 smithi203 systemd[1]: Starting Ceph mds.c for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:47:17.012 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:16 smithi203 podman[69993]: 2024-04-25 17:47:16.620856379 +0000 UTC m=+0.018529158 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:47:17.012 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:16 smithi203 podman[69993]: 2024-04-25 17:47:16.768797493 +0000 UTC m=+0.166470243 container create 9436a344c342df02358250f0b9762281e92e06b51366b45e6a602de1c288fc31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5) 2024-04-25T17:47:17.324 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:17 smithi203 podman[69993]: 2024-04-25 17:47:17.013365646 +0000 UTC m=+0.411038396 container init 9436a344c342df02358250f0b9762281e92e06b51366b45e6a602de1c288fc31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:47:17.325 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:17 smithi203 podman[69993]: 2024-04-25 17:47:17.016450888 +0000 UTC m=+0.414123647 container start 9436a344c342df02358250f0b9762281e92e06b51366b45e6a602de1c288fc31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:47:17.325 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:17 smithi203 ceph-mds[70012]: -- 172.21.15.203:0/3928583199 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d4280a3a40 con 0x55d428076c00 2024-04-25T17:47:17.325 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:17 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[70008]: starting mds.c at 2024-04-25T17:47:17.325 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:17 smithi203 bash[69993]: 9436a344c342df02358250f0b9762281e92e06b51366b45e6a602de1c288fc31 2024-04-25T17:47:17.325 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:47:17 smithi203 systemd[1]: Started Ceph mds.c for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:47:17.325 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:17 smithi203 ceph-mon[32834]: daemon mds.c restarted 2024-04-25T17:47:17.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:17 smithi098 ceph-mon[29192]: daemon mds.c restarted 2024-04-25T17:47:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.489+0000 7f7551d87640 1 -- 172.21.15.98:0/915826924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f754c106360 msgr2=0x7f754c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.489+0000 7f7551d87640 1 --2- 172.21.15.98:0/915826924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f754c106360 0x7f754c106810 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7544009cd0 tx=0x7f754402a970 comp rx=0 tx=0).stop 2024-04-25T17:47:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.489+0000 7f7551d87640 1 -- 172.21.15.98:0/915826924 shutdown_connections 2024-04-25T17:47:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.489+0000 7f7551d87640 1 --2- 172.21.15.98:0/915826924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f754c106360 0x7f754c106810 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.489+0000 7f7551d87640 1 --2- 172.21.15.98:0/915826924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f754c1018d0 0x7f754c105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.489+0000 7f7551d87640 1 --2- 172.21.15.98:0/915826924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f754c100fb0 0x7f754c101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.489+0000 7f7551d87640 1 -- 172.21.15.98:0/915826924 >> 172.21.15.98:0/915826924 conn(0x7f754c078060 msgr2=0x7f754c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.489+0000 7f7551d87640 1 -- 172.21.15.98:0/915826924 shutdown_connections 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.489+0000 7f7551d87640 1 -- 172.21.15.98:0/915826924 wait complete. 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.489+0000 7f7551d87640 1 Processor -- start 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f7551d87640 1 -- start start 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f7551d87640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f754c100fb0 0x7f754c1a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f7551d87640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f754c1018d0 0x7f754c1a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f7551d87640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f754c106360 0x7f754c19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f7551d87640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f754c118830 con 0x7f754c100fb0 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f7551d87640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1186b0 con 0x7f754c106360 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f7551d87640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1189b0 con 0x7f754c1018d0 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754bfff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f754c106360 0x7f754c19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f754c100fb0 0x7f754c1a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f754c100fb0 0x7f754c1a48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36610/0 (socket says 172.21.15.98:36610) 2024-04-25T17:47:17.493 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754b7fe640 1 -- 172.21.15.98:0/390047862 learned_addr learned my addr 172.21.15.98:0/390047862 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754bfff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f754c106360 0x7f754c19ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40348/0 (socket says 172.21.15.98:40348) 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754affd640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f754c1018d0 0x7f754c1a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754b7fe640 1 -- 172.21.15.98:0/390047862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f754c1018d0 msgr2=0x7f754c1a4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754b7fe640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f754c1018d0 0x7f754c1a4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754b7fe640 1 -- 172.21.15.98:0/390047862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f754c106360 msgr2=0x7f754c19ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754b7fe640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f754c106360 0x7f754c19ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754b7fe640 1 -- 172.21.15.98:0/390047862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754c19f1f0 con 0x7f754c100fb0 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754bfff640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f754c106360 0x7f754c19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.490+0000 7f754affd640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f754c1018d0 0x7f754c1a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.491+0000 7f754b7fe640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f754c100fb0 0x7f754c1a48c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f753c00e9f0 tx=0x7f753c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.491+0000 7f7548ff9640 1 -- 172.21.15.98:0/390047862 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c00cd80 con 0x7f754c100fb0 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.491+0000 7f7548ff9640 1 -- 172.21.15.98:0/390047862 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f753c0054d0 con 0x7f754c100fb0 2024-04-25T17:47:17.494 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.491+0000 7f7548ff9640 1 -- 172.21.15.98:0/390047862 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c010690 con 0x7f754c100fb0 2024-04-25T17:47:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.491+0000 7f7551d87640 1 -- 172.21.15.98:0/390047862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754c19f4e0 con 0x7f754c100fb0 2024-04-25T17:47:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.491+0000 7f7551d87640 1 -- 172.21.15.98:0/390047862 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f754c1ab780 con 0x7f754c100fb0 2024-04-25T17:47:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.494+0000 7f7554b7b740 1 Processor -- start 2024-04-25T17:47:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.494+0000 7f7554b7b740 1 -- start start 2024-04-25T17:47:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25b620 0x558c7b25ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25bf40 0x558c7b2662c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b266800 0x558c7b268bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f7554b7b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558c7b1c0970 con 0x558c7b266800 2024-04-25T17:47:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f7554b7b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558c7b15d4a0 con 0x558c7b25bf40 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f7554b7b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c7b1bc930 con 0x558c7b25b620 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25bf40 0x558c7b2662c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25bf40 0x558c7b2662c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40364/0 (socket says 172.21.15.98:40364) 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f754b7fe640 1 -- 172.21.15.98:0/1937023550 learned_addr learned my addr 172.21.15.98:0/1937023550 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f754bfff640 1 --2- 172.21.15.98:0/1937023550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25b620 0x558c7b25ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f754b7fe640 1 -- 172.21.15.98:0/1937023550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25b620 msgr2=0x558c7b25ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f754b7fe640 1 --2- 172.21.15.98:0/1937023550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25b620 0x558c7b25ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f754b7fe640 1 -- 172.21.15.98:0/1937023550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b266800 msgr2=0x558c7b268bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f754b7fe640 1 --2- 172.21.15.98:0/1937023550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b266800 0x558c7b268bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.495+0000 7f754b7fe640 1 -- 172.21.15.98:0/1937023550 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c7aed3210 con 0x558c7b25bf40 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.496+0000 7f7548ff9640 1 -- 172.21.15.98:0/390047862 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f753c004e90 con 0x7f754c100fb0 2024-04-25T17:47:17.497 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.496+0000 7f754b7fe640 1 --2- 172.21.15.98:0/1937023550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25bf40 0x558c7b2662c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f753c0ce200 tx=0x7f753c0d4c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2778c56635eced45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.496+0000 7f7531ffb640 1 -- 172.21.15.98:0/1937023550 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c0e0020 con 0x558c7b25bf40 2024-04-25T17:47:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.496+0000 7f7531ffb640 1 -- 172.21.15.98:0/1937023550 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f753c0d9cc0 con 0x558c7b25bf40 2024-04-25T17:47:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.496+0000 7f7531ffb640 1 -- 172.21.15.98:0/1937023550 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c0db610 con 0x558c7b25bf40 2024-04-25T17:47:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7548ff9640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75340a3ec0 0x7f75340a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f754bfff640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75340a3ec0 0x7f75340a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7554b7b740 1 -- 172.21.15.98:0/1937023550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25bf40 msgr2=0x558c7b2662c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7554b7b740 1 --2- 172.21.15.98:0/1937023550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25bf40 0x558c7b2662c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f753c0ce200 tx=0x7f753c0d4c80 comp rx=0 tx=0).stop 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7548ff9640 1 -- 172.21.15.98:0/390047862 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f753c0d0ba0 con 0x7f754c100fb0 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7554b7b740 1 -- 172.21.15.98:0/1937023550 shutdown_connections 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7554b7b740 1 --2- 172.21.15.98:0/1937023550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b266800 0x558c7b268bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7554b7b740 1 --2- 172.21.15.98:0/1937023550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25bf40 0x558c7b2662c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7554b7b740 1 --2- 172.21.15.98:0/1937023550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25b620 0x558c7b25ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7554b7b740 1 -- 172.21.15.98:0/1937023550 >> 172.21.15.98:0/1937023550 conn(0x558c7b257b30 msgr2=0x558c7b257f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7554b7b740 1 -- 172.21.15.98:0/1937023550 shutdown_connections 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7554b7b740 1 -- 172.21.15.98:0/1937023550 wait complete. 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f754bfff640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75340a3ec0 0x7f75340a6380 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7544004af0 tx=0x7f754403c040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.497+0000 7f7554b7b740 1 -- start start 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f7554b7b740 1 Processor -- start 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b266180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b263470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 0x558c7b2639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f7554b7b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558c7b269430 con 0x558c7b25b620 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f7554b7b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558c7b15b0b0 con 0x558c7b266800 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f7554b7b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c7b2695b0 con 0x558c7b25bf40 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f7554b7b740 10 client.0 _ll_register_callbacks cb 0x558c7b148d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b266180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f754bfff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 0x558c7b2639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f754bfff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 0x558c7b2639f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40370/0 (socket says 172.21.15.98:40370) 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f754bfff640 1 -- 172.21.15.98:0/2712351789 learned_addr learned my addr 172.21.15.98:0/2712351789 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.498+0000 7f754affd640 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b263470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f754b7fe640 1 -- 172.21.15.98:0/2712351789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 msgr2=0x558c7b263470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f754b7fe640 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b263470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f754b7fe640 1 -- 172.21.15.98:0/2712351789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 msgr2=0x558c7b2639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f754b7fe640 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 0x558c7b2639f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f754b7fe640 1 -- 172.21.15.98:0/2712351789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c7af9f620 con 0x558c7b25b620 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f754bfff640 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 0x558c7b2639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f754affd640 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b263470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f754b7fe640 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b266180 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f753c00c6a0 tx=0x7f753c004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b340f5e2106702 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f752affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f752affd640 1 -- 172.21.15.98:0/2712351789 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c0e0070 con 0x558c7b25b620 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f7554b7b740 1 -- 172.21.15.98:0/2712351789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f753c0d9cc0 con 0x558c7b25b620 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f752affd640 1 -- 172.21.15.98:0/2712351789 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f753c09ae10 con 0x558c7b25b620 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f752affd640 1 -- 172.21.15.98:0/2712351789 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f753c098db0 con 0x558c7b25b620 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f7554b7b740 10 client.16539 fetch_fsmap learned FSMap version 47 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f7554b7b740 1 -- 172.21.15.98:0/2712351789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558c7b2753f0 con 0x558c7b25b620 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.499+0000 7f752affd640 1 -- 172.21.15.98:0/2712351789 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f753c0d9cc0 con 0x558c7b25b620 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.500+0000 7f752affd640 1 -- 172.21.15.98:0/2712351789 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f753c099180 con 0x558c7b25b620 2024-04-25T17:47:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.500+0000 7f7554b7b740 10 client.16539 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.500+0000 7f7554b7b740 10 client.16539 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.500+0000 7f7554b7b740 20 client.16539 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.500+0000 7f7554b7b740 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558c7b276740 0x558c7b278b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.500+0000 7f7554b7b740 4 client.16539 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.500+0000 7f7554b7b740 1 -- 172.21.15.98:0/2712351789 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c7b1ce390 con 0x558c7b276740 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.500+0000 7f754affd640 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558c7b276740 0x558c7b278b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.500+0000 7f752affd640 10 client.16539 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.500+0000 7f754affd640 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558c7b276740 0x558c7b278b30 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8047abc9c1a39dd server_cookie=6112cbc62db6e50d in_seq=0 out_seq=0 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.501+0000 7f752affd640 1 -- 172.21.15.98:0/2712351789 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c7b1ce390 con 0x558c7b276740 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.501+0000 7f752affd640 10 client.16539 handle_command_reply: tid=0 2024-04-25T17:47:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.502+0000 7f7554b7b740 1 client.16539 shutdown 2024-04-25T17:47:17.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.503+0000 7f7554b7b740 1 -- 172.21.15.98:0/2712351789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 msgr2=0x558c7b266180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:17.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.503+0000 7f7554b7b740 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b266180 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f753c00c6a0 tx=0x7f753c004990 comp rx=0 tx=0).stop 2024-04-25T17:47:17.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.503+0000 7f7554b7b740 1 -- 172.21.15.98:0/2712351789 shutdown_connections 2024-04-25T17:47:17.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.503+0000 7f7554b7b740 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558c7b276740 0x558c7b278b30 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.507 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.503+0000 7f7554b7b740 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 0x558c7b2639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.503+0000 7f7554b7b740 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b263470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.503+0000 7f7554b7b740 1 --2- 172.21.15.98:0/2712351789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b266180 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.504+0000 7f7554b7b740 1 -- 172.21.15.98:0/2712351789 >> 172.21.15.98:0/2712351789 conn(0x558c7b257b30 msgr2=0x558c7b25d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:17.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.504+0000 7f752affd640 0 client.16539 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:17.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.504+0000 7f7554b7b740 1 -- 172.21.15.98:0/2712351789 shutdown_connections 2024-04-25T17:47:17.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.504+0000 7f7554b7b740 1 -- 172.21.15.98:0/2712351789 wait complete. 2024-04-25T17:47:17.508 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.504+0000 7f7554b7b740 20 client.16539 trim_cache size 0 max 16384 2024-04-25T17:47:17.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:17 smithi173 ceph-mon[32856]: daemon mds.c restarted 2024-04-25T17:47:17.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.528+0000 7f7554b7b740 1 Processor -- start 2024-04-25T17:47:17.532 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.532+0000 7f7554b7b740 1 -- start start 2024-04-25T17:47:17.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.532+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b265cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.532+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b27b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.533 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.532+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 0x558c7b27bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.532+0000 7f7554b7b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b6a10 con 0x558c7b25b620 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.532+0000 7f7554b7b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b4280 con 0x558c7b266800 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.532+0000 7f7554b7b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b20d0 con 0x558c7b25bf40 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.532+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b265cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.532+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b265cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36618/0 (socket says 172.21.15.98:36618) 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.532+0000 7f754b7fe640 1 -- 172.21.15.98:0/261909775 learned_addr learned my addr 172.21.15.98:0/261909775 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.533+0000 7f754b7fe640 1 -- 172.21.15.98:0/261909775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 msgr2=0x558c7b27b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.533+0000 7f754b7fe640 1 --2- 172.21.15.98:0/261909775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b27b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.533+0000 7f754b7fe640 1 -- 172.21.15.98:0/261909775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 msgr2=0x558c7b27bcb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.533+0000 7f754b7fe640 1 --2- 172.21.15.98:0/261909775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 0x558c7b27bcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.533+0000 7f754b7fe640 1 -- 172.21.15.98:0/261909775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c7b263d60 con 0x558c7b25b620 2024-04-25T17:47:17.534 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.533+0000 7f754b7fe640 1 --2- 172.21.15.98:0/261909775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b265cb0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f753c09caa0 tx=0x7f753c09d210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424b605abc197442 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.537+0000 7f75317fa640 1 -- 172.21.15.98:0/261909775 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c0e0070 con 0x558c7b25b620 2024-04-25T17:47:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.537+0000 7f75317fa640 1 -- 172.21.15.98:0/261909775 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f753c0d96a0 con 0x558c7b25b620 2024-04-25T17:47:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.538+0000 7f7554b7b740 1 -- 172.21.15.98:0/261909775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 msgr2=0x558c7b265cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.538+0000 7f7554b7b740 1 --2- 172.21.15.98:0/261909775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b265cb0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f753c09caa0 tx=0x7f753c09d210 comp rx=0 tx=0).stop 2024-04-25T17:47:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.543+0000 7f7554b7b740 1 -- 172.21.15.98:0/261909775 shutdown_connections 2024-04-25T17:47:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.543+0000 7f7554b7b740 1 --2- 172.21.15.98:0/261909775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b266800 0x558c7b27bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.543+0000 7f7554b7b740 1 --2- 172.21.15.98:0/261909775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b27b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.543+0000 7f7554b7b740 1 --2- 172.21.15.98:0/261909775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b25b620 0x558c7b265cb0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.543+0000 7f7554b7b740 1 -- 172.21.15.98:0/261909775 >> 172.21.15.98:0/261909775 conn(0x558c7b257b30 msgr2=0x558c7b257f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.549+0000 7f7554b7b740 1 -- 172.21.15.98:0/261909775 shutdown_connections 2024-04-25T17:47:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.549+0000 7f7554b7b740 1 -- 172.21.15.98:0/261909775 wait complete. 2024-04-25T17:47:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.550+0000 7f7554b7b740 1 -- start start 2024-04-25T17:47:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.550+0000 7f7554b7b740 1 Processor -- start 2024-04-25T17:47:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.553+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25b620 0x558c7b278750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.553+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b278c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.553 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.553+0000 7f7554b7b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b266800 0x558c7b2791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.553+0000 7f7554b7b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b2740 con 0x558c7b266800 2024-04-25T17:47:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.553+0000 7f7554b7b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b3bb0 con 0x558c7b25b620 2024-04-25T17:47:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.553+0000 7f7554b7b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b2e10 con 0x558c7b25bf40 2024-04-25T17:47:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.553+0000 7f7554b7b740 10 client.0 _ll_register_callbacks cb 0x7f753c00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.554+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25b620 0x558c7b278750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.554+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25b620 0x558c7b278750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40378/0 (socket says 172.21.15.98:40378) 2024-04-25T17:47:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.554+0000 7f754b7fe640 1 -- 172.21.15.98:0/743950940 learned_addr learned my addr 172.21.15.98:0/743950940 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.554+0000 7f754affd640 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b278c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.556+0000 7f754b7fe640 1 -- 172.21.15.98:0/743950940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 msgr2=0x558c7b278c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.556+0000 7f754b7fe640 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b278c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.556+0000 7f754b7fe640 1 -- 172.21.15.98:0/743950940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b266800 msgr2=0x558c7b2791d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.556+0000 7f754b7fe640 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b266800 0x558c7b2791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.556+0000 7f754b7fe640 1 -- 172.21.15.98:0/743950940 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c7b2da9e0 con 0x558c7b25b620 2024-04-25T17:47:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.559+0000 7f754b7fe640 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25b620 0x558c7b278750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f753c0bd0b0 tx=0x7f753c0d4c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9a11729776eb047 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.559+0000 7f752affd640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:47:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.559+0000 7f752affd640 1 -- 172.21.15.98:0/743950940 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c0e0070 con 0x558c7b25b620 2024-04-25T17:47:17.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.560+0000 7f752affd640 1 -- 172.21.15.98:0/743950940 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f753c0d4460 con 0x558c7b25b620 2024-04-25T17:47:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.560+0000 7f7554b7b740 1 -- 172.21.15.98:0/743950940 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f753c0d96a0 con 0x558c7b25b620 2024-04-25T17:47:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.560+0000 7f752affd640 1 -- 172.21.15.98:0/743950940 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f753c09c470 con 0x558c7b25b620 2024-04-25T17:47:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.561+0000 7f7554b7b740 10 client.26012 fetch_fsmap learned FSMap version 47 2024-04-25T17:47:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.561+0000 7f7554b7b740 1 -- 172.21.15.98:0/743950940 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558c7b2da9e0 con 0x558c7b25b620 2024-04-25T17:47:17.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.561+0000 7f752affd640 1 -- 172.21.15.98:0/743950940 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f753c0d96a0 con 0x558c7b25b620 2024-04-25T17:47:17.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.561+0000 7f752affd640 1 -- 172.21.15.98:0/743950940 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f753c087360 con 0x558c7b25b620 2024-04-25T17:47:17.562 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.561+0000 7f7554b7b740 10 client.26012 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T17:47:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.561+0000 7f7554b7b740 10 client.26012 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:47:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.561+0000 7f7554b7b740 20 client.26012 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.561+0000 7f7554b7b740 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558c7b2dbef0 0x558c7b2de2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.561+0000 7f7554b7b740 4 client.26012 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:47:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.561+0000 7f7554b7b740 1 -- 172.21.15.98:0/743950940 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c7b1c8200 con 0x558c7b2dbef0 2024-04-25T17:47:17.563 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.563+0000 7f754affd640 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558c7b2dbef0 0x558c7b2de2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.570+0000 7f754affd640 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558c7b2dbef0 0x558c7b2de2e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc75ed5e5290c752 server_cookie=4f8657be3bbcef17 in_seq=0 out_seq=0 2024-04-25T17:47:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.570+0000 7f752affd640 10 client.26012 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f752affd640 1 -- 172.21.15.98:0/743950940 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558c7b1c8200 con 0x558c7b2dbef0 2024-04-25T17:47:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f752affd640 10 client.26012 handle_command_reply: tid=0 2024-04-25T17:47:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f7554b7b740 1 client.26012 shutdown 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f7554b7b740 1 -- 172.21.15.98:0/743950940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25b620 msgr2=0x558c7b278750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f7554b7b740 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25b620 0x558c7b278750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f753c0bd0b0 tx=0x7f753c0d4c80 comp rx=0 tx=0).stop 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f7554b7b740 1 -- 172.21.15.98:0/743950940 shutdown_connections 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f7554b7b740 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x558c7b2dbef0 0x558c7b2de2e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f7554b7b740 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558c7b266800 0x558c7b2791d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f7554b7b740 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c7b25bf40 0x558c7b278c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f7554b7b740 1 --2- 172.21.15.98:0/743950940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558c7b25b620 0x558c7b278750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.608+0000 7f7554b7b740 1 -- 172.21.15.98:0/743950940 >> 172.21.15.98:0/743950940 conn(0x558c7b257b30 msgr2=0x558c7b262d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.609+0000 7f752affd640 0 client.26012 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.609+0000 7f7554b7b740 1 -- 172.21.15.98:0/743950940 shutdown_connections 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.609+0000 7f7554b7b740 1 -- 172.21.15.98:0/743950940 wait complete. 2024-04-25T17:47:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.609+0000 7f7554b7b740 20 client.26012 trim_cache size 0 max 16384 2024-04-25T17:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:47:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.609+0000 7f7551d87640 1 -- 172.21.15.98:0/390047862 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75340a3ec0 msgr2=0x7f75340a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.609+0000 7f7551d87640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75340a3ec0 0x7f75340a6380 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7544004af0 tx=0x7f754403c040 comp rx=0 tx=0).stop 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.609+0000 7f7551d87640 1 -- 172.21.15.98:0/390047862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f754c100fb0 msgr2=0x7f754c1a48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.610+0000 7f7551d87640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f754c100fb0 0x7f754c1a48c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f753c00e9f0 tx=0x7f753c00edf0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.610+0000 7f7551d87640 1 -- 172.21.15.98:0/390047862 shutdown_connections 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.610+0000 7f7551d87640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75340a3ec0 0x7f75340a6380 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.610+0000 7f7551d87640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f754c106360 0x7f754c19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.610+0000 7f7551d87640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f754c1018d0 0x7f754c1a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.610+0000 7f7551d87640 1 --2- 172.21.15.98:0/390047862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f754c100fb0 0x7f754c1a48c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.610+0000 7f7551d87640 1 -- 172.21.15.98:0/390047862 >> 172.21.15.98:0/390047862 conn(0x7f754c078060 msgr2=0x7f754c103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.610+0000 7f7551d87640 1 -- 172.21.15.98:0/390047862 shutdown_connections 2024-04-25T17:47:17.612 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:17.610+0000 7f7551d87640 1 -- 172.21.15.98:0/390047862 wait complete. 2024-04-25T17:47:18.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:18 smithi098 ceph-mon[29192]: pgmap v357: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-04-25T17:47:18.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:18 smithi098 ceph-mon[29192]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T17:47:18.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:18 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.203:6836/3925531495,v1:172.21.15.203:6837/3925531495] up:boot 2024-04-25T17:47:18.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:18 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T17:47:18.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:18 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:47:18.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:18 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:47:18.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:18 smithi203 ceph-mon[32834]: pgmap v357: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-04-25T17:47:18.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:18 smithi203 ceph-mon[32834]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T17:47:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:18 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.203:6836/3925531495,v1:172.21.15.203:6837/3925531495] up:boot 2024-04-25T17:47:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:18 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T17:47:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:18 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:47:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:18 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:47:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:18 smithi173 ceph-mon[32856]: pgmap v357: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-04-25T17:47:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:18 smithi173 ceph-mon[32856]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T17:47:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:18 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.203:6836/3925531495,v1:172.21.15.203:6837/3925531495] up:boot 2024-04-25T17:47:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:18 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T17:47:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:18 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:47:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:18 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:47:20.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:20 smithi098 ceph-mon[29192]: pgmap v359: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 MiB/s wr, 1.25k op/s 2024-04-25T17:47:20.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:20 smithi203 ceph-mon[32834]: pgmap v359: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 MiB/s wr, 1.25k op/s 2024-04-25T17:47:20.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:20 smithi173 ceph-mon[32856]: pgmap v359: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 MiB/s wr, 1.25k op/s 2024-04-25T17:47:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 650.0 s, 2465.0 tps, lat 40.424 ms stddev 44.307 2024-04-25T17:47:22.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:22 smithi098 ceph-mon[29192]: pgmap v360: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-04-25T17:47:22.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:22 smithi203 ceph-mon[32834]: pgmap v360: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-04-25T17:47:22.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:22 smithi173 ceph-mon[32856]: pgmap v360: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 MiB/s wr, 1.05k op/s 2024-04-25T17:47:24.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:24 smithi098 ceph-mon[29192]: pgmap v361: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 MiB/s wr, 1.10k op/s 2024-04-25T17:47:24.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:24 smithi203 ceph-mon[32834]: pgmap v361: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 MiB/s wr, 1.10k op/s 2024-04-25T17:47:24.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:24 smithi173 ceph-mon[32856]: pgmap v361: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 MiB/s wr, 1.10k op/s 2024-04-25T17:47:26.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:26 smithi098 ceph-mon[29192]: pgmap v362: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 MiB/s wr, 1.10k op/s 2024-04-25T17:47:26.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:26 smithi203 ceph-mon[32834]: pgmap v362: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 MiB/s wr, 1.10k op/s 2024-04-25T17:47:26.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:26 smithi173 ceph-mon[32856]: pgmap v362: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 21 MiB/s wr, 1.10k op/s 2024-04-25T17:47:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 655.0 s, 2435.0 tps, lat 41.185 ms stddev 48.197 2024-04-25T17:47:28.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:28 smithi098 ceph-mon[29192]: pgmap v363: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 676 op/s 2024-04-25T17:47:28.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:28 smithi203 ceph-mon[32834]: pgmap v363: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 676 op/s 2024-04-25T17:47:28.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:28 smithi173 ceph-mon[32856]: pgmap v363: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.0 MiB/s wr, 676 op/s 2024-04-25T17:47:30.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:30 smithi098 ceph-mon[29192]: pgmap v364: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.2 MiB/s wr, 596 op/s 2024-04-25T17:47:30.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:30 smithi203 ceph-mon[32834]: pgmap v364: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.2 MiB/s wr, 596 op/s 2024-04-25T17:47:30.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:30 smithi173 ceph-mon[32856]: pgmap v364: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.2 MiB/s wr, 596 op/s 2024-04-25T17:47:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.740+0000 7f013900c640 1 -- 172.21.15.98:0/405696852 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01240318a0 con 0x7f0134103a20 2024-04-25T17:47:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.740+0000 7f013c272640 1 -- 172.21.15.98:0/405696852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0134103a20 msgr2=0x7f013410bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.740+0000 7f013c272640 1 --2- 172.21.15.98:0/405696852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0134103a20 0x7f013410bf00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0124009900 tx=0x7f012402a480 comp rx=0 tx=0).stop 2024-04-25T17:47:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.740+0000 7f013c272640 1 -- 172.21.15.98:0/405696852 shutdown_connections 2024-04-25T17:47:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.740+0000 7f013c272640 1 --2- 172.21.15.98:0/405696852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f013410c520 0x7f013410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.740+0000 7f013c272640 1 --2- 172.21.15.98:0/405696852 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0134103a20 0x7f013410bf00 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.740+0000 7f013c272640 1 --2- 172.21.15.98:0/405696852 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0134103100 0x7f01341034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.740+0000 7f013c272640 1 -- 172.21.15.98:0/405696852 >> 172.21.15.98:0/405696852 conn(0x7f01340fc920 msgr2=0x7f01340fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.740+0000 7f013c272640 1 -- 172.21.15.98:0/405696852 shutdown_connections 2024-04-25T17:47:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.740+0000 7f013c272640 1 -- 172.21.15.98:0/405696852 wait complete. 2024-04-25T17:47:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013c272640 1 Processor -- start 2024-04-25T17:47:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013c272640 1 -- start start 2024-04-25T17:47:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013c272640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0134103100 0x7f013410bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013c272640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0134103a20 0x7f0134108f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013c272640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f013410c520 0x7f01341096b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013c272640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0134102730 con 0x7f0134103a20 2024-04-25T17:47:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013c272640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f01341025b0 con 0x7f013410c520 2024-04-25T17:47:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013c272640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f01341028b0 con 0x7f0134103100 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013980d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0134103a20 0x7f0134108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013a00e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0134103100 0x7f013410bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013a00e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0134103100 0x7f013410bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:43526/0 (socket says 172.21.15.98:43526) 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013a00e640 1 -- 172.21.15.98:0/584005053 learned_addr learned my addr 172.21.15.98:0/584005053 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.741+0000 7f013a80f640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f013410c520 0x7f01341096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.742+0000 7f013a00e640 1 -- 172.21.15.98:0/584005053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f013410c520 msgr2=0x7f01341096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.742+0000 7f013a00e640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f013410c520 0x7f01341096b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.742+0000 7f013a00e640 1 -- 172.21.15.98:0/584005053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0134103a20 msgr2=0x7f0134108f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.742+0000 7f013a00e640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0134103a20 0x7f0134108f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.742+0000 7f013a00e640 1 -- 172.21.15.98:0/584005053 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01340729c0 con 0x7f0134103100 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.742+0000 7f013a00e640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0134103100 0x7f013410bea0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f012c009d50 tx=0x7f012c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.742+0000 7f012b7fe640 1 -- 172.21.15.98:0/584005053 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c003b70 con 0x7f0134103100 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.742+0000 7f013c272640 1 -- 172.21.15.98:0/584005053 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0134072c50 con 0x7f0134103100 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.742+0000 7f013c272640 1 -- 172.21.15.98:0/584005053 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01340731c0 con 0x7f0134103100 2024-04-25T17:47:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.742+0000 7f012b7fe640 1 -- 172.21.15.98:0/584005053 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f012c009820 con 0x7f0134103100 2024-04-25T17:47:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.743+0000 7f012b7fe640 1 -- 172.21.15.98:0/584005053 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c003b70 con 0x7f0134103100 2024-04-25T17:47:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.745+0000 7f013ef83740 1 Processor -- start 2024-04-25T17:47:30.745 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.745+0000 7f013ef83740 1 -- start start 2024-04-25T17:47:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.747+0000 7f013ef83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 0x55f173230a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.747+0000 7f013ef83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 0x55f17323b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.747+0000 7f013ef83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f17323dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.747 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.748+0000 7f013ef83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f173195970 con 0x55f17323b800 2024-04-25T17:47:30.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.748+0000 7f013ef83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f1731324a0 con 0x55f173230f40 2024-04-25T17:47:30.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.748+0000 7f013ef83740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f173191930 con 0x55f173230620 2024-04-25T17:47:30.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.748+0000 7f013a80f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 0x55f173230a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.748+0000 7f013a80f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 0x55f173230a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:43540/0 (socket says 172.21.15.98:43540) 2024-04-25T17:47:30.748 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.748+0000 7f013a80f640 1 -- 172.21.15.98:0/2253783419 learned_addr learned my addr 172.21.15.98:0/2253783419 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.748+0000 7f012b7fe640 1 -- 172.21.15.98:0/584005053 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f012c003fe0 con 0x7f0134103100 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.748+0000 7f013980d640 1 --2- 172.21.15.98:0/2253783419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f17323dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.748+0000 7f013a00e640 1 --2- 172.21.15.98:0/2253783419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 0x55f17323b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f013980d640 1 -- 172.21.15.98:0/2253783419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 msgr2=0x55f173230a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f013980d640 1 --2- 172.21.15.98:0/2253783419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 0x55f173230a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f013980d640 1 -- 172.21.15.98:0/2253783419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 msgr2=0x55f17323b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f013980d640 1 --2- 172.21.15.98:0/2253783419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 0x55f17323b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f013980d640 1 -- 172.21.15.98:0/2253783419 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f172ea8210 con 0x55f17323b800 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f013a00e640 1 --2- 172.21.15.98:0/2253783419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 0x55f17323b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f013980d640 1 --2- 172.21.15.98:0/2253783419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f17323dbf0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f012402aa70 tx=0x7f0124005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50104f2d7d3c8460 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f012b7fe640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f01200a3fc0 0x7f01200a6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f0128bfe640 1 -- 172.21.15.98:0/2253783419 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124035040 con 0x55f17323b800 2024-04-25T17:47:30.750 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f0128bfe640 1 -- 172.21.15.98:0/2253783419 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01240054f0 con 0x55f17323b800 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f0128bfe640 1 -- 172.21.15.98:0/2253783419 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01240035f0 con 0x55f17323b800 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f013a80f640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f01200a3fc0 0x7f01200a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.749+0000 7f012b7fe640 1 -- 172.21.15.98:0/584005053 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f012c0d0a30 con 0x7f0134103100 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.750+0000 7f013a80f640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f01200a3fc0 0x7f01200a6480 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0130009e60 tx=0x7f0130005f90 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.750+0000 7f013ef83740 1 -- 172.21.15.98:0/2253783419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 msgr2=0x55f17323dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.750+0000 7f013ef83740 1 --2- 172.21.15.98:0/2253783419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f17323dbf0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f012402aa70 tx=0x7f0124005c00 comp rx=0 tx=0).stop 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.750+0000 7f013ef83740 1 -- 172.21.15.98:0/2253783419 shutdown_connections 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.750+0000 7f013ef83740 1 --2- 172.21.15.98:0/2253783419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f17323dbf0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.750+0000 7f013ef83740 1 --2- 172.21.15.98:0/2253783419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 0x55f17323b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.750+0000 7f013ef83740 1 --2- 172.21.15.98:0/2253783419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 0x55f173230a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.750+0000 7f013ef83740 1 -- 172.21.15.98:0/2253783419 >> 172.21.15.98:0/2253783419 conn(0x55f17322cb30 msgr2=0x55f17322cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.750+0000 7f013ef83740 1 -- 172.21.15.98:0/2253783419 shutdown_connections 2024-04-25T17:47:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.750+0000 7f013ef83740 1 -- 172.21.15.98:0/2253783419 wait complete. 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013ef83740 1 -- start start 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013ef83740 1 Processor -- start 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013ef83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17323b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013ef83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230f40 0x55f173238370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013ef83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f17323b800 0x55f1732388b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013ef83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f17323e3d0 con 0x55f173230620 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013ef83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f1731300b0 con 0x55f17323b800 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013ef83740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f17323e550 con 0x55f173230f40 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013ef83740 10 client.0 _ll_register_callbacks cb 0x55f17311dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013a00e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17323b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013a00e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17323b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58596/0 (socket says 172.21.15.98:58596) 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013a00e640 1 -- 172.21.15.98:0/4209299727 learned_addr learned my addr 172.21.15.98:0/4209299727 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013980d640 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230f40 0x55f173238370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013a80f640 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f17323b800 0x55f1732388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013a00e640 1 -- 172.21.15.98:0/4209299727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230f40 msgr2=0x55f173238370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013a00e640 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230f40 0x55f173238370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013a00e640 1 -- 172.21.15.98:0/4209299727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f17323b800 msgr2=0x55f1732388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013a00e640 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f17323b800 0x55f1732388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.751+0000 7f013a00e640 1 -- 172.21.15.98:0/4209299727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f172f74620 con 0x55f173230620 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f013a80f640 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f17323b800 0x55f1732388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f013a00e640 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17323b080 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f012c01f000 tx=0x7f012c09c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847e5e13f6adc13b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f011dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f011dffb640 1 -- 172.21.15.98:0/4209299727 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c09d100 con 0x55f173230620 2024-04-25T17:47:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f013ef83740 1 -- 172.21.15.98:0/4209299727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01240054f0 con 0x55f173230620 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f011dffb640 1 -- 172.21.15.98:0/4209299727 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f012c099070 con 0x55f173230620 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f011dffb640 1 -- 172.21.15.98:0/4209299727 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f012c09ca00 con 0x55f173230620 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f013ef83740 10 client.16554 fetch_fsmap learned FSMap version 47 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f013ef83740 1 -- 172.21.15.98:0/4209299727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f17324a310 con 0x55f173230620 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f011dffb640 1 -- 172.21.15.98:0/4209299727 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01240054f0 con 0x55f173230620 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f011dffb640 1 -- 172.21.15.98:0/4209299727 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f012c0ba080 con 0x55f173230620 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f013ef83740 10 client.16554 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f013ef83740 10 client.16554 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f013ef83740 20 client.16554 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.752+0000 7f013ef83740 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f17324b780 0x55f17324dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.753+0000 7f013ef83740 4 client.16554 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.753+0000 7f013ef83740 1 -- 172.21.15.98:0/4209299727 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1731a3390 con 0x55f17324b780 2024-04-25T17:47:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.753+0000 7f013980d640 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f17324b780 0x55f17324dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.753+0000 7f013980d640 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f17324b780 0x55f17324dbb0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed12b0a517d2163 server_cookie=e22c40fe06107d8e in_seq=0 out_seq=0 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.753+0000 7f011dffb640 10 client.16554 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f011dffb640 1 -- 172.21.15.98:0/4209299727 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f1731a3390 con 0x55f17324b780 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f011dffb640 10 client.16554 handle_command_reply: tid=0 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 client.16554 shutdown 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 -- 172.21.15.98:0/4209299727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 msgr2=0x55f17323b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17323b080 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f012c01f000 tx=0x7f012c09c540 comp rx=0 tx=0).stop 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 -- 172.21.15.98:0/4209299727 shutdown_connections 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f17324b780 0x55f17324dbb0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f17323b800 0x55f1732388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f011dffb640 0 client.16554 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230f40 0x55f173238370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 --2- 172.21.15.98:0/4209299727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17323b080 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 -- 172.21.15.98:0/4209299727 >> 172.21.15.98:0/4209299727 conn(0x55f17322cb30 msgr2=0x55f173232860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 -- 172.21.15.98:0/4209299727 shutdown_connections 2024-04-25T17:47:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 1 -- 172.21.15.98:0/4209299727 wait complete. 2024-04-25T17:47:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.754+0000 7f013ef83740 20 client.16554 trim_cache size 0 max 16384 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.774+0000 7f013ef83740 1 Processor -- start 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.774+0000 7f013ef83740 1 -- start start 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.774+0000 7f013ef83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17322bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.774+0000 7f013ef83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230f40 0x55f17323abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.774+0000 7f013ef83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f17323b800 0x55f1732534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.774+0000 7f013ef83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0118010ff0 con 0x55f173230620 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.774+0000 7f013a00e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17322bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.774+0000 7f013ef83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f011800e860 con 0x55f17323b800 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f013a00e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17322bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58600/0 (socket says 172.21.15.98:58600) 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f013ef83740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f011800c6b0 con 0x55f173230f40 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f013a00e640 1 -- 172.21.15.98:0/2714886258 learned_addr learned my addr 172.21.15.98:0/2714886258 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f013980d640 1 --2- 172.21.15.98:0/2714886258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230f40 0x55f17323abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f013a00e640 1 -- 172.21.15.98:0/2714886258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230f40 msgr2=0x55f17323abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f013a00e640 1 --2- 172.21.15.98:0/2714886258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230f40 0x55f17323abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f013a00e640 1 -- 172.21.15.98:0/2714886258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f17323b800 msgr2=0x55f1732534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f013a00e640 1 --2- 172.21.15.98:0/2714886258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f17323b800 0x55f1732534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f013a00e640 1 -- 172.21.15.98:0/2714886258 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f173238c10 con 0x55f173230620 2024-04-25T17:47:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f013a00e640 1 --2- 172.21.15.98:0/2714886258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17322bb30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f012c0ce7b0 tx=0x7f012c0093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8269424f8f5e2acb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f011ffff640 1 -- 172.21.15.98:0/2714886258 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c09d100 con 0x55f173230620 2024-04-25T17:47:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f011ffff640 1 -- 172.21.15.98:0/2714886258 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f012c0b7050 con 0x55f173230620 2024-04-25T17:47:30.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.775+0000 7f011ffff640 1 -- 172.21.15.98:0/2714886258 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c0a5670 con 0x55f173230620 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.776+0000 7f013ef83740 1 -- 172.21.15.98:0/2714886258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 msgr2=0x55f17322bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.776+0000 7f013ef83740 1 --2- 172.21.15.98:0/2714886258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17322bb30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f012c0ce7b0 tx=0x7f012c0093a0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.776+0000 7f013ef83740 1 -- 172.21.15.98:0/2714886258 shutdown_connections 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.776+0000 7f013ef83740 1 --2- 172.21.15.98:0/2714886258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f17323b800 0x55f1732534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.776+0000 7f013ef83740 1 --2- 172.21.15.98:0/2714886258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230f40 0x55f17323abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.776+0000 7f013ef83740 1 --2- 172.21.15.98:0/2714886258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f173230620 0x55f17322bb30 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.776+0000 7f013ef83740 1 -- 172.21.15.98:0/2714886258 >> 172.21.15.98:0/2714886258 conn(0x55f17322cb30 msgr2=0x55f17322b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.776+0000 7f013ef83740 1 -- 172.21.15.98:0/2714886258 shutdown_connections 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.776+0000 7f013ef83740 1 -- 172.21.15.98:0/2714886258 wait complete. 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013ef83740 1 -- start start 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013ef83740 1 Processor -- start 2024-04-25T17:47:30.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013ef83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 0x55f173243520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013ef83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 0x55f173243a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013ef83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f1732409b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013ef83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f011800cd20 con 0x55f17323b800 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013ef83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f011800e190 con 0x55f173230f40 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013ef83740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f011800d3f0 con 0x55f173230620 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013ef83740 10 client.0 _ll_register_callbacks cb 0x7f012c09c7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013a80f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f1732409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013a80f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f1732409b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58610/0 (socket says 172.21.15.98:58610) 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013a80f640 1 -- 172.21.15.98:0/3504036708 learned_addr learned my addr 172.21.15.98:0/3504036708 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013a00e640 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 0x55f173243520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.777+0000 7f013980d640 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 0x55f173243a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f013a80f640 1 -- 172.21.15.98:0/3504036708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 msgr2=0x55f173243520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f013a80f640 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 0x55f173243520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f013a80f640 1 -- 172.21.15.98:0/3504036708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 msgr2=0x55f173243a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:30.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f013a80f640 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 0x55f173243a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f013a80f640 1 -- 172.21.15.98:0/3504036708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1732410f0 con 0x55f17323b800 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f013a80f640 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f1732409b0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f0130009fd0 tx=0x7f01300039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed5120cf1d64d39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f011dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f011dffb640 1 -- 172.21.15.98:0/3504036708 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013002d060 con 0x55f17323b800 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f011dffb640 1 -- 172.21.15.98:0/3504036708 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f0130018070 con 0x55f17323b800 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f013ef83740 1 -- 172.21.15.98:0/3504036708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f012c0b7050 con 0x55f17323b800 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f011dffb640 1 -- 172.21.15.98:0/3504036708 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f0130012e60 con 0x55f17323b800 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f013ef83740 10 client.16560 fetch_fsmap learned FSMap version 47 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f013ef83740 1 -- 172.21.15.98:0/3504036708 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f1732b18f0 con 0x55f17323b800 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.778+0000 7f011dffb640 1 -- 172.21.15.98:0/3504036708 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f012c0b7050 con 0x55f17323b800 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.779+0000 7f011dffb640 1 -- 172.21.15.98:0/3504036708 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f013001d7f0 con 0x55f17323b800 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.779+0000 7f013ef83740 10 client.16560 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.779+0000 7f013ef83740 10 client.16560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:47:30.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.779+0000 7f013ef83740 20 client.16560 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.779+0000 7f013ef83740 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f1732b2070 0x55f1732b4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.779+0000 7f013ef83740 4 client.16560 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:47:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.779+0000 7f013ef83740 1 -- 172.21.15.98:0/3504036708 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f17319d200 con 0x55f1732b2070 2024-04-25T17:47:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.779+0000 7f013a00e640 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f1732b2070 0x55f1732b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.779+0000 7f011dffb640 10 client.16560 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:30.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.779+0000 7f013a00e640 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f1732b2070 0x55f1732b4490 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e68bcc622e02f8f2 server_cookie=a797b36612a9c695 in_seq=0 out_seq=0 2024-04-25T17:47:30.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f011dffb640 1 -- 172.21.15.98:0/3504036708 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f17319d200 con 0x55f1732b2070 2024-04-25T17:47:30.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f011dffb640 10 client.16560 handle_command_reply: tid=0 2024-04-25T17:47:30.789 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f013ef83740 1 client.16560 shutdown 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f013ef83740 1 -- 172.21.15.98:0/3504036708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 msgr2=0x55f1732409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f013ef83740 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f1732409b0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f0130009fd0 tx=0x7f01300039c0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f013ef83740 1 -- 172.21.15.98:0/3504036708 shutdown_connections 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f013ef83740 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55f1732b2070 0x55f1732b4490 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f013ef83740 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f17323b800 0x55f1732409b0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f013ef83740 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f173230f40 0x55f173243a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f013ef83740 1 --2- 172.21.15.98:0/3504036708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f173230620 0x55f173243520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f011dffb640 0 client.16560 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.789+0000 7f013ef83740 1 -- 172.21.15.98:0/3504036708 >> 172.21.15.98:0/3504036708 conn(0x55f17322cb30 msgr2=0x55f173246280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.790+0000 7f013ef83740 1 -- 172.21.15.98:0/3504036708 shutdown_connections 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.790+0000 7f013ef83740 1 -- 172.21.15.98:0/3504036708 wait complete. 2024-04-25T17:47:30.790 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.790+0000 7f013ef83740 20 client.16560 trim_cache size 0 max 16384 2024-04-25T17:47:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:47:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:47:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:47:30.791 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.790+0000 7f013c272640 1 -- 172.21.15.98:0/584005053 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f01200a3fc0 msgr2=0x7f01200a6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.790+0000 7f013c272640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f01200a3fc0 0x7f01200a6480 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0130009e60 tx=0x7f0130005f90 comp rx=0 tx=0).stop 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.790+0000 7f013c272640 1 -- 172.21.15.98:0/584005053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0134103100 msgr2=0x7f013410bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.790+0000 7f013c272640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0134103100 0x7f013410bea0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f012c009d50 tx=0x7f012c00c6a0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.791+0000 7f013c272640 1 -- 172.21.15.98:0/584005053 shutdown_connections 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.791+0000 7f013c272640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f01200a3fc0 0x7f01200a6480 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.791+0000 7f013c272640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f013410c520 0x7f01341096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.791+0000 7f013c272640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0134103a20 0x7f0134108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.791+0000 7f013c272640 1 --2- 172.21.15.98:0/584005053 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0134103100 0x7f013410bea0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.791+0000 7f013c272640 1 -- 172.21.15.98:0/584005053 >> 172.21.15.98:0/584005053 conn(0x7f01340fc920 msgr2=0x7f01341107d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.791+0000 7f013c272640 1 -- 172.21.15.98:0/584005053 shutdown_connections 2024-04-25T17:47:30.792 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:30.791+0000 7f013c272640 1 -- 172.21.15.98:0/584005053 wait complete. 2024-04-25T17:47:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 660.0 s, 1908.6 tps, lat 52.231 ms stddev 56.255 2024-04-25T17:47:32.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:32 smithi098 ceph-mon[29192]: pgmap v365: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.8 MiB/s wr, 564 op/s 2024-04-25T17:47:32.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:32 smithi203 ceph-mon[32834]: pgmap v365: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.8 MiB/s wr, 564 op/s 2024-04-25T17:47:32.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:32 smithi173 ceph-mon[32856]: pgmap v365: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.8 MiB/s wr, 564 op/s 2024-04-25T17:47:34.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:34 smithi098 ceph-mon[29192]: pgmap v366: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 8.4 MiB/s wr, 831 op/s 2024-04-25T17:47:34.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:34 smithi203 ceph-mon[32834]: pgmap v366: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 8.4 MiB/s wr, 831 op/s 2024-04-25T17:47:34.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:34 smithi173 ceph-mon[32856]: pgmap v366: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 8.4 MiB/s wr, 831 op/s 2024-04-25T17:47:36.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:36 smithi098 ceph-mon[29192]: pgmap v367: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.4 MiB/s wr, 547 op/s 2024-04-25T17:47:36.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:36 smithi203 ceph-mon[32834]: pgmap v367: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.4 MiB/s wr, 547 op/s 2024-04-25T17:47:36.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:36 smithi173 ceph-mon[32856]: pgmap v367: 97 pgs: 97 active+clean; 4.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.4 MiB/s wr, 547 op/s 2024-04-25T17:47:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 665.0 s, 2854.4 tps, lat 34.965 ms stddev 40.688 2024-04-25T17:47:38.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:38 smithi203 ceph-mon[32834]: pgmap v368: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 8.2 MiB/s wr, 827 op/s 2024-04-25T17:47:38.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:38 smithi173 ceph-mon[32856]: pgmap v368: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 8.2 MiB/s wr, 827 op/s 2024-04-25T17:47:38.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:38 smithi098 ceph-mon[29192]: pgmap v368: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 8.2 MiB/s wr, 827 op/s 2024-04-25T17:47:40.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:40 smithi173 ceph-mon[32856]: pgmap v369: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.3 MiB/s wr, 547 op/s 2024-04-25T17:47:40.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:40 smithi098 ceph-mon[29192]: pgmap v369: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.3 MiB/s wr, 547 op/s 2024-04-25T17:47:40.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:40 smithi203 ceph-mon[32834]: pgmap v369: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.3 MiB/s wr, 547 op/s 2024-04-25T17:47:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 670.0 s, 1918.4 tps, lat 52.475 ms stddev 61.602 2024-04-25T17:47:42.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:42 smithi173 ceph-mon[32856]: pgmap v370: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.3 MiB/s wr, 547 op/s 2024-04-25T17:47:42.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:42 smithi098 ceph-mon[29192]: pgmap v370: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.3 MiB/s wr, 547 op/s 2024-04-25T17:47:42.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:42 smithi203 ceph-mon[32834]: pgmap v370: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.3 MiB/s wr, 547 op/s 2024-04-25T17:47:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.905+0000 7fe71ece8640 1 -- 172.21.15.98:0/2892040849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7181006c0 msgr2=0x7fe718100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.905+0000 7fe71ece8640 1 --2- 172.21.15.98:0/2892040849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7181006c0 0x7fe718100aa0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009900 tx=0x7fe70c02a410 comp rx=0 tx=0).stop 2024-04-25T17:47:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.906+0000 7fe71ece8640 1 -- 172.21.15.98:0/2892040849 shutdown_connections 2024-04-25T17:47:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.906+0000 7fe71ece8640 1 --2- 172.21.15.98:0/2892040849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe718107280 0x7fe71810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.906+0000 7fe71ece8640 1 --2- 172.21.15.98:0/2892040849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe718101070 0x7fe718106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.906+0000 7fe71ece8640 1 --2- 172.21.15.98:0/2892040849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7181006c0 0x7fe718100aa0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.906+0000 7fe71ece8640 1 -- 172.21.15.98:0/2892040849 >> 172.21.15.98:0/2892040849 conn(0x7fe718077390 msgr2=0x7fe7180fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.906+0000 7fe71ece8640 1 -- 172.21.15.98:0/2892040849 shutdown_connections 2024-04-25T17:47:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.906+0000 7fe71ece8640 1 -- 172.21.15.98:0/2892040849 wait complete. 2024-04-25T17:47:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.907+0000 7fe71ece8640 1 Processor -- start 2024-04-25T17:47:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.907+0000 7fe71ece8640 1 -- start start 2024-04-25T17:47:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.907+0000 7fe71ece8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7181006c0 0x7fe718101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.907+0000 7fe71ece8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe718107280 0x7fe718101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.907+0000 7fe71ece8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7181020b0 0x7fe7181a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.907+0000 7fe71ece8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7181121e0 con 0x7fe7181020b0 2024-04-25T17:47:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.907+0000 7fe71ece8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe718112060 con 0x7fe718107280 2024-04-25T17:47:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.907+0000 7fe71ece8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe718112360 con 0x7fe7181006c0 2024-04-25T17:47:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71d285640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7181020b0 0x7fe7181a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71d285640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7181020b0 0x7fe7181a68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59202/0 (socket says 172.21.15.98:59202) 2024-04-25T17:47:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71d285640 1 -- 172.21.15.98:0/2621229685 learned_addr learned my addr 172.21.15.98:0/2621229685 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe717fff640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe718107280 0x7fe718101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71ca84640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7181006c0 0x7fe718101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71d285640 1 -- 172.21.15.98:0/2621229685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7181006c0 msgr2=0x7fe718101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71d285640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7181006c0 0x7fe718101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71d285640 1 -- 172.21.15.98:0/2621229685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe718107280 msgr2=0x7fe718101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71d285640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe718107280 0x7fe718101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71d285640 1 -- 172.21.15.98:0/2621229685 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7181a6fd0 con 0x7fe7181020b0 2024-04-25T17:47:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71d285640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7181020b0 0x7fe7181a68d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe71000eb10 tx=0x7fe71000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe715ffb640 1 -- 172.21.15.98:0/2621229685 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe710004b20 con 0x7fe7181020b0 2024-04-25T17:47:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71ece8640 1 -- 172.21.15.98:0/2621229685 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7181a72c0 con 0x7fe7181020b0 2024-04-25T17:47:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe715ffb640 1 -- 172.21.15.98:0/2621229685 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe710004cc0 con 0x7fe7181020b0 2024-04-25T17:47:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe715ffb640 1 -- 172.21.15.98:0/2621229685 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe710010670 con 0x7fe7181020b0 2024-04-25T17:47:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.908+0000 7fe71ece8640 1 -- 172.21.15.98:0/2621229685 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7181a77c0 con 0x7fe7181020b0 2024-04-25T17:47:43.912 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.910+0000 7fe715ffb640 1 -- 172.21.15.98:0/2621229685 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe710005000 con 0x7fe7181020b0 2024-04-25T17:47:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.911+0000 7fe721a00740 1 Processor -- start 2024-04-25T17:47:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe721a00740 1 -- start start 2024-04-25T17:47:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe715ffb640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe7080a3e70 0x7fe7080a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe721a00740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314ee30 0x5557f314f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe721a00740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 0x5557f3159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe721a00740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f315a010 0x5557f315c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe721a00740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5557f30ab970 con 0x5557f315a010 2024-04-25T17:47:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe721a00740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5557f30484a0 con 0x5557f314f750 2024-04-25T17:47:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe721a00740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5557f30a7930 con 0x5557f314ee30 2024-04-25T17:47:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe717fff640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe7080a3e70 0x7fe7080a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe715ffb640 1 -- 172.21.15.98:0/2621229685 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7fe7100d0d00 con 0x7fe7181020b0 2024-04-25T17:47:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe71ca84640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314ee30 0x5557f314f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe717fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 0x5557f3159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe71d285640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f315a010 0x5557f315c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe71ca84640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314ee30 0x5557f314f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:41860/0 (socket says 172.21.15.98:41860) 2024-04-25T17:47:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.912+0000 7fe71ca84640 1 -- 172.21.15.98:0/1828396300 learned_addr learned my addr 172.21.15.98:0/1828396300 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.913+0000 7fe717fff640 1 -- 172.21.15.98:0/1828396300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314ee30 msgr2=0x5557f314f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.913+0000 7fe717fff640 1 --2- 172.21.15.98:0/1828396300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314ee30 0x5557f314f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.913+0000 7fe717fff640 1 -- 172.21.15.98:0/1828396300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f315a010 msgr2=0x5557f315c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.913+0000 7fe717fff640 1 --2- 172.21.15.98:0/1828396300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f315a010 0x5557f315c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.913+0000 7fe717fff640 1 -- 172.21.15.98:0/1828396300 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557f2dbe210 con 0x5557f314f750 2024-04-25T17:47:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.913+0000 7fe717fff640 1 --2- 172.21.15.98:0/1828396300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 0x5557f3159ad0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe704005fd0 tx=0x7fe70401c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8ef86a367d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.913+0000 7fe717fff640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe7080a3e70 0x7fe7080a6330 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe70400b620 tx=0x7fe70400bb50 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.913+0000 7fe702ffd640 1 -- 172.21.15.98:0/1828396300 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704024070 con 0x5557f314f750 2024-04-25T17:47:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.913+0000 7fe702ffd640 1 -- 172.21.15.98:0/1828396300 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe704010dd0 con 0x5557f314f750 2024-04-25T17:47:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.913+0000 7fe702ffd640 1 -- 172.21.15.98:0/1828396300 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704004d80 con 0x5557f314f750 2024-04-25T17:47:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.914+0000 7fe721a00740 1 -- 172.21.15.98:0/1828396300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 msgr2=0x5557f3159ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.914+0000 7fe721a00740 1 --2- 172.21.15.98:0/1828396300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 0x5557f3159ad0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe704005fd0 tx=0x7fe70401c000 comp rx=0 tx=0).stop 2024-04-25T17:47:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.914+0000 7fe721a00740 1 -- 172.21.15.98:0/1828396300 shutdown_connections 2024-04-25T17:47:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.914+0000 7fe721a00740 1 --2- 172.21.15.98:0/1828396300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f315a010 0x5557f315c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.914+0000 7fe721a00740 1 --2- 172.21.15.98:0/1828396300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 0x5557f3159ad0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.914+0000 7fe721a00740 1 --2- 172.21.15.98:0/1828396300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314ee30 0x5557f314f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.914+0000 7fe721a00740 1 -- 172.21.15.98:0/1828396300 >> 172.21.15.98:0/1828396300 conn(0x5557f314b340 msgr2=0x5557f314b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 1 -- 172.21.15.98:0/1828396300 shutdown_connections 2024-04-25T17:47:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 1 -- 172.21.15.98:0/1828396300 wait complete. 2024-04-25T17:47:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 1 -- start start 2024-04-25T17:47:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 1 Processor -- start 2024-04-25T17:47:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3159a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314f750 0x5557f3156d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f315a010 0x5557f3157320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5557f316dbf0 con 0x5557f314ee30 2024-04-25T17:47:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5557f30460b0 con 0x5557f315a010 2024-04-25T17:47:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5557f316dd70 con 0x5557f314f750 2024-04-25T17:47:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe721a00740 10 client.0 _ll_register_callbacks cb 0x5557f3033d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:43.928 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.915+0000 7fe71ca84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe71ca84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3159a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59212/0 (socket says 172.21.15.98:59212) 2024-04-25T17:47:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe71ca84640 1 -- 172.21.15.98:0/3877611176 learned_addr learned my addr 172.21.15.98:0/3877611176 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe71d285640 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f315a010 0x5557f3157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe717fff640 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314f750 0x5557f3156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe71d285640 1 -- 172.21.15.98:0/3877611176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314f750 msgr2=0x5557f3156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe71d285640 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314f750 0x5557f3156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe71d285640 1 -- 172.21.15.98:0/3877611176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 msgr2=0x5557f3159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe71d285640 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe71d285640 1 -- 172.21.15.98:0/3877611176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557f2e8a620 con 0x5557f315a010 2024-04-25T17:47:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe71ca84640 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:47:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe71d285640 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f315a010 0x5557f3157320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe7100ce420 tx=0x7fe71009c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef67143ff32c6234 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe6f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:47:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3877611176 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7100a1020 con 0x5557f315a010 2024-04-25T17:47:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe721a00740 1 -- 172.21.15.98:0/3877611176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe704010dd0 con 0x5557f315a010 2024-04-25T17:47:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3877611176 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7fe7100ce6a0 con 0x5557f315a010 2024-04-25T17:47:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3877611176 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fe7100ce8a0 con 0x5557f315a010 2024-04-25T17:47:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe721a00740 10 client.26036 fetch_fsmap learned FSMap version 47 2024-04-25T17:47:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe721a00740 1 -- 172.21.15.98:0/3877611176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5557f3179be0 con 0x5557f315a010 2024-04-25T17:47:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.916+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3877611176 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe704010dd0 con 0x5557f315a010 2024-04-25T17:47:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.917+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3877611176 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7fe71009ced0 con 0x5557f315a010 2024-04-25T17:47:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.917+0000 7fe721a00740 10 client.26036 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T17:47:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.917+0000 7fe721a00740 10 client.26036 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:47:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.917+0000 7fe721a00740 20 client.26036 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.917+0000 7fe721a00740 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557f317b050 0x5557f317d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.917+0000 7fe721a00740 4 client.26036 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:47:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.917+0000 7fe721a00740 1 -- 172.21.15.98:0/3877611176 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557f30b9390 con 0x5557f317b050 2024-04-25T17:47:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.917+0000 7fe71ca84640 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557f317b050 0x5557f317d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.917+0000 7fe6f7fff640 10 client.26036 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.917+0000 7fe71ca84640 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557f317b050 0x5557f317d440 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be3c5e3661abef94 server_cookie=7b8a90991e0e1fd7 in_seq=0 out_seq=0 2024-04-25T17:47:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.918+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3877611176 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557f30b9390 con 0x5557f317b050 2024-04-25T17:47:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.918+0000 7fe6f7fff640 10 client.26036 handle_command_reply: tid=0 2024-04-25T17:47:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.918+0000 7fe721a00740 1 client.26036 shutdown 2024-04-25T17:47:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.918+0000 7fe721a00740 1 -- 172.21.15.98:0/3877611176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f315a010 msgr2=0x5557f3157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.918+0000 7fe721a00740 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f315a010 0x5557f3157320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe7100ce420 tx=0x7fe71009c870 comp rx=0 tx=0).stop 2024-04-25T17:47:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.919+0000 7fe721a00740 1 -- 172.21.15.98:0/3877611176 shutdown_connections 2024-04-25T17:47:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.919+0000 7fe721a00740 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557f317b050 0x5557f317d440 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.919+0000 7fe721a00740 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f315a010 0x5557f3157320 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.919+0000 7fe721a00740 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f314f750 0x5557f3156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.919+0000 7fe721a00740 1 --2- 172.21.15.98:0/3877611176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.919+0000 7fe721a00740 1 -- 172.21.15.98:0/3877611176 >> 172.21.15.98:0/3877611176 conn(0x5557f314b340 msgr2=0x5557f3141cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.919+0000 7fe6f7fff640 0 client.26036 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.919+0000 7fe721a00740 1 -- 172.21.15.98:0/3877611176 shutdown_connections 2024-04-25T17:47:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.919+0000 7fe721a00740 1 -- 172.21.15.98:0/3877611176 wait complete. 2024-04-25T17:47:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.919+0000 7fe721a00740 20 client.26036 trim_cache size 0 max 16384 2024-04-25T17:47:43.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.937+0000 7fe721a00740 1 Processor -- start 2024-04-25T17:47:43.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.937+0000 7fe721a00740 1 -- start start 2024-04-25T17:47:43.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.937+0000 7fe721a00740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314ee30 0x5557f3176ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.937+0000 7fe721a00740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314f750 0x5557f3141b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.937+0000 7fe721a00740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 0x5557f3182d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.937+0000 7fe721a00740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7180720e0 con 0x5557f314f750 2024-04-25T17:47:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.937+0000 7fe721a00740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe71806f950 con 0x5557f314ee30 2024-04-25T17:47:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.937+0000 7fe721a00740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe71806d7a0 con 0x5557f315a010 2024-04-25T17:47:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe717fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314f750 0x5557f3141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe717fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314f750 0x5557f3141b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59214/0 (socket says 172.21.15.98:59214) 2024-04-25T17:47:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe717fff640 1 -- 172.21.15.98:0/2026578434 learned_addr learned my addr 172.21.15.98:0/2026578434 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe71ca84640 1 --2- 172.21.15.98:0/2026578434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314ee30 0x5557f3176ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe71d285640 1 --2- 172.21.15.98:0/2026578434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 0x5557f3182d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe717fff640 1 -- 172.21.15.98:0/2026578434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 msgr2=0x5557f3182d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe717fff640 1 --2- 172.21.15.98:0/2026578434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 0x5557f3182d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe717fff640 1 -- 172.21.15.98:0/2026578434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314ee30 msgr2=0x5557f3176ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe717fff640 1 --2- 172.21.15.98:0/2026578434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314ee30 0x5557f3176ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe717fff640 1 -- 172.21.15.98:0/2026578434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557f317d7b0 con 0x5557f314f750 2024-04-25T17:47:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe717fff640 1 --2- 172.21.15.98:0/2026578434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314f750 0x5557f3141b30 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe704005fd0 tx=0x7fe704014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25077c2308b07466 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe7027fc640 1 -- 172.21.15.98:0/2026578434 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704024040 con 0x5557f314f750 2024-04-25T17:47:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe7027fc640 1 -- 172.21.15.98:0/2026578434 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe704004680 con 0x5557f314f750 2024-04-25T17:47:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.938+0000 7fe7027fc640 1 -- 172.21.15.98:0/2026578434 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe704020b00 con 0x5557f314f750 2024-04-25T17:47:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.939+0000 7fe721a00740 1 -- 172.21.15.98:0/2026578434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314f750 msgr2=0x5557f3141b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.939+0000 7fe721a00740 1 --2- 172.21.15.98:0/2026578434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314f750 0x5557f3141b30 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe704005fd0 tx=0x7fe704014040 comp rx=0 tx=0).stop 2024-04-25T17:47:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.939+0000 7fe721a00740 1 -- 172.21.15.98:0/2026578434 shutdown_connections 2024-04-25T17:47:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.939+0000 7fe721a00740 1 --2- 172.21.15.98:0/2026578434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 0x5557f3182d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.939+0000 7fe721a00740 1 --2- 172.21.15.98:0/2026578434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314f750 0x5557f3141b30 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.939+0000 7fe721a00740 1 --2- 172.21.15.98:0/2026578434 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314ee30 0x5557f3176ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.939+0000 7fe721a00740 1 -- 172.21.15.98:0/2026578434 >> 172.21.15.98:0/2026578434 conn(0x5557f314b340 msgr2=0x5557f3141420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.939+0000 7fe721a00740 1 -- 172.21.15.98:0/2026578434 shutdown_connections 2024-04-25T17:47:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.939+0000 7fe721a00740 1 -- 172.21.15.98:0/2026578434 wait complete. 2024-04-25T17:47:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.939+0000 7fe721a00740 1 -- start start 2024-04-25T17:47:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.940+0000 7fe721a00740 1 Processor -- start 2024-04-25T17:47:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.940+0000 7fe721a00740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3173320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.940+0000 7fe721a00740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 0x5557f31706b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.940+0000 7fe721a00740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 0x5557f3170bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe71ca84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3173320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe721a00740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe71806de10 con 0x5557f314ee30 2024-04-25T17:47:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe71ca84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3173320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59218/0 (socket says 172.21.15.98:59218) 2024-04-25T17:47:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe71ca84640 1 -- 172.21.15.98:0/3628713832 learned_addr learned my addr 172.21.15.98:0/3628713832 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:43.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe717fff640 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 0x5557f31706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe721a00740 1 -- 172.21.15.98:0/3628713832 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe71806f280 con 0x5557f314f750 2024-04-25T17:47:43.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe721a00740 1 -- 172.21.15.98:0/3628713832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe71806e4e0 con 0x5557f315a010 2024-04-25T17:47:43.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe721a00740 10 client.0 _ll_register_callbacks cb 0x7fe71009ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:43.954 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe71d285640 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 0x5557f3170bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe71d285640 1 -- 172.21.15.98:0/3628713832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 msgr2=0x5557f31706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe71d285640 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 0x5557f31706b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe71d285640 1 -- 172.21.15.98:0/3628713832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 msgr2=0x5557f3173320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe71d285640 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3173320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe71d285640 1 -- 172.21.15.98:0/3628713832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557f31713c0 con 0x5557f315a010 2024-04-25T17:47:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe717fff640 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 0x5557f31706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:47:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe71d285640 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 0x5557f3170bf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe710099640 tx=0x7fe71000cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4ba865590aee28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.941+0000 7fe6f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:47:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3628713832 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7100ce8b0 con 0x5557f315a010 2024-04-25T17:47:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe721a00740 1 -- 172.21.15.98:0/3628713832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5557f30b3200 con 0x5557f315a010 2024-04-25T17:47:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3628713832 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7fe7100a5450 con 0x5557f315a010 2024-04-25T17:47:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3628713832 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fe7100b61e0 con 0x5557f315a010 2024-04-25T17:47:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe721a00740 10 client.26077 fetch_fsmap learned FSMap version 47 2024-04-25T17:47:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe721a00740 1 -- 172.21.15.98:0/3628713832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5557f31e1580 con 0x5557f315a010 2024-04-25T17:47:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3628713832 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5557f30b3200 con 0x5557f315a010 2024-04-25T17:47:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3628713832 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7fe7100ceb70 con 0x5557f315a010 2024-04-25T17:47:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe721a00740 10 client.26077 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T17:47:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe721a00740 10 client.26077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:47:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe721a00740 20 client.26077 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe721a00740 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557f31e1ac0 0x5557f31e3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe721a00740 4 client.26077 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:47:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe721a00740 1 -- 172.21.15.98:0/3628713832 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5557f302f5f0 con 0x5557f31e1ac0 2024-04-25T17:47:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.942+0000 7fe71ca84640 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557f31e1ac0 0x5557f31e3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:43.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.943+0000 7fe6f7fff640 10 client.26077 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:43.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.943+0000 7fe71ca84640 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557f31e1ac0 0x5557f31e3eb0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8b6ed60e05dc6e server_cookie=8e7a979b6b06c324 in_seq=0 out_seq=0 2024-04-25T17:47:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe6f7fff640 1 -- 172.21.15.98:0/3628713832 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5557f302f5f0 con 0x5557f31e1ac0 2024-04-25T17:47:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe6f7fff640 10 client.26077 handle_command_reply: tid=0 2024-04-25T17:47:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 client.26077 shutdown 2024-04-25T17:47:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 -- 172.21.15.98:0/3628713832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 msgr2=0x5557f3170bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 0x5557f3170bf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe710099640 tx=0x7fe71000cf70 comp rx=0 tx=0).stop 2024-04-25T17:47:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 -- 172.21.15.98:0/3628713832 shutdown_connections 2024-04-25T17:47:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5557f31e1ac0 0x5557f31e3eb0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557f315a010 0x5557f3170bf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5557f314f750 0x5557f31706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe6f7fff640 0 client.26077 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:43.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 --2- 172.21.15.98:0/3628713832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5557f314ee30 0x5557f3173320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 -- 172.21.15.98:0/3628713832 >> 172.21.15.98:0/3628713832 conn(0x5557f314b340 msgr2=0x5557f3141cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:43.962 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 -- 172.21.15.98:0/3628713832 shutdown_connections 2024-04-25T17:47:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.954+0000 7fe721a00740 1 -- 172.21.15.98:0/3628713832 wait complete. 2024-04-25T17:47:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe721a00740 20 client.26077 trim_cache size 0 max 16384 2024-04-25T17:47:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:47:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:47:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:47:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:47:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 -- 172.21.15.98:0/2621229685 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe7080a3e70 msgr2=0x7fe7080a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe7080a3e70 0x7fe7080a6330 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe70400b620 tx=0x7fe70400bb50 comp rx=0 tx=0).stop 2024-04-25T17:47:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 -- 172.21.15.98:0/2621229685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7181020b0 msgr2=0x7fe7181a68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7181020b0 0x7fe7181a68d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe71000eb10 tx=0x7fe71000c2d0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 -- 172.21.15.98:0/2621229685 shutdown_connections 2024-04-25T17:47:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fe7080a3e70 0x7fe7080a6330 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7181020b0 0x7fe7181a68d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe718107280 0x7fe718101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 --2- 172.21.15.98:0/2621229685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7181006c0 0x7fe718101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 -- 172.21.15.98:0/2621229685 >> 172.21.15.98:0/2621229685 conn(0x7fe718077390 msgr2=0x7fe7180fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 -- 172.21.15.98:0/2621229685 shutdown_connections 2024-04-25T17:47:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:43.955+0000 7fe71ece8640 1 -- 172.21.15.98:0/2621229685 wait complete. 2024-04-25T17:47:44.269 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 16.8 2024-04-25T17:47:44.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:47:44.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:44 smithi173 ceph-mon[32856]: pgmap v371: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 8.4 MiB/s wr, 846 op/s 2024-04-25T17:47:44.585 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:47:44.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:44 smithi098 ceph-mon[29192]: pgmap v371: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 8.4 MiB/s wr, 846 op/s 2024-04-25T17:47:44.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:44 smithi203 ceph-mon[32834]: pgmap v371: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 8.4 MiB/s wr, 846 op/s 2024-04-25T17:47:46.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:46 smithi098 ceph-mon[29192]: pgmap v372: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 5.9 MiB/s wr, 579 op/s 2024-04-25T17:47:46.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:46 smithi173 ceph-mon[32856]: pgmap v372: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 5.9 MiB/s wr, 579 op/s 2024-04-25T17:47:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 675.0 s, 2701.2 tps, lat 36.679 ms stddev 40.555 2024-04-25T17:47:46.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:46 smithi203 ceph-mon[32834]: pgmap v372: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 5.9 MiB/s wr, 579 op/s 2024-04-25T17:47:46.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.738+0000 7faa64f4b700 1 -- 172.21.15.98:0/3270751587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60101940 msgr2=0x7faa60101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:46.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.738+0000 7faa64f4b700 1 --2- 172.21.15.98:0/3270751587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60101940 0x7faa60101dc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faa5400d960 tx=0x7faa5400dc70 comp rx=0 tx=0).stop 2024-04-25T17:47:46.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.739+0000 7faa64f4b700 1 -- 172.21.15.98:0/3270751587 shutdown_connections 2024-04-25T17:47:46.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.739+0000 7faa64f4b700 1 --2- 172.21.15.98:0/3270751587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa60102300 0x7faa6010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:46.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.739+0000 7faa64f4b700 1 --2- 172.21.15.98:0/3270751587 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa60107990 0x7faa60107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:46.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.739+0000 7faa64f4b700 1 --2- 172.21.15.98:0/3270751587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60101940 0x7faa60101dc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:46.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.739+0000 7faa64f4b700 1 -- 172.21.15.98:0/3270751587 >> 172.21.15.98:0/3270751587 conn(0x7faa600788f0 msgr2=0x7faa60078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:46.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.740+0000 7faa64f4b700 1 -- 172.21.15.98:0/3270751587 shutdown_connections 2024-04-25T17:47:46.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.741+0000 7faa64f4b700 1 -- 172.21.15.98:0/3270751587 wait complete. 2024-04-25T17:47:46.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.742+0000 7faa64f4b700 1 Processor -- start 2024-04-25T17:47:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.743+0000 7faa64f4b700 1 -- start start 2024-04-25T17:47:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.743+0000 7faa64f4b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60101940 0x7faa6019db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.743+0000 7faa64f4b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa60102300 0x7faa6019e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.743+0000 7faa64f4b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa60107990 0x7faa60196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.743+0000 7faa64f4b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faa60110b40 con 0x7faa60107990 2024-04-25T17:47:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.743+0000 7faa64f4b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7faa601109c0 con 0x7faa60102300 2024-04-25T17:47:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.743+0000 7faa64f4b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa60110cc0 con 0x7faa60101940 2024-04-25T17:47:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.743+0000 7faa5dd9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa60102300 0x7faa6019e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:46.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.744+0000 7faa5dd9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa60102300 0x7faa6019e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40356/0 (socket says 172.21.15.98:40356) 2024-04-25T17:47:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.744+0000 7faa5dd9b700 1 -- 172.21.15.98:0/3584486388 learned_addr learned my addr 172.21.15.98:0/3584486388 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.744+0000 7faa5e59c700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60101940 0x7faa6019db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.744+0000 7faa5dd9b700 1 -- 172.21.15.98:0/3584486388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60101940 msgr2=0x7faa6019db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.744+0000 7faa5dd9b700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60101940 0x7faa6019db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.744+0000 7faa5dd9b700 1 -- 172.21.15.98:0/3584486388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa60107990 msgr2=0x7faa60196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:46.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.744+0000 7faa5dd9b700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa60107990 0x7faa60196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.744+0000 7faa5dd9b700 1 -- 172.21.15.98:0/3584486388 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa60197410 con 0x7faa60102300 2024-04-25T17:47:46.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.745+0000 7faa5dd9b700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa60102300 0x7faa6019e050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faa540159b0 tx=0x7faa540159e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.748+0000 7faa4f7fe700 1 -- 172.21.15.98:0/3584486388 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5401a070 con 0x7faa60102300 2024-04-25T17:47:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.748+0000 7faa4f7fe700 1 -- 172.21.15.98:0/3584486388 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa5401c070 con 0x7faa60102300 2024-04-25T17:47:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.748+0000 7faa4f7fe700 1 -- 172.21.15.98:0/3584486388 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa540207d0 con 0x7faa60102300 2024-04-25T17:47:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.748+0000 7faa64f4b700 1 -- 172.21.15.98:0/3584486388 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa601976a0 con 0x7faa60102300 2024-04-25T17:47:46.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.749+0000 7faa64f4b700 1 -- 172.21.15.98:0/3584486388 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa6019b640 con 0x7faa60102300 2024-04-25T17:47:46.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.755+0000 7faa64f4b700 1 -- 172.21.15.98:0/3584486388 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa40005330 con 0x7faa60102300 2024-04-25T17:47:46.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.757+0000 7faa4f7fe700 1 -- 172.21.15.98:0/3584486388 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faa5400c070 con 0x7faa60102300 2024-04-25T17:47:46.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.758+0000 7faa4f7fe700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7faa480a3f70 0x7faa480a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:46.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.758+0000 7faa4f7fe700 1 -- 172.21.15.98:0/3584486388 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7faa540d13d0 con 0x7faa60102300 2024-04-25T17:47:46.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.759+0000 7faa5e59c700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7faa480a3f70 0x7faa480a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:46.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.763+0000 7faa5e59c700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7faa480a3f70 0x7faa480a6430 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faa50005950 tx=0x7faa5000b410 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:46.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:46.766+0000 7faa4f7fe700 1 -- 172.21.15.98:0/3584486388 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa54009e20 con 0x7faa60102300 2024-04-25T17:47:47.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.148+0000 7faa64f4b700 1 -- 172.21.15.98:0/3584486388 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa40005620 con 0x7faa60102300 2024-04-25T17:47:47.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.149+0000 7faa4f7fe700 1 -- 172.21.15.98:0/3584486388 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11881 (secure 0 0 0) 0x7faa5409dad0 con 0x7faa60102300 2024-04-25T17:47:47.149 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:47:47.149 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:47:17.179077+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26026":{"gid":26026,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/3925531495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":3925531495},{"type":"v1","addr":"172.21.15.203:6837","nonce":3925531495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:47:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 -- 172.21.15.98:0/3584486388 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7faa480a3f70 msgr2=0x7faa480a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7faa480a3f70 0x7faa480a6430 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faa50005950 tx=0x7faa5000b410 comp rx=0 tx=0).stop 2024-04-25T17:47:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 -- 172.21.15.98:0/3584486388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa60102300 msgr2=0x7faa6019e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa60102300 0x7faa6019e050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faa540159b0 tx=0x7faa540159e0 comp rx=0 tx=0).stop 2024-04-25T17:47:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 -- 172.21.15.98:0/3584486388 shutdown_connections 2024-04-25T17:47:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7faa480a3f70 0x7faa480a6430 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa60101940 0x7faa6019db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faa60107990 0x7faa60196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 --2- 172.21.15.98:0/3584486388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa60102300 0x7faa6019e050 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 -- 172.21.15.98:0/3584486388 >> 172.21.15.98:0/3584486388 conn(0x7faa600788f0 msgr2=0x7faa60100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 -- 172.21.15.98:0/3584486388 shutdown_connections 2024-04-25T17:47:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:47.153+0000 7faa64f4b700 1 -- 172.21.15.98:0/3584486388 wait complete. 2024-04-25T17:47:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 47 2024-04-25T17:47:47.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:47 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3584486388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:47:47.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:47 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3584486388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:47:47.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:47 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3584486388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:47:47.904 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 76.8 --await-for 42.0' 2024-04-25T17:47:47.904 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 76.8 --await-for 42.0 2024-04-25T17:47:48.298 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:47:48.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:48 smithi173 ceph-mon[32856]: pgmap v373: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 9.4 MiB/s wr, 898 op/s 2024-04-25T17:47:48.552 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:48 smithi098 ceph-mon[29192]: pgmap v373: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 9.4 MiB/s wr, 898 op/s 2024-04-25T17:47:48.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:48 smithi203 ceph-mon[32834]: pgmap v373: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 9.4 MiB/s wr, 898 op/s 2024-04-25T17:47:50.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.467+0000 7f132e9f4700 1 -- 172.21.15.98:0/4117489740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f132811c220 msgr2=0x7f132811e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:50.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:50 smithi098 ceph-mon[29192]: pgmap v374: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.6 MiB/s wr, 618 op/s 2024-04-25T17:47:50.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.468+0000 7f132e9f4700 1 --2- 172.21.15.98:0/4117489740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f132811c220 0x7f132811e610 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f132001c0d0 tx=0x7f132001c3e0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.468+0000 7f132e9f4700 1 -- 172.21.15.98:0/4117489740 shutdown_connections 2024-04-25T17:47:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.468+0000 7f132e9f4700 1 --2- 172.21.15.98:0/4117489740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1328111290 0x7f1328109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.468+0000 7f132e9f4700 1 --2- 172.21.15.98:0/4117489740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f132811c220 0x7f132811e610 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.468+0000 7f132e9f4700 1 --2- 172.21.15.98:0/4117489740 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1328109f80 0x7f132810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.468+0000 7f132e9f4700 1 -- 172.21.15.98:0/4117489740 >> 172.21.15.98:0/4117489740 conn(0x7f132806dbb0 msgr2=0x7f132806dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:50.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.472+0000 7f132e9f4700 1 -- 172.21.15.98:0/4117489740 shutdown_connections 2024-04-25T17:47:50.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.475+0000 7f132e9f4700 1 -- 172.21.15.98:0/4117489740 wait complete. 2024-04-25T17:47:50.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.477+0000 7f132e9f4700 1 Processor -- start 2024-04-25T17:47:50.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.477+0000 7f132e9f4700 1 -- start start 2024-04-25T17:47:50.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.478+0000 7f132e9f4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1328109f80 0x7f1328117a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.478+0000 7f132e9f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1328111290 0x7f1328117f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.478+0000 7f132e9f4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f132811c220 0x7f13281a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.478+0000 7f132e9f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13281141f0 con 0x7f1328111290 2024-04-25T17:47:50.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.478+0000 7f132e9f4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1328114070 con 0x7f1328109f80 2024-04-25T17:47:50.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.479+0000 7f132e9f4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1328114370 con 0x7f132811c220 2024-04-25T17:47:50.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.479+0000 7f13277fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1328111290 0x7f1328117f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.479+0000 7f13277fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1328111290 0x7f1328117f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59254/0 (socket says 172.21.15.98:59254) 2024-04-25T17:47:50.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.479+0000 7f13277fe700 1 -- 172.21.15.98:0/3302311103 learned_addr learned my addr 172.21.15.98:0/3302311103 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.481+0000 7f13277fe700 1 -- 172.21.15.98:0/3302311103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f132811c220 msgr2=0x7f13281a4000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T17:47:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.481+0000 7f13277fe700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f132811c220 0x7f13281a4000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.481+0000 7f13277fe700 1 -- 172.21.15.98:0/3302311103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1328109f80 msgr2=0x7f1328117a40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T17:47:50.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.481+0000 7f1327fff700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1328109f80 0x7f1328117a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.482+0000 7f13277fe700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1328109f80 0x7f1328117a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.482+0000 7f13277fe700 1 -- 172.21.15.98:0/3302311103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13281a4780 con 0x7f1328111290 2024-04-25T17:47:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.482+0000 7f13277fe700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1328111290 0x7f1328117f80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f131800e920 tx=0x7f131800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:50.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.482+0000 7f1327fff700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1328109f80 0x7f1328117a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:47:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.482+0000 7f13257fa700 1 -- 172.21.15.98:0/3302311103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131800cbd0 con 0x7f1328111290 2024-04-25T17:47:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.483+0000 7f132e9f4700 1 -- 172.21.15.98:0/3302311103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13281a4a70 con 0x7f1328111290 2024-04-25T17:47:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.483+0000 7f132e9f4700 1 -- 172.21.15.98:0/3302311103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13281ac1c0 con 0x7f1328111290 2024-04-25T17:47:50.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.483+0000 7f13257fa700 1 -- 172.21.15.98:0/3302311103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f131800cd70 con 0x7f1328111290 2024-04-25T17:47:50.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.484+0000 7f13257fa700 1 -- 172.21.15.98:0/3302311103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1318018680 con 0x7f1328111290 2024-04-25T17:47:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.488+0000 7f13257fa700 1 -- 172.21.15.98:0/3302311103 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1318010430 con 0x7f1328111290 2024-04-25T17:47:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.492+0000 7f13257fa700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13100a3ed0 0x7f13100a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.492+0000 7f13257fa700 1 -- 172.21.15.98:0/3302311103 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f13180d09e0 con 0x7f1328111290 2024-04-25T17:47:50.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.493+0000 7f1327fff700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13100a3ed0 0x7f13100a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.493+0000 7f133927db80 1 Processor -- start 2024-04-25T17:47:50.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.493+0000 7f1327fff700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13100a3ed0 0x7f13100a6390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f131c006fd0 tx=0x7f131c008040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:50.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.493+0000 7f133927db80 1 -- start start 2024-04-25T17:47:50.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f133927db80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef26e0 0x56548eef2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f133927db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef3000 0x56548eefd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f133927db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 0x56548ef002f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f133927db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56548ee45e20 con 0x56548eef3000 2024-04-25T17:47:50.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f133927db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56548edecac0 con 0x56548eefdf00 2024-04-25T17:47:50.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f133927db80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56548ee41de0 con 0x56548eef26e0 2024-04-25T17:47:50.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f13277fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef3000 0x56548eefd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f13277fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef3000 0x56548eefd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59268/0 (socket says 172.21.15.98:59268) 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f13277fe700 1 -- 172.21.15.98:0/2503699986 learned_addr learned my addr 172.21.15.98:0/2503699986 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f132cf91700 1 --2- 172.21.15.98:0/2503699986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 0x56548ef002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f13277fe700 1 -- 172.21.15.98:0/2503699986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef26e0 msgr2=0x56548eef2ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f13277fe700 1 --2- 172.21.15.98:0/2503699986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef26e0 0x56548eef2ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f13277fe700 1 -- 172.21.15.98:0/2503699986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 msgr2=0x56548ef002f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f13277fe700 1 --2- 172.21.15.98:0/2503699986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 0x56548ef002f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f13277fe700 1 -- 172.21.15.98:0/2503699986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56548edb08d0 con 0x56548eef3000 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.494+0000 7f132cf91700 1 --2- 172.21.15.98:0/2503699986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 0x56548ef002f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.495+0000 7f13277fe700 1 --2- 172.21.15.98:0/2503699986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef3000 0x56548eefd9c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f13180a5090 tx=0x7f131809c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd601c306c38cd8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:50.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.495+0000 7f130e7fc700 1 -- 172.21.15.98:0/2503699986 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13180a55c0 con 0x56548eef3000 2024-04-25T17:47:50.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.495+0000 7f130e7fc700 1 -- 172.21.15.98:0/2503699986 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13180ce380 con 0x56548eef3000 2024-04-25T17:47:50.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.495+0000 7f130e7fc700 1 -- 172.21.15.98:0/2503699986 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13180bbb10 con 0x56548eef3000 2024-04-25T17:47:50.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.496+0000 7f133927db80 1 -- 172.21.15.98:0/2503699986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef3000 msgr2=0x56548eefd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:50.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.496+0000 7f133927db80 1 --2- 172.21.15.98:0/2503699986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef3000 0x56548eefd9c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f13180a5090 tx=0x7f131809c5c0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.499+0000 7f133927db80 1 -- 172.21.15.98:0/2503699986 shutdown_connections 2024-04-25T17:47:50.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.499+0000 7f133927db80 1 --2- 172.21.15.98:0/2503699986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef26e0 0x56548eef2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.499+0000 7f133927db80 1 --2- 172.21.15.98:0/2503699986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 0x56548ef002f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.499+0000 7f133927db80 1 --2- 172.21.15.98:0/2503699986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef3000 0x56548eefd9c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.499+0000 7f133927db80 1 -- 172.21.15.98:0/2503699986 >> 172.21.15.98:0/2503699986 conn(0x56548ee1fa20 msgr2=0x56548eeeeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:50.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.500+0000 7f133927db80 1 -- 172.21.15.98:0/2503699986 shutdown_connections 2024-04-25T17:47:50.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.500+0000 7f133927db80 1 -- 172.21.15.98:0/2503699986 wait complete. 2024-04-25T17:47:50.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.500+0000 7f133927db80 1 -- start start 2024-04-25T17:47:50.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.500+0000 7f133927db80 1 Processor -- start 2024-04-25T17:47:50.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.501+0000 7f133927db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 0x56548eefd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.501+0000 7f133927db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef3000 0x56548eefaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.501+0000 7f133927db80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eefdf00 0x56548eefaf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.501+0000 7f133927db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56548ef00990 con 0x56548eef26e0 2024-04-25T17:47:50.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.501+0000 7f133927db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56548edff740 con 0x56548eef3000 2024-04-25T17:47:50.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.501+0000 7f133927db80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56548ef00b10 con 0x56548eefdf00 2024-04-25T17:47:50.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.501+0000 7f133927db80 10 client.0 _ll_register_callbacks cb 0x56548eddeab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:50.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.503+0000 7f132cf91700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eefdf00 0x56548eefaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.504+0000 7f132cf91700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eefdf00 0x56548eefaf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:41928/0 (socket says 172.21.15.98:41928) 2024-04-25T17:47:50.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.504+0000 7f132cf91700 1 -- 172.21.15.98:0/1098534578 learned_addr learned my addr 172.21.15.98:0/1098534578 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.504+0000 7f13277fe700 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef3000 0x56548eefaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.504+0000 7f1327fff700 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 0x56548eefd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.507+0000 7f13277fe700 1 -- 172.21.15.98:0/1098534578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eefdf00 msgr2=0x56548eefaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.507+0000 7f13277fe700 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eefdf00 0x56548eefaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.507+0000 7f13277fe700 1 -- 172.21.15.98:0/1098534578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 msgr2=0x56548eefd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:50.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.507+0000 7f13277fe700 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 0x56548eefd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.507+0000 7f13277fe700 1 -- 172.21.15.98:0/1098534578 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56548ed93a60 con 0x56548eef3000 2024-04-25T17:47:50.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.508+0000 7f13277fe700 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef3000 0x56548eefaa50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1318099320 tx=0x7f131809ae60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93da9dec9c040f11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.510+0000 7f13077fe700 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T17:47:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.510+0000 7f13077fe700 1 -- 172.21.15.98:0/1098534578 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13180b8070 con 0x56548eef3000 2024-04-25T17:47:50.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.510+0000 7f13077fe700 1 -- 172.21.15.98:0/1098534578 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f131809cdc0 con 0x56548eef3000 2024-04-25T17:47:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.511+0000 7f133927db80 1 -- 172.21.15.98:0/1098534578 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13180ce380 con 0x56548eef3000 2024-04-25T17:47:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.512+0000 7f13077fe700 1 -- 172.21.15.98:0/1098534578 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f13180b5070 con 0x56548eef3000 2024-04-25T17:47:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.512+0000 7f133927db80 10 client.26054 fetch_fsmap learned FSMap version 47 2024-04-25T17:47:50.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.512+0000 7f133927db80 1 -- 172.21.15.98:0/1098534578 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56548ef0b2e0 con 0x56548eef3000 2024-04-25T17:47:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.514+0000 7f13077fe700 1 -- 172.21.15.98:0/1098534578 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13180ce380 con 0x56548eef3000 2024-04-25T17:47:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.514+0000 7f13077fe700 1 -- 172.21.15.98:0/1098534578 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f131809b810 con 0x56548eef3000 2024-04-25T17:47:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.514+0000 7f133927db80 10 client.26054 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T17:47:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.514+0000 7f133927db80 10 client.26054 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:47:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.514+0000 7f133927db80 20 client.26054 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.514+0000 7f133927db80 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56548ef0d7b0 0x56548ef0fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.514+0000 7f133927db80 4 client.26054 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:47:50.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.514+0000 7f133927db80 1 -- 172.21.15.98:0/1098534578 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56548ee53610 con 0x56548ef0d7b0 2024-04-25T17:47:50.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.516+0000 7f1327fff700 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56548ef0d7b0 0x56548ef0fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:50 smithi173 ceph-mon[32856]: pgmap v374: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.6 MiB/s wr, 618 op/s 2024-04-25T17:47:50.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.526+0000 7f1327fff700 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56548ef0d7b0 0x56548ef0fba0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bacf1d39fbdd1bc server_cookie=d21abf236b8dd2c4 in_seq=0 out_seq=0 2024-04-25T17:47:50.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.527+0000 7f13077fe700 10 client.26054 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:50.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.528+0000 7f13077fe700 1 -- 172.21.15.98:0/1098534578 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56548ee53610 con 0x56548ef0d7b0 2024-04-25T17:47:50.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.528+0000 7f13077fe700 10 client.26054 handle_command_reply: tid=0 2024-04-25T17:47:50.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.529+0000 7f133927db80 1 client.26054 shutdown 2024-04-25T17:47:50.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.530+0000 7f133927db80 1 -- 172.21.15.98:0/1098534578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef3000 msgr2=0x56548eefaa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:50.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.530+0000 7f133927db80 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef3000 0x56548eefaa50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1318099320 tx=0x7f131809ae60 comp rx=0 tx=0).stop 2024-04-25T17:47:50.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.530+0000 7f133927db80 1 -- 172.21.15.98:0/1098534578 shutdown_connections 2024-04-25T17:47:50.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.530+0000 7f133927db80 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56548ef0d7b0 0x56548ef0fba0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.530+0000 7f133927db80 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eefdf00 0x56548eefaf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.530+0000 7f133927db80 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 0x56548eefd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.530+0000 7f133927db80 1 --2- 172.21.15.98:0/1098534578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef3000 0x56548eefaa50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.530+0000 7f133927db80 1 -- 172.21.15.98:0/1098534578 >> 172.21.15.98:0/1098534578 conn(0x56548ee1fa20 msgr2=0x56548eef5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:50.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.530+0000 7f13077fe700 0 client.26054 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:50.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.535+0000 7f133927db80 1 -- 172.21.15.98:0/1098534578 shutdown_connections 2024-04-25T17:47:50.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.537+0000 7f133927db80 1 -- 172.21.15.98:0/1098534578 wait complete. 2024-04-25T17:47:50.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.537+0000 7f133927db80 20 client.26054 trim_cache size 0 max 16384 2024-04-25T17:47:50.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f133927db80 1 Processor -- start 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f133927db80 1 -- start start 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f133927db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 0x56548eeeed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f133927db80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef3000 0x56548eefd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f133927db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 0x56548ef12bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f133927db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f131400ff40 con 0x56548eef26e0 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f133927db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f131400d770 con 0x56548eefdf00 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f133927db80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f131400b5c0 con 0x56548eef3000 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f1327fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 0x56548eeeed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f1327fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 0x56548eeeed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59290/0 (socket says 172.21.15.98:59290) 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f1327fff700 1 -- 172.21.15.98:0/552480232 learned_addr learned my addr 172.21.15.98:0/552480232 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.576+0000 7f132cf91700 1 --2- 172.21.15.98:0/552480232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 0x56548ef12bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.577+0000 7f1327fff700 1 -- 172.21.15.98:0/552480232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef3000 msgr2=0x56548eefd290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.577+0000 7f1327fff700 1 --2- 172.21.15.98:0/552480232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef3000 0x56548eefd290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.577+0000 7f1327fff700 1 -- 172.21.15.98:0/552480232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 msgr2=0x56548ef12bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.577+0000 7f1327fff700 1 --2- 172.21.15.98:0/552480232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 0x56548ef12bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.577+0000 7f1327fff700 1 -- 172.21.15.98:0/552480232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56548eef4170 con 0x56548eef26e0 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.577+0000 7f1327fff700 1 --2- 172.21.15.98:0/552480232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 0x56548eeeed10 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x56548ef00c20 tx=0x7f131c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97970ea81e2c9b99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.577+0000 7f130dffb700 1 -- 172.21.15.98:0/552480232 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c04c030 con 0x56548eef26e0 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.578+0000 7f130dffb700 1 -- 172.21.15.98:0/552480232 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f131c0044d0 con 0x56548eef26e0 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.578+0000 7f130dffb700 1 -- 172.21.15.98:0/552480232 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c019040 con 0x56548eef26e0 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.579+0000 7f133927db80 1 -- 172.21.15.98:0/552480232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 msgr2=0x56548eeeed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.579+0000 7f133927db80 1 --2- 172.21.15.98:0/552480232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 0x56548eeeed10 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x56548ef00c20 tx=0x7f131c00f000 comp rx=0 tx=0).stop 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.579+0000 7f133927db80 1 -- 172.21.15.98:0/552480232 shutdown_connections 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.579+0000 7f133927db80 1 --2- 172.21.15.98:0/552480232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eefdf00 0x56548ef12bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.579+0000 7f133927db80 1 --2- 172.21.15.98:0/552480232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eef26e0 0x56548eeeed10 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.579+0000 7f133927db80 1 --2- 172.21.15.98:0/552480232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef3000 0x56548eefd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.579+0000 7f133927db80 1 -- 172.21.15.98:0/552480232 >> 172.21.15.98:0/552480232 conn(0x56548ee1fa20 msgr2=0x56548eef4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.580+0000 7f133927db80 1 -- 172.21.15.98:0/552480232 shutdown_connections 2024-04-25T17:47:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.582+0000 7f133927db80 1 -- 172.21.15.98:0/552480232 wait complete. 2024-04-25T17:47:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.583+0000 7f133927db80 1 -- start start 2024-04-25T17:47:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.583+0000 7f133927db80 1 Processor -- start 2024-04-25T17:47:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.584+0000 7f133927db80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef26e0 0x56548ef12e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.584+0000 7f133927db80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef3000 0x56548ef0f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.584+0000 7f133927db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eefdf00 0x56548ef0f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.584+0000 7f133927db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f131400bc30 con 0x56548eefdf00 2024-04-25T17:47:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.584+0000 7f133927db80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f131400d0a0 con 0x56548eef26e0 2024-04-25T17:47:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.585+0000 7f133927db80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f131400c300 con 0x56548eef3000 2024-04-25T17:47:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.585+0000 7f133927db80 10 client.0 _ll_register_callbacks cb 0x7f131809d050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.585+0000 7f1327fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef26e0 0x56548ef12e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.585+0000 7f13277fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef3000 0x56548ef0f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.585+0000 7f132cf91700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eefdf00 0x56548ef0f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.585+0000 7f132cf91700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eefdf00 0x56548ef0f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59302/0 (socket says 172.21.15.98:59302) 2024-04-25T17:47:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.585+0000 7f132cf91700 1 -- 172.21.15.98:0/1946261870 learned_addr learned my addr 172.21.15.98:0/1946261870 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.586+0000 7f132cf91700 1 -- 172.21.15.98:0/1946261870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef3000 msgr2=0x56548ef0f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.586+0000 7f132cf91700 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef3000 0x56548ef0f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.586+0000 7f132cf91700 1 -- 172.21.15.98:0/1946261870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef26e0 msgr2=0x56548ef12e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.586+0000 7f132cf91700 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef26e0 0x56548ef12e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.586+0000 7f132cf91700 1 -- 172.21.15.98:0/1946261870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56548ef0fe70 con 0x56548eefdf00 2024-04-25T17:47:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.587+0000 7f132cf91700 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eefdf00 0x56548ef0f760 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f132001cf10 tx=0x7f1320003a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f595683571874e02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.587+0000 7f13077fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:47:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.588+0000 7f13077fe700 1 -- 172.21.15.98:0/1946261870 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13200184f0 con 0x56548eefdf00 2024-04-25T17:47:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.588+0000 7f13077fe700 1 -- 172.21.15.98:0/1946261870 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f1320018690 con 0x56548eefdf00 2024-04-25T17:47:50.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.588+0000 7f133927db80 1 -- 172.21.15.98:0/1946261870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f131c0044d0 con 0x56548eefdf00 2024-04-25T17:47:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.591+0000 7f13077fe700 1 -- 172.21.15.98:0/1946261870 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f1320018890 con 0x56548eefdf00 2024-04-25T17:47:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.591+0000 7f133927db80 10 client.16599 fetch_fsmap learned FSMap version 47 2024-04-25T17:47:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.591+0000 7f133927db80 1 -- 172.21.15.98:0/1946261870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56548ef748b0 con 0x56548eefdf00 2024-04-25T17:47:50.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.594+0000 7f13077fe700 1 -- 172.21.15.98:0/1946261870 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f131c0044d0 con 0x56548eefdf00 2024-04-25T17:47:50.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.594+0000 7f13077fe700 1 -- 172.21.15.98:0/1946261870 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f1320022070 con 0x56548eefdf00 2024-04-25T17:47:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.594+0000 7f133927db80 10 client.16599 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T17:47:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.594+0000 7f133927db80 10 client.16599 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:47:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.594+0000 7f133927db80 20 client.16599 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.594+0000 7f133927db80 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56548ef75080 0x56548ef774a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.594+0000 7f133927db80 4 client.16599 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 76.8, "await_for": 42.0}] 2024-04-25T17:47:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.594+0000 7f133927db80 1 -- 172.21.15.98:0/1946261870 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 76.8, "await_for": 42.0}) -- 0x56548ee4d6b0 con 0x56548ef75080 2024-04-25T17:47:50.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.594+0000 7f1327fff700 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56548ef75080 0x56548ef774a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.603+0000 7f1327fff700 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56548ef75080 0x56548ef774a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=712ab759922f95f server_cookie=5d04ca94570eb77 in_seq=0 out_seq=0 2024-04-25T17:47:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.603+0000 7f13077fe700 10 client.16599 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:50.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:50 smithi203 ceph-mon[32834]: pgmap v374: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.6 MiB/s wr, 618 op/s 2024-04-25T17:47:50.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.963+0000 7f13077fe700 1 -- 172.21.15.98:0/1946261870 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56548ee4d6b0 con 0x56548ef75080 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.963+0000 7f13077fe700 10 client.16599 handle_command_reply: tid=0 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.963+0000 7f133927db80 1 client.16599 shutdown 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 1 -- 172.21.15.98:0/1946261870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eefdf00 msgr2=0x56548ef0f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eefdf00 0x56548ef0f760 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f132001cf10 tx=0x7f1320003a00 comp rx=0 tx=0).stop 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 1 -- 172.21.15.98:0/1946261870 shutdown_connections 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x56548ef75080 0x56548ef774a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56548eefdf00 0x56548ef0f760 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56548eef26e0 0x56548ef12e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 1 --2- 172.21.15.98:0/1946261870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56548eef3000 0x56548ef0f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 1 -- 172.21.15.98:0/1946261870 >> 172.21.15.98:0/1946261870 conn(0x56548ee1fa20 msgr2=0x56548eef9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f13077fe700 0 client.16599 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 1 -- 172.21.15.98:0/1946261870 shutdown_connections 2024-04-25T17:47:50.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 1 -- 172.21.15.98:0/1946261870 wait complete. 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.964+0000 7f133927db80 20 client.16599 trim_cache size 0 max 16384 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 47, 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 34, 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "964d443e": { 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 34, 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:47:50.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 76.8, 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:47:50.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:47:50.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:47:50.967 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:47:50.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 -- 172.21.15.98:0/3302311103 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13100a3ed0 msgr2=0x7f13100a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:50.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13100a3ed0 0x7f13100a6390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f131c006fd0 tx=0x7f131c008040 comp rx=0 tx=0).stop 2024-04-25T17:47:50.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 -- 172.21.15.98:0/3302311103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1328111290 msgr2=0x7f1328117f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:50.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1328111290 0x7f1328117f80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f131800e920 tx=0x7f131800ece0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 -- 172.21.15.98:0/3302311103 shutdown_connections 2024-04-25T17:47:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f13100a3ed0 0x7f13100a6390 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f132811c220 0x7f13281a4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1328109f80 0x7f1328117a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 --2- 172.21.15.98:0/3302311103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1328111290 0x7f1328117f80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 -- 172.21.15.98:0/3302311103 >> 172.21.15.98:0/3302311103 conn(0x7f132806dbb0 msgr2=0x7f132811c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 -- 172.21.15.98:0/3302311103 shutdown_connections 2024-04-25T17:47:50.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:47:50.965+0000 7f132e9f4700 1 -- 172.21.15.98:0/3302311103 wait complete. 2024-04-25T17:47:51.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 680.0 s, 1658.3 tps, lat 42.573 ms stddev 60.663 2024-04-25T17:47:51.672 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '964d443e', quiesce took 7.4 seconds. Will release after: 16.8 2024-04-25T17:47:52.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:52 smithi173 ceph-mon[32856]: pgmap v375: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.6 MiB/s wr, 618 op/s 2024-04-25T17:47:52.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:52 smithi098 ceph-mon[29192]: pgmap v375: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.6 MiB/s wr, 618 op/s 2024-04-25T17:47:52.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:52 smithi203 ceph-mon[32834]: pgmap v375: 97 pgs: 97 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.6 MiB/s wr, 618 op/s 2024-04-25T17:47:54.514 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:54 smithi203 ceph-mon[32834]: pgmap v376: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 31 MiB/s wr, 1.85k op/s 2024-04-25T17:47:54.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:54 smithi173 ceph-mon[32856]: pgmap v376: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 31 MiB/s wr, 1.85k op/s 2024-04-25T17:47:54.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:54 smithi098 ceph-mon[29192]: pgmap v376: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 31 MiB/s wr, 1.85k op/s 2024-04-25T17:47:56.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:56 smithi173 ceph-mon[32856]: pgmap v377: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 1.55k op/s 2024-04-25T17:47:56.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 685.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:47:56.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:56 smithi098 ceph-mon[29192]: pgmap v377: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 1.55k op/s 2024-04-25T17:47:56.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:56 smithi203 ceph-mon[32834]: pgmap v377: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 28 MiB/s wr, 1.55k op/s 2024-04-25T17:47:57.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 -- 172.21.15.98:0/1417501526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b81061b0 msgr2=0x7f86b810dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:57.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 --2- 172.21.15.98:0/1417501526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b81061b0 0x7f86b810dd80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009ca0 tx=0x7f86b002a8c0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 -- 172.21.15.98:0/1417501526 shutdown_connections 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 --2- 172.21.15.98:0/1417501526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b81061b0 0x7f86b810dd80 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 --2- 172.21.15.98:0/1417501526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b8101a90 0x7f86b8105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 --2- 172.21.15.98:0/1417501526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b81010e0 0x7f86b81014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 -- 172.21.15.98:0/1417501526 >> 172.21.15.98:0/1417501526 conn(0x7f86b80771d0 msgr2=0x7f86b80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 -- 172.21.15.98:0/1417501526 shutdown_connections 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 -- 172.21.15.98:0/1417501526 wait complete. 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 Processor -- start 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.060+0000 7f86be269640 1 -- start start 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86be269640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b81010e0 0x7f86b8197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86be269640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b81061b0 0x7f86b8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86be269640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b819be20 0x7f86b819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86be269640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86b8111120 con 0x7f86b819be20 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86be269640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86b8110fa0 con 0x7f86b81061b0 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86be269640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86b81112a0 con 0x7f86b81010e0 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86bc806640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b819be20 0x7f86b819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86bc806640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b819be20 0x7f86b819c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52246/0 (socket says 172.21.15.98:52246) 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86bc806640 1 -- 172.21.15.98:0/1474691537 learned_addr learned my addr 172.21.15.98:0/1474691537 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86b7fff640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b81010e0 0x7f86b8197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86b77fe640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b81061b0 0x7f86b8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86bc806640 1 -- 172.21.15.98:0/1474691537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b81010e0 msgr2=0x7f86b8197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86bc806640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b81010e0 0x7f86b8197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86bc806640 1 -- 172.21.15.98:0/1474691537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b81061b0 msgr2=0x7f86b8197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86bc806640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b81061b0 0x7f86b8197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86bc806640 1 -- 172.21.15.98:0/1474691537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b819c910 con 0x7f86b819be20 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86b77fe640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b81061b0 0x7f86b8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:47:57.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86bc806640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b819be20 0x7f86b819c2d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f86b0005cc0 tx=0x7f86b002f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86b57fa640 1 -- 172.21.15.98:0/1474691537 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b0003cf0 con 0x7f86b819be20 2024-04-25T17:47:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86b57fa640 1 -- 172.21.15.98:0/1474691537 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86b00044f0 con 0x7f86b819be20 2024-04-25T17:47:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86be269640 1 -- 172.21.15.98:0/1474691537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b819cba0 con 0x7f86b819be20 2024-04-25T17:47:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.061+0000 7f86b57fa640 1 -- 172.21.15.98:0/1474691537 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b002fd10 con 0x7f86b819be20 2024-04-25T17:47:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.062+0000 7f86be269640 1 -- 172.21.15.98:0/1474691537 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b81a43c0 con 0x7f86b819be20 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.064+0000 7f86b57fa640 1 -- 172.21.15.98:0/1474691537 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f86b0004040 con 0x7f86b819be20 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.064+0000 7f86c0f77740 1 Processor -- start 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.064+0000 7f86c0f77740 1 -- start start 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.064+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 0x564d1037e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.064+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d1038b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.064+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d1038de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.064+0000 7f86c0f77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564d102e3970 con 0x564d1038ba10 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.064+0000 7f86c0f77740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564d102804a0 con 0x564d1037e4c0 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.064+0000 7f86c0f77740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564d102df930 con 0x564d1037ede0 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.064+0000 7f86b57fa640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f86a00a3dc0 0x7f86a00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d1038de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d1038de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52256/0 (socket says 172.21.15.98:52256) 2024-04-25T17:47:57.071 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b77fe640 1 -- 172.21.15.98:0/2928996027 learned_addr learned my addr 172.21.15.98:0/2928996027 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86bc806640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f86a00a3dc0 0x7f86a00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b7fff640 1 --2- 172.21.15.98:0/2928996027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d1038b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b77fe640 1 --2- 172.21.15.98:0/2928996027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 0x564d1037e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b57fa640 1 -- 172.21.15.98:0/1474691537 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f86b00f1f70 con 0x7f86b819be20 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b77fe640 1 -- 172.21.15.98:0/2928996027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 msgr2=0x564d1038b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b77fe640 1 --2- 172.21.15.98:0/2928996027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d1038b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b77fe640 1 -- 172.21.15.98:0/2928996027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 msgr2=0x564d1037e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b77fe640 1 --2- 172.21.15.98:0/2928996027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 0x564d1037e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b77fe640 1 -- 172.21.15.98:0/2928996027 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d0fff6210 con 0x564d1038ba10 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b7fff640 1 --2- 172.21.15.98:0/2928996027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d1038b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86b77fe640 1 --2- 172.21.15.98:0/2928996027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d1038de00 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f86ac012520 tx=0x7f86ac00b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6d55f280969f71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f86bc806640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f86a00a3dc0 0x7f86a00a6280 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f86b00ba4b0 tx=0x7f86b008ca00 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f869e7fc640 1 -- 172.21.15.98:0/2928996027 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac018030 con 0x564d1038ba10 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f869e7fc640 1 -- 172.21.15.98:0/2928996027 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86ac0106a0 con 0x564d1038ba10 2024-04-25T17:47:57.072 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.065+0000 7f869e7fc640 1 -- 172.21.15.98:0/2928996027 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0037f0 con 0x564d1038ba10 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.066+0000 7f86c0f77740 1 -- 172.21.15.98:0/2928996027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 msgr2=0x564d1038de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.066+0000 7f86c0f77740 1 --2- 172.21.15.98:0/2928996027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d1038de00 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f86ac012520 tx=0x7f86ac00b600 comp rx=0 tx=0).stop 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.066+0000 7f86c0f77740 1 -- 172.21.15.98:0/2928996027 shutdown_connections 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.066+0000 7f86c0f77740 1 --2- 172.21.15.98:0/2928996027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d1038de00 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.066+0000 7f86c0f77740 1 --2- 172.21.15.98:0/2928996027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d1038b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.066+0000 7f86c0f77740 1 --2- 172.21.15.98:0/2928996027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 0x564d1037e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.066+0000 7f86c0f77740 1 -- 172.21.15.98:0/2928996027 >> 172.21.15.98:0/2928996027 conn(0x564d1037ab30 msgr2=0x564d1037af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.066+0000 7f86c0f77740 1 -- 172.21.15.98:0/2928996027 shutdown_connections 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.066+0000 7f86c0f77740 1 -- 172.21.15.98:0/2928996027 wait complete. 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.066+0000 7f86c0f77740 1 -- start start 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86c0f77740 1 Processor -- start 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1037e4c0 0x564d1038b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d10388670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.073 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1038ba10 0x564d10388bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86c0f77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564d1038e640 con 0x564d1037e4c0 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86c0f77740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564d1027e0b0 con 0x564d1038ba10 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86c0f77740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564d1038e7c0 con 0x564d1037ede0 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86c0f77740 10 client.0 _ll_register_callbacks cb 0x564d1026bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86b77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1037e4c0 0x564d1038b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86b77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1037e4c0 0x564d1038b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52258/0 (socket says 172.21.15.98:52258) 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86b77fe640 1 -- 172.21.15.98:0/2910328712 learned_addr learned my addr 172.21.15.98:0/2910328712 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86b7fff640 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d10388670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86b77fe640 1 -- 172.21.15.98:0/2910328712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 msgr2=0x564d10388670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86b77fe640 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d10388670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86b77fe640 1 -- 172.21.15.98:0/2910328712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1038ba10 msgr2=0x564d10388bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86b77fe640 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1038ba10 0x564d10388bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.067+0000 7f86b77fe640 1 -- 172.21.15.98:0/2910328712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d100c2620 con 0x564d1037e4c0 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86b77fe640 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1037e4c0 0x564d1038b380 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0045b0 tx=0x7f86ac00bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbcd81d0f2d94dd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86977fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86977fe640 1 -- 172.21.15.98:0/2910328712 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac018040 con 0x564d1037e4c0 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86977fe640 1 -- 172.21.15.98:0/2910328712 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f86ac010d00 con 0x564d1037e4c0 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86c0f77740 1 -- 172.21.15.98:0/2910328712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86ac0106a0 con 0x564d1037e4c0 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86977fe640 1 -- 172.21.15.98:0/2910328712 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f86ac005040 con 0x564d1037e4c0 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86c0f77740 10 client.16611 fetch_fsmap learned FSMap version 47 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86c0f77740 1 -- 172.21.15.98:0/2910328712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x564d1039a5b0 con 0x564d1037e4c0 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86977fe640 1 -- 172.21.15.98:0/2910328712 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86ac0106a0 con 0x564d1037e4c0 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86977fe640 1 -- 172.21.15.98:0/2910328712 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f86ac0035f0 con 0x564d1037e4c0 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86c0f77740 10 client.16611 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T17:47:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86c0f77740 10 client.16611 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86c0f77740 20 client.16611 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86c0f77740 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564d1039b9c0 0x564d1039ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86c0f77740 4 client.16611 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.068+0000 7f86c0f77740 1 -- 172.21.15.98:0/2910328712 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d102f1390 con 0x564d1039b9c0 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.069+0000 7f86b7fff640 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564d1039b9c0 0x564d1039ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.069+0000 7f86b7fff640 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564d1039b9c0 0x564d1039ddb0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7f94216bae1a98d server_cookie=42a1eed1a0058801 in_seq=0 out_seq=0 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.069+0000 7f86977fe640 10 client.16611 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86977fe640 1 -- 172.21.15.98:0/2910328712 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d102f1390 con 0x564d1039b9c0 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86977fe640 10 client.16611 handle_command_reply: tid=0 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 client.16611 shutdown 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 -- 172.21.15.98:0/2910328712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1037e4c0 msgr2=0x564d1038b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1037e4c0 0x564d1038b380 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0045b0 tx=0x7f86ac00bcb0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 -- 172.21.15.98:0/2910328712 shutdown_connections 2024-04-25T17:47:57.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564d1039b9c0 0x564d1039ddb0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1038ba10 0x564d10388bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d10388670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 --2- 172.21.15.98:0/2910328712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1037e4c0 0x564d1038b380 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86977fe640 0 client.16611 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:57.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 -- 172.21.15.98:0/2910328712 >> 172.21.15.98:0/2910328712 conn(0x564d1037ab30 msgr2=0x564d103809a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:57.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 -- 172.21.15.98:0/2910328712 shutdown_connections 2024-04-25T17:47:57.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 1 -- 172.21.15.98:0/2910328712 wait complete. 2024-04-25T17:47:57.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.070+0000 7f86c0f77740 20 client.16611 trim_cache size 0 max 16384 2024-04-25T17:47:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86c0f77740 1 Processor -- start 2024-04-25T17:47:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86c0f77740 1 -- start start 2024-04-25T17:47:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 0x564d10397500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d1038aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d103a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86c0f77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86b81ae0b0 con 0x564d1038ba10 2024-04-25T17:47:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86c0f77740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86b81ab920 con 0x564d1037e4c0 2024-04-25T17:47:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86c0f77740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86b81a9790 con 0x564d1037ede0 2024-04-25T17:47:57.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86b77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d103a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86b77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d103a33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52260/0 (socket says 172.21.15.98:52260) 2024-04-25T17:47:57.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86b77fe640 1 -- 172.21.15.98:0/1171971066 learned_addr learned my addr 172.21.15.98:0/1171971066 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:57.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86b7fff640 1 --2- 172.21.15.98:0/1171971066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d1038aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86b77fe640 1 -- 172.21.15.98:0/1171971066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 msgr2=0x564d1038aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:57.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86b77fe640 1 --2- 172.21.15.98:0/1171971066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d1038aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86b77fe640 1 -- 172.21.15.98:0/1171971066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 msgr2=0x564d10397500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:57.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86b77fe640 1 --2- 172.21.15.98:0/1171971066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 0x564d10397500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86b77fe640 1 -- 172.21.15.98:0/1171971066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d10388f40 con 0x564d1038ba10 2024-04-25T17:47:57.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.085+0000 7f86b77fe640 1 --2- 172.21.15.98:0/1171971066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d103a33e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f86ac011010 tx=0x7f86ac003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14788fb59c90e3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f869dffb640 1 -- 172.21.15.98:0/1171971066 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac018030 con 0x564d1038ba10 2024-04-25T17:47:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f869dffb640 1 -- 172.21.15.98:0/1171971066 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86ac023030 con 0x564d1038ba10 2024-04-25T17:47:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f869dffb640 1 -- 172.21.15.98:0/1171971066 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0049c0 con 0x564d1038ba10 2024-04-25T17:47:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f86c0f77740 1 -- 172.21.15.98:0/1171971066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 msgr2=0x564d103a33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f86c0f77740 1 --2- 172.21.15.98:0/1171971066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d103a33e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f86ac011010 tx=0x7f86ac003f20 comp rx=0 tx=0).stop 2024-04-25T17:47:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f86c0f77740 1 -- 172.21.15.98:0/1171971066 shutdown_connections 2024-04-25T17:47:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f86c0f77740 1 --2- 172.21.15.98:0/1171971066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d103a33e0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f86c0f77740 1 --2- 172.21.15.98:0/1171971066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d1038aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f86c0f77740 1 --2- 172.21.15.98:0/1171971066 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 0x564d10397500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f86c0f77740 1 -- 172.21.15.98:0/1171971066 >> 172.21.15.98:0/1171971066 conn(0x564d1037ab30 msgr2=0x564d10383d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:57.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f86c0f77740 1 -- 172.21.15.98:0/1171971066 shutdown_connections 2024-04-25T17:47:57.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f86c0f77740 1 -- 172.21.15.98:0/1171971066 wait complete. 2024-04-25T17:47:57.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.086+0000 7f86c0f77740 1 -- start start 2024-04-25T17:47:57.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86c0f77740 1 Processor -- start 2024-04-25T17:47:57.091 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 0x564d10385760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d10385ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86c0f77740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d10400c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86c0f77740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86b81a9de0 con 0x564d1038ba10 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86c0f77740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86b81ab250 con 0x564d1037e4c0 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86c0f77740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86b81aa4b0 con 0x564d1037ede0 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86c0f77740 10 client.0 _ll_register_callbacks cb 0x7f86ac004470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86b77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d10400c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86b77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d10400c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52272/0 (socket says 172.21.15.98:52272) 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86b77fe640 1 -- 172.21.15.98:0/1420282520 learned_addr learned my addr 172.21.15.98:0/1420282520 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86b7fff640 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d10385ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86b77fe640 1 -- 172.21.15.98:0/1420282520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 msgr2=0x564d10385ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86b77fe640 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d10385ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86b77fe640 1 -- 172.21.15.98:0/1420282520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 msgr2=0x564d10385760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86b77fe640 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 0x564d10385760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.087+0000 7f86b77fe640 1 -- 172.21.15.98:0/1420282520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d10401190 con 0x564d1038ba10 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86b77fe640 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d10400c50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f86ac012e70 tx=0x7f86ac0049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95abeec03f0bf89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86977fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86977fe640 1 -- 172.21.15.98:0/1420282520 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac018070 con 0x564d1038ba10 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86977fe640 1 -- 172.21.15.98:0/1420282520 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8469+0+0 (secure 0 0 0) 0x7f86ac004660 con 0x564d1038ba10 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86c0f77740 1 -- 172.21.15.98:0/1420282520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86ac023030 con 0x564d1038ba10 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86977fe640 1 -- 172.21.15.98:0/1420282520 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f86ac036040 con 0x564d1038ba10 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86c0f77740 10 client.16617 fetch_fsmap learned FSMap version 47 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86c0f77740 1 -- 172.21.15.98:0/1420282520 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x564d104022e0 con 0x564d1038ba10 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86977fe640 1 -- 172.21.15.98:0/1420282520 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86ac023030 con 0x564d1038ba10 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86977fe640 1 -- 172.21.15.98:0/1420282520 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 47) ==== 8350+0+0 (secure 0 0 0) 0x7f86ac0051e0 con 0x564d1038ba10 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86c0f77740 10 client.16617 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86c0f77740 10 client.16617 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86c0f77740 20 client.16617 populate_metadata read hostname 'smithi098' 2024-04-25T17:47:57.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86c0f77740 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564d104028e0 0x564d10404d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:47:57.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86c0f77740 4 client.16617 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:47:57.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.088+0000 7f86c0f77740 1 -- 172.21.15.98:0/1420282520 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564d102eb200 con 0x564d104028e0 2024-04-25T17:47:57.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.089+0000 7f86b7fff640 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564d104028e0 0x564d10404d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:47:57.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.089+0000 7f86977fe640 10 client.16617 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:57.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.089+0000 7f86b7fff640 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564d104028e0 0x564d10404d00 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5003beff0bd30b62 server_cookie=43431be248ac00eb in_seq=0 out_seq=0 2024-04-25T17:47:57.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.099+0000 7f86977fe640 1 -- 172.21.15.98:0/1420282520 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564d102eb200 con 0x564d104028e0 2024-04-25T17:47:57.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.099+0000 7f86977fe640 10 client.16617 handle_command_reply: tid=0 2024-04-25T17:47:57.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.099+0000 7f86c0f77740 1 client.16617 shutdown 2024-04-25T17:47:57.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.099+0000 7f86c0f77740 1 -- 172.21.15.98:0/1420282520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 msgr2=0x564d10400c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.099+0000 7f86c0f77740 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d10400c50 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f86ac012e70 tx=0x7f86ac0049c0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86c0f77740 1 -- 172.21.15.98:0/1420282520 shutdown_connections 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86c0f77740 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564d104028e0 0x564d10404d00 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86c0f77740 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564d1038ba10 0x564d10400c50 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86977fe640 0 client.16617 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86c0f77740 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564d1037ede0 0x564d10385ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86c0f77740 1 --2- 172.21.15.98:0/1420282520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564d1037e4c0 0x564d10385760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86c0f77740 1 -- 172.21.15.98:0/1420282520 >> 172.21.15.98:0/1420282520 conn(0x564d1037ab30 msgr2=0x564d10379420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86c0f77740 1 -- 172.21.15.98:0/1420282520 shutdown_connections 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86c0f77740 1 -- 172.21.15.98:0/1420282520 wait complete. 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86c0f77740 20 client.16617 trim_cache size 0 max 16384 2024-04-25T17:47:57.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:47:57.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:47:57.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:47:57.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:47:57.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 -- 172.21.15.98:0/1474691537 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f86a00a3dc0 msgr2=0x7f86a00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:57.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f86a00a3dc0 0x7f86a00a6280 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f86b00ba4b0 tx=0x7f86b008ca00 comp rx=0 tx=0).stop 2024-04-25T17:47:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 -- 172.21.15.98:0/1474691537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b819be20 msgr2=0x7f86b819c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:47:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b819be20 0x7f86b819c2d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f86b0005cc0 tx=0x7f86b002f470 comp rx=0 tx=0).stop 2024-04-25T17:47:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 -- 172.21.15.98:0/1474691537 shutdown_connections 2024-04-25T17:47:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f86a00a3dc0 0x7f86a00a6280 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b819be20 0x7f86b819c2d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b81061b0 0x7f86b8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 --2- 172.21.15.98:0/1474691537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b81010e0 0x7f86b8197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:47:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 -- 172.21.15.98:0/1474691537 >> 172.21.15.98:0/1474691537 conn(0x7f86b80771d0 msgr2=0x7f86b80feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:47:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 -- 172.21.15.98:0/1474691537 shutdown_connections 2024-04-25T17:47:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:47:57.100+0000 7f86be269640 1 -- 172.21.15.98:0/1474691537 wait complete. 2024-04-25T17:47:58.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:47:58 smithi173 ceph-mon[32856]: pgmap v378: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 30 MiB/s wr, 1.69k op/s 2024-04-25T17:47:58.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:47:58 smithi098 ceph-mon[29192]: pgmap v378: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 30 MiB/s wr, 1.69k op/s 2024-04-25T17:47:58.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:47:58 smithi203 ceph-mon[32834]: pgmap v378: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 30 MiB/s wr, 1.69k op/s 2024-04-25T17:48:00.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:00 smithi173 ceph-mon[32856]: pgmap v379: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 26 MiB/s wr, 1.37k op/s 2024-04-25T17:48:00.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:00 smithi098 ceph-mon[29192]: pgmap v379: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 26 MiB/s wr, 1.37k op/s 2024-04-25T17:48:00.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:00 smithi203 ceph-mon[32834]: pgmap v379: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 26 MiB/s wr, 1.37k op/s 2024-04-25T17:48:01.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:01 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:48:01.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:01 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:48:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 690.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:48:01.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:01 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:48:02.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:02 smithi173 ceph-mon[32856]: pgmap v380: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 26 MiB/s wr, 1.37k op/s 2024-04-25T17:48:02.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:02 smithi098 ceph-mon[29192]: pgmap v380: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 26 MiB/s wr, 1.37k op/s 2024-04-25T17:48:02.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:02 smithi203 ceph-mon[32834]: pgmap v380: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 26 MiB/s wr, 1.37k op/s 2024-04-25T17:48:04.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:04 smithi098 ceph-mon[29192]: pgmap v381: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 26 MiB/s wr, 1.39k op/s 2024-04-25T17:48:04.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:04 smithi203 ceph-mon[32834]: pgmap v381: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 26 MiB/s wr, 1.39k op/s 2024-04-25T17:48:04.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:04 smithi173 ceph-mon[32856]: pgmap v381: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 26 MiB/s wr, 1.39k op/s 2024-04-25T17:48:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 695.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:48:06.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:06 smithi098 ceph-mon[29192]: pgmap v382: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-04-25T17:48:06.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:06 smithi098 ceph-mon[29192]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T17:48:06.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:06 smithi203 ceph-mon[32834]: pgmap v382: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-04-25T17:48:06.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:06 smithi203 ceph-mon[32834]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T17:48:06.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:06 smithi173 ceph-mon[32856]: pgmap v382: 97 pgs: 97 active+clean; 4.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 1.9 MiB/s wr, 158 op/s 2024-04-25T17:48:06.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:06 smithi173 ceph-mon[32856]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T17:48:08.473 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:48:08.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:08 smithi098 ceph-mon[29192]: pgmap v384: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 83 KiB/s wr, 34 op/s 2024-04-25T17:48:08.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:08 smithi203 ceph-mon[32834]: pgmap v384: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 83 KiB/s wr, 34 op/s 2024-04-25T17:48:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:48:08.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:08 smithi173 ceph-mon[32856]: pgmap v384: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 83 KiB/s wr, 34 op/s 2024-04-25T17:48:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.193+0000 7f10fd2ae640 1 -- 172.21.15.98:0/2047873871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10f8102b70 msgr2=0x7f10f8106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.193+0000 7f10fd2ae640 1 --2- 172.21.15.98:0/2047873871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10f8102b70 0x7f10f8106bb0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009900 tx=0x7f10e002a410 comp rx=0 tx=0).stop 2024-04-25T17:48:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.193+0000 7f10fd2ae640 1 -- 172.21.15.98:0/2047873871 shutdown_connections 2024-04-25T17:48:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.193+0000 7f10fd2ae640 1 --2- 172.21.15.98:0/2047873871 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10f8107290 0x7f10f810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.193+0000 7f10fd2ae640 1 --2- 172.21.15.98:0/2047873871 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10f8102b70 0x7f10f8106bb0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.193+0000 7f10fd2ae640 1 --2- 172.21.15.98:0/2047873871 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f81021c0 0x7f10f81025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.193+0000 7f10fd2ae640 1 -- 172.21.15.98:0/2047873871 >> 172.21.15.98:0/2047873871 conn(0x7f10f80fda20 msgr2=0x7f10f80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.193+0000 7f10fd2ae640 1 -- 172.21.15.98:0/2047873871 shutdown_connections 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.193+0000 7f10fd2ae640 1 -- 172.21.15.98:0/2047873871 wait complete. 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10fd2ae640 1 Processor -- start 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10fd2ae640 1 -- start start 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10fd2ae640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10f81021c0 0x7f10f81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10fd2ae640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f8107290 0x7f10f8198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10fd2ae640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10f819cf60 0x7f10f819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10fd2ae640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81121b0 con 0x7f10f81021c0 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10fd2ae640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10f8112030 con 0x7f10f8107290 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10fd2ae640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f10f8112330 con 0x7f10f819cf60 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10f81021c0 0x7f10f81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10f81021c0 0x7f10f81986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59436/0 (socket says 172.21.15.98:59436) 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f6ffd640 1 -- 172.21.15.98:0/3965163009 learned_addr learned my addr 172.21.15.98:0/3965163009 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f77fe640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10f819cf60 0x7f10f819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10effff640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f8107290 0x7f10f8198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f6ffd640 1 -- 172.21.15.98:0/3965163009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10f819cf60 msgr2=0x7f10f819d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10f819cf60 0x7f10f819d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f6ffd640 1 -- 172.21.15.98:0/3965163009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f8107290 msgr2=0x7f10f8198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f8107290 0x7f10f8198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f6ffd640 1 -- 172.21.15.98:0/3965163009 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f819da50 con 0x7f10f81021c0 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f77fe640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10f819cf60 0x7f10f819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.194+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10f81021c0 0x7f10f81986f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f10e800c920 tx=0x7f10e800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.195+0000 7f10f4ff9640 1 -- 172.21.15.98:0/3965163009 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e8007c70 con 0x7f10f81021c0 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.195+0000 7f10fd2ae640 1 -- 172.21.15.98:0/3965163009 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f819dd40 con 0x7f10f81021c0 2024-04-25T17:48:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.195+0000 7f10f4ff9640 1 -- 172.21.15.98:0/3965163009 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10e8007e10 con 0x7f10f81021c0 2024-04-25T17:48:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.195+0000 7f10f4ff9640 1 -- 172.21.15.98:0/3965163009 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e8018820 con 0x7f10f81021c0 2024-04-25T17:48:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.195+0000 7f10fd2ae640 1 -- 172.21.15.98:0/3965163009 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f81a5500 con 0x7f10f81021c0 2024-04-25T17:48:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.197+0000 7f10f4ff9640 1 -- 172.21.15.98:0/3965163009 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f10e800f450 con 0x7f10f81021c0 2024-04-25T17:48:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.197+0000 7f10fffc9740 1 Processor -- start 2024-04-25T17:48:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.197+0000 7f10fffc9740 1 -- start start 2024-04-25T17:48:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f4ff9640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f10e40a3dc0 0x7f10e40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4620 0x564ff3ba4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4f40 0x564ff3baf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3bb1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10fffc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564ff3b09970 con 0x564ff3baf800 2024-04-25T17:48:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10fffc9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564ff3aa64a0 con 0x564ff3ba4620 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10fffc9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564ff3b05930 con 0x564ff3ba4f40 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f77fe640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f10e40a3dc0 0x7f10e40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10effff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4f40 0x564ff3baf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3bb1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f4ff9640 1 -- 172.21.15.98:0/3965163009 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f10e80d0a50 con 0x7f10f81021c0 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3bb1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59448/0 (socket says 172.21.15.98:59448) 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10effff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4f40 0x564ff3baf2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:55958/0 (socket says 172.21.15.98:55958) 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f77fe640 1 -- 172.21.15.98:0/177271903 learned_addr learned my addr 172.21.15.98:0/177271903 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/177271903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4620 0x564ff3ba4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f77fe640 1 -- 172.21.15.98:0/177271903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4f40 msgr2=0x564ff3baf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f77fe640 1 --2- 172.21.15.98:0/177271903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4f40 0x564ff3baf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f77fe640 1 -- 172.21.15.98:0/177271903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4620 msgr2=0x564ff3ba4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f77fe640 1 --2- 172.21.15.98:0/177271903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4620 0x564ff3ba4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10f77fe640 1 -- 172.21.15.98:0/177271903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ff381c210 con 0x564ff3baf800 2024-04-25T17:48:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.198+0000 7f10effff640 1 --2- 172.21.15.98:0/177271903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4f40 0x564ff3baf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.199+0000 7f10f77fe640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f10e40a3dc0 0x7f10e40a6280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f10f0012be0 tx=0x7f10f0009db0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.199+0000 7f10f77fe640 1 --2- 172.21.15.98:0/177271903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3bb1bf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f10f0005b30 tx=0x7f10f001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377dc0be63fbeec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.199+0000 7f10ed7fa640 1 -- 172.21.15.98:0/177271903 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f000c6e0 con 0x564ff3baf800 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.199+0000 7f10ed7fa640 1 -- 172.21.15.98:0/177271903 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f0025070 con 0x564ff3baf800 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.199+0000 7f10ed7fa640 1 -- 172.21.15.98:0/177271903 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0020b80 con 0x564ff3baf800 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.199+0000 7f10fffc9740 1 -- 172.21.15.98:0/177271903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 msgr2=0x564ff3bb1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.199+0000 7f10fffc9740 1 --2- 172.21.15.98:0/177271903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3bb1bf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f10f0005b30 tx=0x7f10f001c000 comp rx=0 tx=0).stop 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 -- 172.21.15.98:0/177271903 shutdown_connections 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 --2- 172.21.15.98:0/177271903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3bb1bf0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 --2- 172.21.15.98:0/177271903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4f40 0x564ff3baf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 --2- 172.21.15.98:0/177271903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4620 0x564ff3ba4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 -- 172.21.15.98:0/177271903 >> 172.21.15.98:0/177271903 conn(0x564ff3ba0b30 msgr2=0x564ff3ba0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 -- 172.21.15.98:0/177271903 shutdown_connections 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 -- 172.21.15.98:0/177271903 wait complete. 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 -- start start 2024-04-25T17:48:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 Processor -- start 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 0x564ff3baf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4f40 0x564ff3bac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3baca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564ff3bb23d0 con 0x564ff3baf800 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564ff3aa40b0 con 0x564ff3ba4f40 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564ff3bb2550 con 0x564ff3ba4620 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10fffc9740 10 client.0 _ll_register_callbacks cb 0x564ff3a91d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10f77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3baca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10f77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3baca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59464/0 (socket says 172.21.15.98:59464) 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10f77fe640 1 -- 172.21.15.98:0/1870954407 learned_addr learned my addr 172.21.15.98:0/1870954407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.200+0000 7f10effff640 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4f40 0x564ff3bac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 0x564ff3baf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10f77fe640 1 -- 172.21.15.98:0/1870954407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 msgr2=0x564ff3baf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10f77fe640 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 0x564ff3baf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10f77fe640 1 -- 172.21.15.98:0/1870954407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4f40 msgr2=0x564ff3bac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10f77fe640 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4f40 0x564ff3bac4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10f77fe640 1 -- 172.21.15.98:0/1870954407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ff38e8620 con 0x564ff3baf800 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 0x564ff3baf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10f77fe640 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3baca40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009a40 tx=0x7f10f0020700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f305237c60a8c2e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10de7fc640 1 -- 172.21.15.98:0/1870954407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f002f070 con 0x564ff3baf800 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10de7fc640 1 -- 172.21.15.98:0/1870954407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f10f0020d30 con 0x564ff3baf800 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10fffc9740 1 -- 172.21.15.98:0/1870954407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10f0025070 con 0x564ff3baf800 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10de7fc640 1 -- 172.21.15.98:0/1870954407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f10f003b050 con 0x564ff3baf800 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10fffc9740 10 client.16629 fetch_fsmap learned FSMap version 48 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10fffc9740 1 -- 172.21.15.98:0/1870954407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x564ff3bbe380 con 0x564ff3baf800 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10de7fc640 1 -- 172.21.15.98:0/1870954407 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f0025070 con 0x564ff3baf800 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10de7fc640 1 -- 172.21.15.98:0/1870954407 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 48) ==== 8346+0+0 (secure 0 0 0) 0x7f10f001f070 con 0x564ff3baf800 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10fffc9740 10 client.16629 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10fffc9740 10 client.16629 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.201+0000 7f10fffc9740 20 client.16629 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.202+0000 7f10fffc9740 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564ff3bbf790 0x564ff3bc1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.202+0000 7f10fffc9740 4 client.16629 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:48:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.202+0000 7f10fffc9740 1 -- 172.21.15.98:0/1870954407 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ff3b17390 con 0x564ff3bbf790 2024-04-25T17:48:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.202+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564ff3bbf790 0x564ff3bc1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.202+0000 7f10de7fc640 10 client.16629 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.202+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564ff3bbf790 0x564ff3bc1b80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6cbf231b3e7a97 server_cookie=bd6eadefe502e6e0 in_seq=0 out_seq=0 2024-04-25T17:48:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10de7fc640 1 -- 172.21.15.98:0/1870954407 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ff3b17390 con 0x564ff3bbf790 2024-04-25T17:48:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10de7fc640 10 client.16629 handle_command_reply: tid=0 2024-04-25T17:48:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 client.16629 shutdown 2024-04-25T17:48:10.210 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 -- 172.21.15.98:0/1870954407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 msgr2=0x564ff3baca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3baca40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009a40 tx=0x7f10f0020700 comp rx=0 tx=0).stop 2024-04-25T17:48:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 -- 172.21.15.98:0/1870954407 shutdown_connections 2024-04-25T17:48:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564ff3bbf790 0x564ff3bc1b80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3baf800 0x564ff3baca40 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4f40 0x564ff3bac4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10de7fc640 0 client.16629 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 --2- 172.21.15.98:0/1870954407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 0x564ff3baf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.211 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 -- 172.21.15.98:0/1870954407 >> 172.21.15.98:0/1870954407 conn(0x564ff3ba0b30 msgr2=0x564ff3ba68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:10.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 -- 172.21.15.98:0/1870954407 shutdown_connections 2024-04-25T17:48:10.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 1 -- 172.21.15.98:0/1870954407 wait complete. 2024-04-25T17:48:10.212 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.203+0000 7f10fffc9740 20 client.16629 trim_cache size 0 max 16384 2024-04-25T17:48:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10fffc9740 1 Processor -- start 2024-04-25T17:48:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10fffc9740 1 -- start start 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 0x564ff3bbb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3babdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3baf800 0x564ff3b9f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10fffc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81b0490 con 0x564ff3ba4f40 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10fffc9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81add00 con 0x564ff3baf800 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10fffc9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81abb70 con 0x564ff3ba4620 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10effff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3babdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10effff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3babdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59466/0 (socket says 172.21.15.98:59466) 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10effff640 1 -- 172.21.15.98:0/4039574946 learned_addr learned my addr 172.21.15.98:0/4039574946 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/4039574946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 0x564ff3bbb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10f77fe640 1 --2- 172.21.15.98:0/4039574946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3baf800 0x564ff3b9f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10effff640 1 -- 172.21.15.98:0/4039574946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 msgr2=0x564ff3bbb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10effff640 1 --2- 172.21.15.98:0/4039574946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 0x564ff3bbb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10effff640 1 -- 172.21.15.98:0/4039574946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3baf800 msgr2=0x564ff3b9f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10effff640 1 --2- 172.21.15.98:0/4039574946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3baf800 0x564ff3b9f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.218+0000 7f10effff640 1 -- 172.21.15.98:0/4039574946 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ff3bb1bf0 con 0x564ff3ba4f40 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.219+0000 7f10effff640 1 --2- 172.21.15.98:0/4039574946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3babdc0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f10f806b5f0 tx=0x7f10e0032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f47cc7a089f48a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.219+0000 7f10ecff9640 1 -- 172.21.15.98:0/4039574946 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0035040 con 0x564ff3ba4f40 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.219+0000 7f10ecff9640 1 -- 172.21.15.98:0/4039574946 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10e0031880 con 0x564ff3ba4f40 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.219+0000 7f10ecff9640 1 -- 172.21.15.98:0/4039574946 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0004920 con 0x564ff3ba4f40 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.219+0000 7f10fffc9740 1 -- 172.21.15.98:0/4039574946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 msgr2=0x564ff3babdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.219+0000 7f10fffc9740 1 --2- 172.21.15.98:0/4039574946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3babdc0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f10f806b5f0 tx=0x7f10e0032f60 comp rx=0 tx=0).stop 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.219+0000 7f10fffc9740 1 -- 172.21.15.98:0/4039574946 shutdown_connections 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.219+0000 7f10fffc9740 1 --2- 172.21.15.98:0/4039574946 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3baf800 0x564ff3b9f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.219+0000 7f10fffc9740 1 --2- 172.21.15.98:0/4039574946 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3babdc0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.219+0000 7f10fffc9740 1 --2- 172.21.15.98:0/4039574946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3ba4620 0x564ff3bbb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 -- 172.21.15.98:0/4039574946 >> 172.21.15.98:0/4039574946 conn(0x564ff3ba0b30 msgr2=0x564ff3bbab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 -- 172.21.15.98:0/4039574946 shutdown_connections 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 -- 172.21.15.98:0/4039574946 wait complete. 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 -- start start 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 Processor -- start 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4620 0x564ff3bb5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3bb5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3baf800 0x564ff3bb5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81ac1c0 con 0x564ff3ba4f40 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81ad630 con 0x564ff3ba4620 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f10f81ac890 con 0x564ff3baf800 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10fffc9740 10 client.0 _ll_register_callbacks cb 0x7f10f00108f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10effff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3bb5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10effff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3bb5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59474/0 (socket says 172.21.15.98:59474) 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10effff640 1 -- 172.21.15.98:0/2821958216 learned_addr learned my addr 172.21.15.98:0/2821958216 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10f77fe640 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3baf800 0x564ff3bb5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4620 0x564ff3bb5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10effff640 1 -- 172.21.15.98:0/2821958216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3baf800 msgr2=0x564ff3bb5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10effff640 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3baf800 0x564ff3bb5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10effff640 1 -- 172.21.15.98:0/2821958216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4620 msgr2=0x564ff3bb5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10effff640 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4620 0x564ff3bb5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.220+0000 7f10effff640 1 -- 172.21.15.98:0/2821958216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ff3bc20e0 con 0x564ff3ba4f40 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10effff640 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3bb5790 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f10e00057c0 tx=0x7f10e00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17809de41bf9a9aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10de7fc640 1 -- 172.21.15.98:0/2821958216 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0035030 con 0x564ff3ba4f40 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10de7fc640 1 -- 172.21.15.98:0/2821958216 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f10e0003ca0 con 0x564ff3ba4f40 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10fffc9740 1 -- 172.21.15.98:0/2821958216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10e0031880 con 0x564ff3ba4f40 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10de7fc640 1 -- 172.21.15.98:0/2821958216 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f10e0055050 con 0x564ff3ba4f40 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10fffc9740 10 client.16635 fetch_fsmap learned FSMap version 48 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10fffc9740 1 -- 172.21.15.98:0/2821958216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x564ff3c25d30 con 0x564ff3ba4f40 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10de7fc640 1 -- 172.21.15.98:0/2821958216 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10e0031880 con 0x564ff3ba4f40 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10de7fc640 1 -- 172.21.15.98:0/2821958216 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 48) ==== 8346+0+0 (secure 0 0 0) 0x7f10e003b430 con 0x564ff3ba4f40 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10fffc9740 10 client.16635 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.221+0000 7f10fffc9740 10 client.16635 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.222+0000 7f10fffc9740 20 client.16635 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.222+0000 7f10fffc9740 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564ff3c26480 0x564ff3c288a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.222+0000 7f10fffc9740 4 client.16635 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:48:10.227 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.222+0000 7f10fffc9740 1 -- 172.21.15.98:0/2821958216 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564ff3b11200 con 0x564ff3c26480 2024-04-25T17:48:10.227 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.222+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564ff3c26480 0x564ff3c288a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.227 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.222+0000 7f10f6ffd640 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564ff3c26480 0x564ff3c288a0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1a210b8a76af869 server_cookie=d8403a4fe06b396f in_seq=0 out_seq=0 2024-04-25T17:48:10.227 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.222+0000 7f10de7fc640 10 client.16635 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.230+0000 7f10de7fc640 1 -- 172.21.15.98:0/2821958216 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564ff3b11200 con 0x564ff3c26480 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.230+0000 7f10de7fc640 10 client.16635 handle_command_reply: tid=0 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 client.16635 shutdown 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 -- 172.21.15.98:0/2821958216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 msgr2=0x564ff3bb5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3bb5790 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f10e00057c0 tx=0x7f10e00038a0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 -- 172.21.15.98:0/2821958216 shutdown_connections 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564ff3c26480 0x564ff3c288a0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564ff3baf800 0x564ff3bb5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564ff3ba4f40 0x564ff3bb5790 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 --2- 172.21.15.98:0/2821958216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564ff3ba4620 0x564ff3bb5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.231 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10de7fc640 0 client.16635 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 -- 172.21.15.98:0/2821958216 >> 172.21.15.98:0/2821958216 conn(0x564ff3ba0b30 msgr2=0x564ff3bba280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 -- 172.21.15.98:0/2821958216 shutdown_connections 2024-04-25T17:48:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 1 -- 172.21.15.98:0/2821958216 wait complete. 2024-04-25T17:48:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fffc9740 20 client.16635 trim_cache size 0 max 16384 2024-04-25T17:48:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:48:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:48:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:48:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:48:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fd2ae640 1 -- 172.21.15.98:0/3965163009 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f10e40a3dc0 msgr2=0x7f10e40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.231+0000 7f10fd2ae640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f10e40a3dc0 0x7f10e40a6280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f10f0012be0 tx=0x7f10f0009db0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.232+0000 7f10fd2ae640 1 -- 172.21.15.98:0/3965163009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10f81021c0 msgr2=0x7f10f81986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.232+0000 7f10fd2ae640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10f81021c0 0x7f10f81986f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f10e800c920 tx=0x7f10e800cde0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.232+0000 7f10fd2ae640 1 -- 172.21.15.98:0/3965163009 shutdown_connections 2024-04-25T17:48:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.232+0000 7f10fd2ae640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f10e40a3dc0 0x7f10e40a6280 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.232+0000 7f10fd2ae640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f10f819cf60 0x7f10f819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.232+0000 7f10fd2ae640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10f8107290 0x7f10f8198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.232+0000 7f10fd2ae640 1 --2- 172.21.15.98:0/3965163009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10f81021c0 0x7f10f81986f0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.232+0000 7f10fd2ae640 1 -- 172.21.15.98:0/3965163009 >> 172.21.15.98:0/3965163009 conn(0x7f10f80fda20 msgr2=0x7f10f80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.232+0000 7f10fd2ae640 1 -- 172.21.15.98:0/3965163009 shutdown_connections 2024-04-25T17:48:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:10.232+0000 7f10fd2ae640 1 -- 172.21.15.98:0/3965163009 wait complete. 2024-04-25T17:48:10.442 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:10 smithi098 ceph-mon[29192]: pgmap v385: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 83 KiB/s wr, 34 op/s 2024-04-25T17:48:10.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.544+0000 7f55a9c00700 1 -- 172.21.15.98:0/2031542031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a4102300 msgr2=0x7f55a410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.544+0000 7f55a9c00700 1 --2- 172.21.15.98:0/2031542031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a4102300 0x7f55a410e7f0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f559c00b0a0 tx=0x7f559c00b3b0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.544+0000 7f55a9c00700 1 -- 172.21.15.98:0/2031542031 shutdown_connections 2024-04-25T17:48:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.544+0000 7f55a9c00700 1 --2- 172.21.15.98:0/2031542031 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a4102300 0x7f55a410e7f0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.544+0000 7f55a9c00700 1 --2- 172.21.15.98:0/2031542031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a4107990 0x7f55a4107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.544+0000 7f55a9c00700 1 --2- 172.21.15.98:0/2031542031 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4101940 0x7f55a4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.544+0000 7f55a9c00700 1 -- 172.21.15.98:0/2031542031 >> 172.21.15.98:0/2031542031 conn(0x7f55a40788f0 msgr2=0x7f55a4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.544+0000 7f55a9c00700 1 -- 172.21.15.98:0/2031542031 shutdown_connections 2024-04-25T17:48:10.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.544+0000 7f55a9c00700 1 -- 172.21.15.98:0/2031542031 wait complete. 2024-04-25T17:48:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a9c00700 1 Processor -- start 2024-04-25T17:48:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a9c00700 1 -- start start 2024-04-25T17:48:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a9c00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a4101940 0x7f55a41034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a9c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4102300 0x7f55a4103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a9c00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a4107990 0x7f55a4103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a9c00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55a4111af0 con 0x7f55a4107990 2024-04-25T17:48:10.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a9c00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f55a4111970 con 0x7f55a4101940 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a9c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55a4111c70 con 0x7f55a4102300 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a4107990 0x7f55a4103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a4107990 0x7f55a4103f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59494/0 (socket says 172.21.15.98:59494) 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a3fff700 1 -- 172.21.15.98:0/3478260405 learned_addr learned my addr 172.21.15.98:0/3478260405 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a3fff700 1 -- 172.21.15.98:0/3478260405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4102300 msgr2=0x7f55a4103a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a3fff700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4102300 0x7f55a4103a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a3fff700 1 -- 172.21.15.98:0/3478260405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a4101940 msgr2=0x7f55a41034f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a3fff700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a4101940 0x7f55a41034f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.545+0000 7f55a3fff700 1 -- 172.21.15.98:0/3478260405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a41a74b0 con 0x7f55a4107990 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.546+0000 7f55a3fff700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a4107990 0x7f55a4103f70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f559c00b830 tx=0x7f559c004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.546+0000 7f55a0ff9700 1 -- 172.21.15.98:0/3478260405 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c021070 con 0x7f55a4107990 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.546+0000 7f55a0ff9700 1 -- 172.21.15.98:0/3478260405 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f559c003c00 con 0x7f55a4107990 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.546+0000 7f55a9c00700 1 -- 172.21.15.98:0/3478260405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a41a7740 con 0x7f55a4107990 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.546+0000 7f55a0ff9700 1 -- 172.21.15.98:0/3478260405 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c004730 con 0x7f55a4107990 2024-04-25T17:48:10.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.546+0000 7f55a9c00700 1 -- 172.21.15.98:0/3478260405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a41a7bf0 con 0x7f55a4107990 2024-04-25T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.547+0000 7f55a9c00700 1 -- 172.21.15.98:0/3478260405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a4069730 con 0x7f55a4107990 2024-04-25T17:48:10.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.548+0000 7f55a0ff9700 1 -- 172.21.15.98:0/3478260405 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f559c007a00 con 0x7f55a4107990 2024-04-25T17:48:10.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.548+0000 7f55a0ff9700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f558c0a3db0 0x7f558c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:10.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.549+0000 7f55a37fe700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f558c0a3db0 0x7f558c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:10.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.549+0000 7f55a0ff9700 1 -- 172.21.15.98:0/3478260405 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f559c00f020 con 0x7f55a4107990 2024-04-25T17:48:10.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.549+0000 7f55a37fe700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f558c0a3db0 0x7f558c0a6270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f55980059d0 tx=0x7f5598005c20 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:10.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.552+0000 7f55a0ff9700 1 -- 172.21.15.98:0/3478260405 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f559c02a020 con 0x7f55a4107990 2024-04-25T17:48:10.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:10 smithi203 ceph-mon[32834]: pgmap v385: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 83 KiB/s wr, 34 op/s 2024-04-25T17:48:10.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:10 smithi173 ceph-mon[32856]: pgmap v385: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 83 KiB/s wr, 34 op/s 2024-04-25T17:48:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.796+0000 7f55a9c00700 1 -- 172.21.15.98:0/3478260405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55a402e140 con 0x7f55a4107990 2024-04-25T17:48:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.797+0000 7f55a0ff9700 1 -- 172.21.15.98:0/3478260405 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11880 (secure 0 0 0) 0x7f559c02e3e0 con 0x7f55a4107990 2024-04-25T17:48:10.796 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:48:10.797 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:47:59.912394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26026":{"gid":26026,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/3925531495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":3925531495},{"type":"v1","addr":"172.21.15.203:6837","nonce":3925531495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:48:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 -- 172.21.15.98:0/3478260405 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f558c0a3db0 msgr2=0x7f558c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f558c0a3db0 0x7f558c0a6270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f55980059d0 tx=0x7f5598005c20 comp rx=0 tx=0).stop 2024-04-25T17:48:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 -- 172.21.15.98:0/3478260405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a4107990 msgr2=0x7f55a4103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a4107990 0x7f55a4103f70 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f559c00b830 tx=0x7f559c004d90 comp rx=0 tx=0).stop 2024-04-25T17:48:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 -- 172.21.15.98:0/3478260405 shutdown_connections 2024-04-25T17:48:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f558c0a3db0 0x7f558c0a6270 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a4107990 0x7f55a4103f70 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a4101940 0x7f55a41034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 --2- 172.21.15.98:0/3478260405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a4102300 0x7f55a4103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 -- 172.21.15.98:0/3478260405 >> 172.21.15.98:0/3478260405 conn(0x7f55a40788f0 msgr2=0x7f55a406a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.799+0000 7f55a9c00700 1 -- 172.21.15.98:0/3478260405 shutdown_connections 2024-04-25T17:48:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:10.800+0000 7f55a9c00700 1 -- 172.21.15.98:0/3478260405 wait complete. 2024-04-25T17:48:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 48 2024-04-25T17:48:11.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:11 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3478260405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:48:11.535 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id 964d443e --release --await' 2024-04-25T17:48:11.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id 964d443e --release --await 2024-04-25T17:48:11.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 700.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:48:11.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:11 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3478260405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:48:11.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:11 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3478260405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:48:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:48:12.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:12 smithi098 ceph-mon[29192]: pgmap v386: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 83 KiB/s wr, 34 op/s 2024-04-25T17:48:12.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:12 smithi203 ceph-mon[32834]: pgmap v386: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 83 KiB/s wr, 34 op/s 2024-04-25T17:48:12.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:12 smithi173 ceph-mon[32856]: pgmap v386: 97 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 90 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 83 KiB/s wr, 34 op/s 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.446+0000 7f543fb5c700 1 -- 172.21.15.98:0/2542573807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5438102ac0 msgr2=0x7f5438102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.446+0000 7f543fb5c700 1 --2- 172.21.15.98:0/2542573807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5438102ac0 0x7f5438102f20 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5428009990 tx=0x7f5428009ca0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.446+0000 7f543fb5c700 1 -- 172.21.15.98:0/2542573807 shutdown_connections 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.446+0000 7f543fb5c700 1 --2- 172.21.15.98:0/2542573807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5438108a70 0x7f5438108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.446+0000 7f543fb5c700 1 --2- 172.21.15.98:0/2542573807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5438103460 0x7f543810f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.446+0000 7f543fb5c700 1 --2- 172.21.15.98:0/2542573807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5438102ac0 0x7f5438102f20 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.446+0000 7f543fb5c700 1 -- 172.21.15.98:0/2542573807 >> 172.21.15.98:0/2542573807 conn(0x7f5438079b10 msgr2=0x7f5438079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.446+0000 7f543fb5c700 1 -- 172.21.15.98:0/2542573807 shutdown_connections 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.446+0000 7f543fb5c700 1 -- 172.21.15.98:0/2542573807 wait complete. 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543fb5c700 1 Processor -- start 2024-04-25T17:48:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543fb5c700 1 -- start start 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543fb5c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5438102ac0 0x7f5438074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543fb5c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5438103460 0x7f5438075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543fb5c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5438108a70 0x7f54380757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543fb5c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5438112c10 con 0x7f5438108a70 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543fb5c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5438112a90 con 0x7f5438102ac0 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543fb5c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5438112d90 con 0x7f5438103460 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543e0f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5438108a70 0x7f54380757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543e0f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5438108a70 0x7f54380757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59514/0 (socket says 172.21.15.98:59514) 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.447+0000 7f543e0f9700 1 -- 172.21.15.98:0/3070534711 learned_addr learned my addr 172.21.15.98:0/3070534711 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543d0f7700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5438103460 0x7f5438075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543d8f8700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5438102ac0 0x7f5438074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543e0f9700 1 -- 172.21.15.98:0/3070534711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5438103460 msgr2=0x7f5438075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543e0f9700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5438103460 0x7f5438075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543e0f9700 1 -- 172.21.15.98:0/3070534711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5438102ac0 msgr2=0x7f5438074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543e0f9700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5438102ac0 0x7f5438074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543e0f9700 1 -- 172.21.15.98:0/3070534711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54381a7600 con 0x7f5438108a70 2024-04-25T17:48:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543d8f8700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5438102ac0 0x7f5438074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:48:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543e0f9700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5438108a70 0x7f54380757c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5430007d70 tx=0x7f543000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f542effd700 1 -- 172.21.15.98:0/3070534711 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5430016020 con 0x7f5438108a70 2024-04-25T17:48:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543fb5c700 1 -- 172.21.15.98:0/3070534711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54381a78f0 con 0x7f5438108a70 2024-04-25T17:48:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f542effd700 1 -- 172.21.15.98:0/3070534711 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f543000ad40 con 0x7f5438108a70 2024-04-25T17:48:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f542effd700 1 -- 172.21.15.98:0/3070534711 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54300115e0 con 0x7f5438108a70 2024-04-25T17:48:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.448+0000 7f543fb5c700 1 -- 172.21.15.98:0/3070534711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54381a7e80 con 0x7f5438108a70 2024-04-25T17:48:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.450+0000 7f542effd700 1 -- 172.21.15.98:0/3070534711 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5430019020 con 0x7f5438108a70 2024-04-25T17:48:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.451+0000 7f542effd700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f54240a3db0 0x7f54240a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.451+0000 7f543d8f8700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f54240a3db0 0x7f54240a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.451+0000 7f542effd700 1 -- 172.21.15.98:0/3070534711 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f54300d0730 con 0x7f5438108a70 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.451+0000 7f543d8f8700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f54240a3db0 0x7f54240a6270 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5434005fd0 tx=0x7f5434005d40 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f544a3e5b80 1 Processor -- start 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f544a3e5b80 1 -- start start 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587453629f0 0x558745362dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558745363310 0x55874536dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55874536e1e0 0x5587453705d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5587452b5e30 con 0x5587453629f0 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55874525ca60 con 0x55874536e1e0 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5587452b1df0 con 0x558745363310 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543d8f8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587453629f0 0x558745362dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543d8f8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587453629f0 0x558745362dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59518/0 (socket says 172.21.15.98:59518) 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543d8f8700 1 -- 172.21.15.98:0/2704222544 learned_addr learned my addr 172.21.15.98:0/2704222544 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543e0f9700 1 --2- 172.21.15.98:0/2704222544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55874536e1e0 0x5587453705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543d0f7700 1 --2- 172.21.15.98:0/2704222544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558745363310 0x55874536dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543d8f8700 1 -- 172.21.15.98:0/2704222544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558745363310 msgr2=0x55874536dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543d8f8700 1 --2- 172.21.15.98:0/2704222544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558745363310 0x55874536dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543d8f8700 1 -- 172.21.15.98:0/2704222544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55874536e1e0 msgr2=0x5587453705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543d8f8700 1 --2- 172.21.15.98:0/2704222544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55874536e1e0 0x5587453705d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543d8f8700 1 -- 172.21.15.98:0/2704222544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587452039d0 con 0x5587453629f0 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.452+0000 7f543e0f9700 1 --2- 172.21.15.98:0/2704222544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55874536e1e0 0x5587453705d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.453+0000 7f543d8f8700 1 --2- 172.21.15.98:0/2704222544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587453629f0 0x558745362dd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f543400dbd0 tx=0x7f5434010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461dfaef1c14249b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.453+0000 7f5423fff700 1 -- 172.21.15.98:0/2704222544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5434020070 con 0x5587453629f0 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.453+0000 7f5423fff700 1 -- 172.21.15.98:0/2704222544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f543400fde0 con 0x5587453629f0 2024-04-25T17:48:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.453+0000 7f5423fff700 1 -- 172.21.15.98:0/2704222544 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543401b3b0 con 0x5587453629f0 2024-04-25T17:48:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.453+0000 7f544a3e5b80 1 -- 172.21.15.98:0/2704222544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587453629f0 msgr2=0x558745362dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/2704222544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587453629f0 0x558745362dd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f543400dbd0 tx=0x7f5434010000 comp rx=0 tx=0).stop 2024-04-25T17:48:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 -- 172.21.15.98:0/2704222544 shutdown_connections 2024-04-25T17:48:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/2704222544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55874536e1e0 0x5587453705d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/2704222544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5587453629f0 0x558745362dd0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/2704222544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558745363310 0x55874536dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 -- 172.21.15.98:0/2704222544 >> 172.21.15.98:0/2704222544 conn(0x558745290790 msgr2=0x55874535eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 -- 172.21.15.98:0/2704222544 shutdown_connections 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 -- 172.21.15.98:0/2704222544 wait complete. 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 -- start start 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 Processor -- start 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587453629f0 0x55874536d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558745363310 0x55874536ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.454+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558745370bd0 con 0x55874536e1e0 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558745254cd0 con 0x5587453629f0 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558745370d50 con 0x558745363310 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f544a3e5b80 10 client.0 _ll_register_callbacks cb 0x55874524ea50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543e0f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543e0f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59520/0 (socket says 172.21.15.98:59520) 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543e0f9700 1 -- 172.21.15.98:0/3342573908 learned_addr learned my addr 172.21.15.98:0/3342573908 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543d0f7700 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558745363310 0x55874536ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543d8f8700 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587453629f0 0x55874536d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543e0f9700 1 -- 172.21.15.98:0/3342573908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558745363310 msgr2=0x55874536ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543e0f9700 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558745363310 0x55874536ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543e0f9700 1 -- 172.21.15.98:0/3342573908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587453629f0 msgr2=0x55874536d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543e0f9700 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587453629f0 0x55874536d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543e0f9700 1 -- 172.21.15.98:0/3342573908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55874526d370 con 0x55874536e1e0 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f543e0f9700 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536b160 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f54300cdfd0 tx=0x7f543009c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb78bef67aa32ad6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f54217fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f54217fa700 1 -- 172.21.15.98:0/3342573908 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543009cc20 con 0x55874536e1e0 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f54217fa700 1 -- 172.21.15.98:0/3342573908 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f54300a4fa0 con 0x55874536e1e0 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f544a3e5b80 1 -- 172.21.15.98:0/3342573908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f543400fde0 con 0x55874536e1e0 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f54217fa700 1 -- 172.21.15.98:0/3342573908 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f543009cec0 con 0x55874536e1e0 2024-04-25T17:48:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f544a3e5b80 10 client.16653 fetch_fsmap learned FSMap version 48 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.455+0000 7f544a3e5b80 1 -- 172.21.15.98:0/3342573908 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55874537b3d0 con 0x55874536e1e0 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.456+0000 7f54217fa700 1 -- 172.21.15.98:0/3342573908 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f543400fde0 con 0x55874536e1e0 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.456+0000 7f54217fa700 1 -- 172.21.15.98:0/3342573908 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 48) ==== 8346+0+0 (secure 0 0 0) 0x7f543009daf0 con 0x55874536e1e0 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.456+0000 7f544a3e5b80 10 client.16653 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.456+0000 7f544a3e5b80 10 client.16653 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.456+0000 7f544a3e5b80 20 client.16653 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.456+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55874537da10 0x55874537fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.456+0000 7f544a3e5b80 4 client.16653 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.456+0000 7f544a3e5b80 1 -- 172.21.15.98:0/3342573908 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587452c3620 con 0x55874537da10 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.456+0000 7f543d8f8700 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55874537da10 0x55874537fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.457+0000 7f543d8f8700 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55874537da10 0x55874537fe00 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bec00fceeb9e4fb0 server_cookie=6bcc09a311eae8ef in_seq=0 out_seq=0 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.457+0000 7f54217fa700 10 client.16653 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.457+0000 7f54217fa700 1 -- 172.21.15.98:0/3342573908 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5587452c3620 con 0x55874537da10 2024-04-25T17:48:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.457+0000 7f54217fa700 10 client.16653 handle_command_reply: tid=0 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.457+0000 7f544a3e5b80 1 client.16653 shutdown 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.457+0000 7f544a3e5b80 1 -- 172.21.15.98:0/3342573908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 msgr2=0x55874536b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.457+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536b160 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f54300cdfd0 tx=0x7f543009c3f0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.457+0000 7f544a3e5b80 1 -- 172.21.15.98:0/3342573908 shutdown_connections 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.458+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55874537da10 0x55874537fe00 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.458+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536b160 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.458+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5587453629f0 0x55874536d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.458+0000 7f54217fa700 0 client.16653 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.458+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/3342573908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558745363310 0x55874536ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.458+0000 7f544a3e5b80 1 -- 172.21.15.98:0/3342573908 >> 172.21.15.98:0/3342573908 conn(0x558745290790 msgr2=0x55874535eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.458+0000 7f544a3e5b80 1 -- 172.21.15.98:0/3342573908 shutdown_connections 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.458+0000 7f544a3e5b80 1 -- 172.21.15.98:0/3342573908 wait complete. 2024-04-25T17:48:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.458+0000 7f544a3e5b80 20 client.16653 trim_cache size 0 max 16384 2024-04-25T17:48:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f544a3e5b80 1 Processor -- start 2024-04-25T17:48:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f544a3e5b80 1 -- start start 2024-04-25T17:48:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587453629f0 0x558745369b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558745363310 0x558745379a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54381b1f00 con 0x55874536e1e0 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f54381af730 con 0x558745363310 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f54381ad5a0 con 0x5587453629f0 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543e0f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543e0f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55044/0 (socket says 172.21.15.98:55044) 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543e0f9700 1 -- 172.21.15.98:0/2965613771 learned_addr learned my addr 172.21.15.98:0/2965613771 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543d8f8700 1 --2- 172.21.15.98:0/2965613771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587453629f0 0x558745369b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543d0f7700 1 --2- 172.21.15.98:0/2965613771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558745363310 0x558745379a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543e0f9700 1 -- 172.21.15.98:0/2965613771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587453629f0 msgr2=0x558745369b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543e0f9700 1 --2- 172.21.15.98:0/2965613771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587453629f0 0x558745369b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543e0f9700 1 -- 172.21.15.98:0/2965613771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558745363310 msgr2=0x558745379a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543e0f9700 1 --2- 172.21.15.98:0/2965613771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558745363310 0x558745379a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543e0f9700 1 -- 172.21.15.98:0/2965613771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558745380390 con 0x55874536e1e0 2024-04-25T17:48:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.477+0000 7f543e0f9700 1 --2- 172.21.15.98:0/2965613771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536cb60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f543000a740 tx=0x7f5430098460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9efae2285644951 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.478+0000 7f54237fe700 1 -- 172.21.15.98:0/2965613771 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5430012420 con 0x55874536e1e0 2024-04-25T17:48:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.478+0000 7f54237fe700 1 -- 172.21.15.98:0/2965613771 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54300a0b70 con 0x55874536e1e0 2024-04-25T17:48:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.478+0000 7f54237fe700 1 -- 172.21.15.98:0/2965613771 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54300985e0 con 0x55874536e1e0 2024-04-25T17:48:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.478+0000 7f544a3e5b80 1 -- 172.21.15.98:0/2965613771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 msgr2=0x55874536cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/2965613771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536cb60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f543000a740 tx=0x7f5430098460 comp rx=0 tx=0).stop 2024-04-25T17:48:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 -- 172.21.15.98:0/2965613771 shutdown_connections 2024-04-25T17:48:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/2965613771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587453629f0 0x558745369b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/2965613771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x55874536cb60 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/2965613771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558745363310 0x558745379a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 -- 172.21.15.98:0/2965613771 >> 172.21.15.98:0/2965613771 conn(0x558745290790 msgr2=0x55874536c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 -- 172.21.15.98:0/2965613771 shutdown_connections 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 -- 172.21.15.98:0/2965613771 wait complete. 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 -- start start 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 Processor -- start 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.479+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587453629f0 0x558745374b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558745363310 0x558745375090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f544a3e5b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x5587453755d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f54381adbf0 con 0x55874536e1e0 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f54381af060 con 0x558745363310 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f544a3e5b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f54381ae2c0 con 0x5587453629f0 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f544a3e5b80 10 client.0 _ll_register_callbacks cb 0x55874524ea50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543e0f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x5587453755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543e0f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x5587453755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55046/0 (socket says 172.21.15.98:55046) 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543e0f9700 1 -- 172.21.15.98:0/1831368334 learned_addr learned my addr 172.21.15.98:0/1831368334 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543d8f8700 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587453629f0 0x558745374b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543d0f7700 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558745363310 0x558745375090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543e0f9700 1 -- 172.21.15.98:0/1831368334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587453629f0 msgr2=0x558745374b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543e0f9700 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587453629f0 0x558745374b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543e0f9700 1 -- 172.21.15.98:0/1831368334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558745363310 msgr2=0x558745375090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543e0f9700 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558745363310 0x558745375090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543e0f9700 1 -- 172.21.15.98:0/1831368334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587453e3fe0 con 0x55874536e1e0 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f543e0f9700 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x5587453755d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f54300ce460 tx=0x7f5430098840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888cfa6f9cec5ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f54217fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f54217fa700 1 -- 172.21.15.98:0/1831368334 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54300a5430 con 0x55874536e1e0 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f54217fa700 1 -- 172.21.15.98:0/1831368334 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f54300672b0 con 0x55874536e1e0 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.480+0000 7f544a3e5b80 1 -- 172.21.15.98:0/1831368334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54300a0b70 con 0x55874536e1e0 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f54217fa700 1 -- 172.21.15.98:0/1831368334 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f543009cc30 con 0x55874536e1e0 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f544a3e5b80 10 client.16659 fetch_fsmap learned FSMap version 48 2024-04-25T17:48:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f544a3e5b80 1 -- 172.21.15.98:0/1831368334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5587453e5300 con 0x55874536e1e0 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f54217fa700 1 -- 172.21.15.98:0/1831368334 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54300a4fa0 con 0x55874536e1e0 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f54217fa700 1 -- 172.21.15.98:0/1831368334 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 48) ==== 8346+0+0 (secure 0 0 0) 0x7f543009b680 con 0x55874536e1e0 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f544a3e5b80 10 client.16659 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f544a3e5b80 10 client.16659 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f544a3e5b80 20 client.16659 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5587453e5710 0x5587453e7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f544a3e5b80 4 client.16659 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "964d443e", "release": true, "await": true}] 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f544a3e5b80 1 -- 172.21.15.98:0/1831368334 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "964d443e", "release": true, "await": true}) -- 0x7f54300a0b70 con 0x5587453e5710 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.481+0000 7f543d8f8700 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5587453e5710 0x5587453e7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.482+0000 7f54217fa700 10 client.16659 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.482+0000 7f543d8f8700 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5587453e5710 0x5587453e7b00 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=580852e534d4c7bd server_cookie=fd3195c50ffd3171 in_seq=0 out_seq=0 2024-04-25T17:48:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.485+0000 7f54217fa700 1 -- 172.21.15.98:0/1831368334 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x7f54300a0b70 con 0x5587453e5710 2024-04-25T17:48:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.485+0000 7f54217fa700 10 client.16659 handle_command_reply: tid=0 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.485+0000 7f544a3e5b80 1 client.16659 shutdown 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 1 -- 172.21.15.98:0/1831368334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 msgr2=0x5587453755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x5587453755d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f54300ce460 tx=0x7f5430098840 comp rx=0 tx=0).stop 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 1 -- 172.21.15.98:0/1831368334 shutdown_connections 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5587453e5710 0x5587453e7b00 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5587453629f0 0x558745374b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55874536e1e0 0x5587453755d0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f54217fa700 0 client.16659 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 1 --2- 172.21.15.98:0/1831368334 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558745363310 0x558745375090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 1 -- 172.21.15.98:0/1831368334 >> 172.21.15.98:0/1831368334 conn(0x558745290790 msgr2=0x55874536cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 1 -- 172.21.15.98:0/1831368334 shutdown_connections 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 1 -- 172.21.15.98:0/1831368334 wait complete. 2024-04-25T17:48:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f544a3e5b80 20 client.16659 trim_cache size 0 max 16384 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 48, 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 36, 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "964d443e": { 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 36, 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:48:13.488 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 76.8, 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:48:13.489 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:48:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f543fb5c700 1 -- 172.21.15.98:0/3070534711 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f54240a3db0 msgr2=0x7f54240a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f543fb5c700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f54240a3db0 0x7f54240a6270 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5434005fd0 tx=0x7f5434005d40 comp rx=0 tx=0).stop 2024-04-25T17:48:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f543fb5c700 1 -- 172.21.15.98:0/3070534711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5438108a70 msgr2=0x7f54380757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.486+0000 7f543fb5c700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5438108a70 0x7f54380757c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5430007d70 tx=0x7f543000a340 comp rx=0 tx=0).stop 2024-04-25T17:48:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.487+0000 7f543fb5c700 1 -- 172.21.15.98:0/3070534711 shutdown_connections 2024-04-25T17:48:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.487+0000 7f543fb5c700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f54240a3db0 0x7f54240a6270 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.487+0000 7f543fb5c700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5438108a70 0x7f54380757c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.487+0000 7f543fb5c700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5438102ac0 0x7f5438074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.487+0000 7f543fb5c700 1 --2- 172.21.15.98:0/3070534711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5438103460 0x7f5438075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.487+0000 7f543fb5c700 1 -- 172.21.15.98:0/3070534711 >> 172.21.15.98:0/3070534711 conn(0x7f5438079b10 msgr2=0x7f5438100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.487+0000 7f543fb5c700 1 -- 172.21.15.98:0/3070534711 shutdown_connections 2024-04-25T17:48:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:13.487+0000 7f543fb5c700 1 -- 172.21.15.98:0/3070534711 wait complete. 2024-04-25T17:48:14.292 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '964d443e', seconds elapsed: 30.0 2024-04-25T17:48:14.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:14 smithi098 ceph-mon[29192]: pgmap v387: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 KiB/s wr, 12 op/s 2024-04-25T17:48:14.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:14 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:48:14.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:14 smithi203 ceph-mon[32834]: pgmap v387: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 KiB/s wr, 12 op/s 2024-04-25T17:48:14.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:14 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:48:14.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:14 smithi173 ceph-mon[32856]: pgmap v387: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 KiB/s wr, 12 op/s 2024-04-25T17:48:14.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:14 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:48:15.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:15 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:48:15.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:15 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:48:15.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:15 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:48:15.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:15 smithi098 ceph-mon[29192]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T17:48:15.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:15 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:48:15.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:15 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:48:15.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:15 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:48:15.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:15 smithi203 ceph-mon[32834]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T17:48:15.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:15 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:48:15.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:15 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:48:15.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:15 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:48:15.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:15 smithi173 ceph-mon[32856]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T17:48:16.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 705.0 s, 1174.0 tps, lat 451.873 ms stddev 3061.619 2024-04-25T17:48:16.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:16 smithi098 ceph-mon[29192]: pgmap v388: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 KiB/s wr, 12 op/s 2024-04-25T17:48:16.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:16 smithi203 ceph-mon[32834]: pgmap v388: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 KiB/s wr, 12 op/s 2024-04-25T17:48:16.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:16 smithi173 ceph-mon[32856]: pgmap v388: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 28 KiB/s wr, 12 op/s 2024-04-25T17:48:18.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:18 smithi098 ceph-mon[29192]: pgmap v390: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-25T17:48:18.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:18 smithi203 ceph-mon[32834]: pgmap v390: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-25T17:48:18.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:18 smithi173 ceph-mon[32856]: pgmap v390: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-25T17:48:20.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:20 smithi098 ceph-mon[29192]: pgmap v391: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-25T17:48:20.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:20 smithi203 ceph-mon[32834]: pgmap v391: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-25T17:48:20.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:20 smithi173 ceph-mon[32856]: pgmap v391: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-25T17:48:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 710.0 s, 2167.0 tps, lat 45.941 ms stddev 56.393 2024-04-25T17:48:22.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:22 smithi098 ceph-mon[29192]: pgmap v392: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-25T17:48:22.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:22 smithi203 ceph-mon[32834]: pgmap v392: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-25T17:48:22.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:22 smithi173 ceph-mon[32856]: pgmap v392: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-04-25T17:48:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.365+0000 7fc3186cf640 1 -- 172.21.15.98:0/2507340564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc310101110 msgr2=0x7fc3101014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:23.364 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.365+0000 7fc3186cf640 1 --2- 172.21.15.98:0/2507340564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc310101110 0x7fc3101014f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009900 tx=0x7fc30c02a410 comp rx=0 tx=0).stop 2024-04-25T17:48:23.365 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.365+0000 7fc3186cf640 1 -- 172.21.15.98:0/2507340564 shutdown_connections 2024-04-25T17:48:23.365 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.365+0000 7fc3186cf640 1 --2- 172.21.15.98:0/2507340564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc310106190 0x7fc31010dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.365 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.365+0000 7fc3186cf640 1 --2- 172.21.15.98:0/2507340564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc310101ac0 0x7fc310105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.365 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.365+0000 7fc3186cf640 1 --2- 172.21.15.98:0/2507340564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc310101110 0x7fc3101014f0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.365 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.365+0000 7fc3186cf640 1 -- 172.21.15.98:0/2507340564 >> 172.21.15.98:0/2507340564 conn(0x7fc310077fa0 msgr2=0x7fc3100783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:23.365 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.365+0000 7fc3186cf640 1 -- 172.21.15.98:0/2507340564 shutdown_connections 2024-04-25T17:48:23.365 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.365+0000 7fc3186cf640 1 -- 172.21.15.98:0/2507340564 wait complete. 2024-04-25T17:48:23.367 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.366+0000 7fc3186cf640 1 Processor -- start 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.367+0000 7fc3186cf640 1 -- start start 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.367+0000 7fc3186cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc310101110 0x7fc310196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.367+0000 7fc3186cf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc310106190 0x7fc310196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.367+0000 7fc3186cf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc31019adf0 0x7fc31019b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.367+0000 7fc3186cf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010fff0 con 0x7fc310101110 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.367+0000 7fc315c6a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc310106190 0x7fc310196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.367+0000 7fc315c6a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc310106190 0x7fc310196a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:60944/0 (socket says 172.21.15.98:60944) 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.367+0000 7fc315c6a640 1 -- 172.21.15.98:0/984200257 learned_addr learned my addr 172.21.15.98:0/984200257 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.368+0000 7fc316c6c640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc31019adf0 0x7fc31019b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.368+0000 7fc31646b640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc310101110 0x7fc310196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.368+0000 7fc3186cf640 1 -- 172.21.15.98:0/984200257 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc31010fe70 con 0x7fc31019adf0 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.368+0000 7fc3186cf640 1 -- 172.21.15.98:0/984200257 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc310110170 con 0x7fc310106190 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.368+0000 7fc316c6c640 1 -- 172.21.15.98:0/984200257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc310106190 msgr2=0x7fc310196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.368+0000 7fc316c6c640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc310106190 0x7fc310196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.368+0000 7fc316c6c640 1 -- 172.21.15.98:0/984200257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc310101110 msgr2=0x7fc310196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.368+0000 7fc316c6c640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc310101110 0x7fc310196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.368+0000 7fc316c6c640 1 -- 172.21.15.98:0/984200257 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31019b8e0 con 0x7fc31019adf0 2024-04-25T17:48:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.368+0000 7fc316c6c640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc31019adf0 0x7fc31019b2a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc308010f90 tx=0x7fc30800e490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:23.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.369+0000 7fc3077fe640 1 -- 172.21.15.98:0/984200257 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30800b230 con 0x7fc31019adf0 2024-04-25T17:48:23.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.369+0000 7fc3186cf640 1 -- 172.21.15.98:0/984200257 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31019bbd0 con 0x7fc31019adf0 2024-04-25T17:48:23.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.369+0000 7fc3077fe640 1 -- 172.21.15.98:0/984200257 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30800eb20 con 0x7fc31019adf0 2024-04-25T17:48:23.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.369+0000 7fc3077fe640 1 -- 172.21.15.98:0/984200257 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308015d10 con 0x7fc31019adf0 2024-04-25T17:48:23.370 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.369+0000 7fc3186cf640 1 -- 172.21.15.98:0/984200257 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3101a3410 con 0x7fc31019adf0 2024-04-25T17:48:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.372+0000 7fc3077fe640 1 -- 172.21.15.98:0/984200257 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc308004a40 con 0x7fc31019adf0 2024-04-25T17:48:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.373+0000 7fc3077fe640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc2fc0a3dc0 0x7fc2fc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.373 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.373+0000 7fc3077fe640 1 -- 172.21.15.98:0/984200257 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7fc3080d4590 con 0x7fc31019adf0 2024-04-25T17:48:23.373 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc31b3e3740 1 Processor -- start 2024-04-25T17:48:23.374 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc31646b640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc2fc0a3dc0 0x7fc2fc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc31b3e3740 1 -- start start 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836ac9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836ad42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ad4800 0x557836ad6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557836a2e970 con 0x557836ac9620 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5578369cb4a0 con 0x557836ad4800 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557836a2a930 con 0x557836ac9f40 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc315c6a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836ad42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc315c6a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836ad42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:60956/0 (socket says 172.21.15.98:60956) 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.374+0000 7fc315c6a640 1 -- 172.21.15.98:0/1617411626 learned_addr learned my addr 172.21.15.98:0/1617411626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc316c6c640 1 --2- 172.21.15.98:0/1617411626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ad4800 0x557836ad6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc315c6a640 1 -- 172.21.15.98:0/1617411626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ad4800 msgr2=0x557836ad6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc315c6a640 1 --2- 172.21.15.98:0/1617411626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ad4800 0x557836ad6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc315c6a640 1 -- 172.21.15.98:0/1617411626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 msgr2=0x557836ac9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc315c6a640 1 --2- 172.21.15.98:0/1617411626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836ac9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc315c6a640 1 -- 172.21.15.98:0/1617411626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557836741210 con 0x557836ac9f40 2024-04-25T17:48:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc315c6a640 1 --2- 172.21.15.98:0/1617411626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836ad42c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc300009d00 tx=0x7fc300011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c917ccb14eb48d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc31646b640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc2fc0a3dc0 0x7fc2fc0a6280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc30c031440 tx=0x7fc30c032bc0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc304ff9640 1 -- 172.21.15.98:0/1617411626 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300005220 con 0x557836ac9f40 2024-04-25T17:48:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc304ff9640 1 -- 172.21.15.98:0/1617411626 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc300004c50 con 0x557836ac9f40 2024-04-25T17:48:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.375+0000 7fc304ff9640 1 -- 172.21.15.98:0/1617411626 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300019740 con 0x557836ac9f40 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.376+0000 7fc31b3e3740 1 -- 172.21.15.98:0/1617411626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 msgr2=0x557836ad42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.376+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/1617411626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836ad42c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc300009d00 tx=0x7fc300011810 comp rx=0 tx=0).stop 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.376+0000 7fc31b3e3740 1 -- 172.21.15.98:0/1617411626 shutdown_connections 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.376+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/1617411626 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ad4800 0x557836ad6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.376+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/1617411626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836ad42c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.376+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/1617411626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836ac9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.376+0000 7fc31b3e3740 1 -- 172.21.15.98:0/1617411626 >> 172.21.15.98:0/1617411626 conn(0x557836ac5b30 msgr2=0x557836ac5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.376+0000 7fc31b3e3740 1 -- 172.21.15.98:0/1617411626 shutdown_connections 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.376+0000 7fc31b3e3740 1 -- 172.21.15.98:0/1617411626 wait complete. 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.377+0000 7fc31b3e3740 1 -- start start 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.377+0000 7fc31b3e3740 1 Processor -- start 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836ad40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac9f40 0x557836ad13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ad4800 0x557836ad18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557836ad7430 con 0x557836ac9620 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5578369c90b0 con 0x557836ac9f40 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557836ad75b0 con 0x557836ad4800 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc31646b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836ad40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc31b3e3740 10 client.0 _ll_register_callbacks cb 0x5578369b6d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc31646b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836ad40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55082/0 (socket says 172.21.15.98:55082) 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc31646b640 1 -- 172.21.15.98:0/2736993548 learned_addr learned my addr 172.21.15.98:0/2736993548 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc315c6a640 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac9f40 0x557836ad13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc316c6c640 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ad4800 0x557836ad18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.378+0000 7fc316c6c640 1 -- 172.21.15.98:0/2736993548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac9f40 msgr2=0x557836ad13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.379+0000 7fc316c6c640 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac9f40 0x557836ad13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.379+0000 7fc316c6c640 1 -- 172.21.15.98:0/2736993548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 msgr2=0x557836ad40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.379+0000 7fc316c6c640 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836ad40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.379+0000 7fc316c6c640 1 -- 172.21.15.98:0/2736993548 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55783680d620 con 0x557836ad4800 2024-04-25T17:48:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.379+0000 7fc316c6c640 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ad4800 0x557836ad18f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc3080d1ce0 tx=0x7fc30809ff40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95992b929f94f4d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.379+0000 7fc2f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:48:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.379+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/2736993548 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308023050 con 0x557836ad4800 2024-04-25T17:48:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.379+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/2736993548 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7fc30809cc50 con 0x557836ad4800 2024-04-25T17:48:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.379+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2736993548 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc300004c50 con 0x557836ad4800 2024-04-25T17:48:23.381 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.379+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/2736993548 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc3080a48e0 con 0x557836ad4800 2024-04-25T17:48:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.380+0000 7fc31b3e3740 10 client.26146 fetch_fsmap learned FSMap version 48 2024-04-25T17:48:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.380+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2736993548 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x557836ae32e0 con 0x557836ad4800 2024-04-25T17:48:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.380+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/2736993548 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc300004c50 con 0x557836ad4800 2024-04-25T17:48:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.380+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/2736993548 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 48) ==== 8346+0+0 (secure 0 0 0) 0x7fc3080bf1b0 con 0x557836ad4800 2024-04-25T17:48:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.380+0000 7fc31b3e3740 10 client.26146 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T17:48:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.380+0000 7fc31b3e3740 10 client.26146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:48:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.380+0000 7fc31b3e3740 20 client.26146 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:23.383 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.380+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x557836ae4750 0x557836ae6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.380+0000 7fc31b3e3740 4 client.26146 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.380+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2736993548 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557836a3c390 con 0x557836ae4750 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.381+0000 7fc31646b640 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x557836ae4750 0x557836ae6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.381+0000 7fc2f9ffb640 10 client.26146 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.381+0000 7fc31646b640 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x557836ae4750 0x557836ae6b40 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d23d2b8992f95a63 server_cookie=a319e033e337dde0 in_seq=0 out_seq=0 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/2736993548 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557836a3c390 con 0x557836ae4750 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc2f9ffb640 10 client.26146 handle_command_reply: tid=0 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 client.26146 shutdown 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2736993548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ad4800 msgr2=0x557836ad18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ad4800 0x557836ad18f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc3080d1ce0 tx=0x7fc30809ff40 comp rx=0 tx=0).stop 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2736993548 shutdown_connections 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x557836ae4750 0x557836ae6b40 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ad4800 0x557836ad18f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac9f40 0x557836ad13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/2736993548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836ad40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2736993548 >> 172.21.15.98:0/2736993548 conn(0x557836ac5b30 msgr2=0x557836acb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc2f9ffb640 0 client.26146 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2736993548 shutdown_connections 2024-04-25T17:48:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2736993548 wait complete. 2024-04-25T17:48:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.382+0000 7fc31b3e3740 20 client.26146 trim_cache size 0 max 16384 2024-04-25T17:48:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.399+0000 7fc31b3e3740 1 Processor -- start 2024-04-25T17:48:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.400+0000 7fc31b3e3740 1 -- start start 2024-04-25T17:48:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 0x557836ae02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836aec400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836aec940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101ae380 con 0x557836ac9620 2024-04-25T17:48:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101abbf0 con 0x557836ac5b30 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101a9a60 con 0x557836ac9f40 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc315c6a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836aec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc315c6a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836aec400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55088/0 (socket says 172.21.15.98:55088) 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc315c6a640 1 -- 172.21.15.98:0/2466675335 learned_addr learned my addr 172.21.15.98:0/2466675335 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc31646b640 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 0x557836ae02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc316c6c640 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836aec940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc315c6a640 1 -- 172.21.15.98:0/2466675335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 msgr2=0x557836aec940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc315c6a640 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836aec940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc315c6a640 1 -- 172.21.15.98:0/2466675335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 msgr2=0x557836ae02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc315c6a640 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 0x557836ae02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.407 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc315c6a640 1 -- 172.21.15.98:0/2466675335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557836ac12e0 con 0x557836ac9620 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc316c6c640 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836aec940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc31646b640 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 0x557836ae02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.401+0000 7fc315c6a640 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836aec400 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc300009c50 tx=0x7fc3000093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139b1caa0a819ce7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.402+0000 7fc2fbfff640 1 -- 172.21.15.98:0/2466675335 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3000050b0 con 0x557836ac9620 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.402+0000 7fc2fbfff640 1 -- 172.21.15.98:0/2466675335 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3000035f0 con 0x557836ac9620 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.402+0000 7fc2fbfff640 1 -- 172.21.15.98:0/2466675335 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3000266c0 con 0x557836ac9620 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.402+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2466675335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 msgr2=0x557836aec400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.402+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836aec400 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc300009c50 tx=0x7fc3000093c0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.402+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2466675335 shutdown_connections 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.402+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9f40 0x557836aec940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.402+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9620 0x557836aec400 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.402+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/2466675335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 0x557836ae02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.408 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.402+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2466675335 >> 172.21.15.98:0/2466675335 conn(0x557836ad1a50 msgr2=0x557836ac4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2466675335 shutdown_connections 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 1 -- 172.21.15.98:0/2466675335 wait complete. 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 1 -- start start 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 1 Processor -- start 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 0x557836adccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9620 0x557836ada040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9f40 0x557836ada650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101aa0b0 con 0x557836ac9f40 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101ab520 con 0x557836ac5b30 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3101aa780 con 0x557836ac9620 2024-04-25T17:48:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31b3e3740 10 client.0 _ll_register_callbacks cb 0x7fc3080a03b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc316c6c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9f40 0x557836ada650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc31646b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 0x557836adccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc316c6c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9f40 0x557836ada650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55100/0 (socket says 172.21.15.98:55100) 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc316c6c640 1 -- 172.21.15.98:0/4169464016 learned_addr learned my addr 172.21.15.98:0/4169464016 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.403+0000 7fc315c6a640 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9620 0x557836ada040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc316c6c640 1 -- 172.21.15.98:0/4169464016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9620 msgr2=0x557836ada040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc316c6c640 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9620 0x557836ada040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc316c6c640 1 -- 172.21.15.98:0/4169464016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 msgr2=0x557836adccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc316c6c640 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 0x557836adccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc316c6c640 1 -- 172.21.15.98:0/4169464016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557836adae80 con 0x557836ac9f40 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc31646b640 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 0x557836adccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:48:23.410 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc316c6c640 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9f40 0x557836ada650 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc30809cfb0 tx=0x7fc3080a0070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7315fc678485aab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc2f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/4169464016 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308023070 con 0x557836ac9f40 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc31b3e3740 1 -- 172.21.15.98:0/4169464016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3000035f0 con 0x557836ac9f40 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/4169464016 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7fc30809fc30 con 0x557836ac9f40 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/4169464016 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc3080d43d0 con 0x557836ac9f40 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc31b3e3740 10 client.16674 fetch_fsmap learned FSMap version 48 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc31b3e3740 1 -- 172.21.15.98:0/4169464016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x557836b4ad90 con 0x557836ac9f40 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/4169464016 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3000035f0 con 0x557836ac9f40 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/4169464016 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 48) ==== 8346+0+0 (secure 0 0 0) 0x7fc30809c330 con 0x557836ac9f40 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc31b3e3740 10 client.16674 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T17:48:23.411 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.404+0000 7fc31b3e3740 10 client.16674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:48:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.405+0000 7fc31b3e3740 20 client.16674 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.405+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x557836ad4800 0x557836b4d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.405+0000 7fc31b3e3740 4 client.16674 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:48:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.405+0000 7fc31b3e3740 1 -- 172.21.15.98:0/4169464016 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557836a36200 con 0x557836ad4800 2024-04-25T17:48:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.405+0000 7fc31646b640 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x557836ad4800 0x557836b4d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.405+0000 7fc31646b640 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x557836ad4800 0x557836b4d2e0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb8e00696144e892 server_cookie=b67e4c6e61a16bd3 in_seq=0 out_seq=0 2024-04-25T17:48:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.405+0000 7fc2f9ffb640 10 client.16674 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc2f9ffb640 1 -- 172.21.15.98:0/4169464016 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557836a36200 con 0x557836ad4800 2024-04-25T17:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc2f9ffb640 10 client.16674 handle_command_reply: tid=0 2024-04-25T17:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 client.16674 shutdown 2024-04-25T17:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 -- 172.21.15.98:0/4169464016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9f40 msgr2=0x557836ada650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9f40 0x557836ada650 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc30809cfb0 tx=0x7fc3080a0070 comp rx=0 tx=0).stop 2024-04-25T17:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 -- 172.21.15.98:0/4169464016 shutdown_connections 2024-04-25T17:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x557836ad4800 0x557836b4d2e0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557836ac9f40 0x557836ada650 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557836ac9620 0x557836ada040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 --2- 172.21.15.98:0/4169464016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557836ac5b30 0x557836adccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 -- 172.21.15.98:0/4169464016 >> 172.21.15.98:0/4169464016 conn(0x557836ad1a50 msgr2=0x557836ac4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc2f9ffb640 0 client.16674 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 -- 172.21.15.98:0/4169464016 shutdown_connections 2024-04-25T17:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 1 -- 172.21.15.98:0/4169464016 wait complete. 2024-04-25T17:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.415+0000 7fc31b3e3740 20 client.16674 trim_cache size 0 max 16384 2024-04-25T17:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:48:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:48:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 -- 172.21.15.98:0/984200257 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc2fc0a3dc0 msgr2=0x7fc2fc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc2fc0a3dc0 0x7fc2fc0a6280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc30c031440 tx=0x7fc30c032bc0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 -- 172.21.15.98:0/984200257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc31019adf0 msgr2=0x7fc31019b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc31019adf0 0x7fc31019b2a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc308010f90 tx=0x7fc30800e490 comp rx=0 tx=0).stop 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 -- 172.21.15.98:0/984200257 shutdown_connections 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fc2fc0a3dc0 0x7fc2fc0a6280 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc31019adf0 0x7fc31019b2a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc310106190 0x7fc310196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 --2- 172.21.15.98:0/984200257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc310101110 0x7fc310196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 -- 172.21.15.98:0/984200257 >> 172.21.15.98:0/984200257 conn(0x7fc310077fa0 msgr2=0x7fc3100ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:23.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 -- 172.21.15.98:0/984200257 shutdown_connections 2024-04-25T17:48:23.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:23.416+0000 7fc3186cf640 1 -- 172.21.15.98:0/984200257 wait complete. 2024-04-25T17:48:24.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:24 smithi098 ceph-mon[29192]: pgmap v393: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.7 MiB/s wr, 493 op/s 2024-04-25T17:48:24.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:24 smithi203 ceph-mon[32834]: pgmap v393: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.7 MiB/s wr, 493 op/s 2024-04-25T17:48:24.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:24 smithi173 ceph-mon[32856]: pgmap v393: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.7 MiB/s wr, 493 op/s 2024-04-25T17:48:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 715.0 s, 2038.0 tps, lat 49.341 ms stddev 53.559 2024-04-25T17:48:26.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:26 smithi098 ceph-mon[29192]: pgmap v394: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.7 MiB/s wr, 493 op/s 2024-04-25T17:48:26.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:26 smithi203 ceph-mon[32834]: pgmap v394: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.7 MiB/s wr, 493 op/s 2024-04-25T17:48:26.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:26 smithi173 ceph-mon[32856]: pgmap v394: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.7 MiB/s wr, 493 op/s 2024-04-25T17:48:28.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:28 smithi098 ceph-mon[29192]: pgmap v395: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 MiB/s wr, 711 op/s 2024-04-25T17:48:28.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:28 smithi203 ceph-mon[32834]: pgmap v395: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 MiB/s wr, 711 op/s 2024-04-25T17:48:28.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:28 smithi173 ceph-mon[32856]: pgmap v395: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 MiB/s wr, 711 op/s 2024-04-25T17:48:30.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:30 smithi098 ceph-mon[29192]: pgmap v396: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 537 op/s 2024-04-25T17:48:30.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:30 smithi203 ceph-mon[32834]: pgmap v396: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 537 op/s 2024-04-25T17:48:30.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:30 smithi173 ceph-mon[32856]: pgmap v396: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 537 op/s 2024-04-25T17:48:31.594 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 720.0 s, 1936.4 tps, lat 50.942 ms stddev 64.368 2024-04-25T17:48:32.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:32 smithi098 ceph-mon[29192]: pgmap v397: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 537 op/s 2024-04-25T17:48:32.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:32 smithi203 ceph-mon[32834]: pgmap v397: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 537 op/s 2024-04-25T17:48:32.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:32 smithi173 ceph-mon[32856]: pgmap v397: 97 pgs: 97 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 537 op/s 2024-04-25T17:48:34.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:34 smithi098 ceph-mon[29192]: pgmap v398: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 7.5 MiB/s wr, 802 op/s 2024-04-25T17:48:34.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:34 smithi203 ceph-mon[32834]: pgmap v398: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 7.5 MiB/s wr, 802 op/s 2024-04-25T17:48:34.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:34 smithi173 ceph-mon[32856]: pgmap v398: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 7.5 MiB/s wr, 802 op/s 2024-04-25T17:48:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.540+0000 7f0eb5f95640 1 -- 172.21.15.98:0/930993311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0eb00742f0 msgr2=0x7f0eb00746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.540+0000 7f0eb5f95640 1 --2- 172.21.15.98:0/930993311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0eb00742f0 0x7f0eb00746d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009900 tx=0x7f0ea002a480 comp rx=0 tx=0).stop 2024-04-25T17:48:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.540+0000 7f0eb5f95640 1 -- 172.21.15.98:0/930993311 shutdown_connections 2024-04-25T17:48:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.540+0000 7f0eb5f95640 1 --2- 172.21.15.98:0/930993311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb0112f40 0x7f0eb0115330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.540+0000 7f0eb5f95640 1 --2- 172.21.15.98:0/930993311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0eb0074c10 0x7f0eb0112a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.540+0000 7f0eb5f95640 1 --2- 172.21.15.98:0/930993311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0eb00742f0 0x7f0eb00746d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.540+0000 7f0eb5f95640 1 -- 172.21.15.98:0/930993311 >> 172.21.15.98:0/930993311 conn(0x7f0eb00fda00 msgr2=0x7f0eb00ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.540+0000 7f0eb5f95640 1 -- 172.21.15.98:0/930993311 shutdown_connections 2024-04-25T17:48:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.540+0000 7f0eb5f95640 1 -- 172.21.15.98:0/930993311 wait complete. 2024-04-25T17:48:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.541+0000 7f0eb5f95640 1 Processor -- start 2024-04-25T17:48:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.541+0000 7f0eb5f95640 1 -- start start 2024-04-25T17:48:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.541+0000 7f0eb5f95640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb00742f0 0x7f0eb0073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.541+0000 7f0eb5f95640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0eb0074c10 0x7f0eb0073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.541+0000 7f0eaf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb00742f0 0x7f0eb0073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.541+0000 7f0eb5f95640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0eb0112f40 0x7f0eb006d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.541+0000 7f0eaf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb00742f0 0x7f0eb0073570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46626/0 (socket says 172.21.15.98:46626) 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eaf7fe640 1 -- 172.21.15.98:0/2450496300 learned_addr learned my addr 172.21.15.98:0/2450496300 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb006c0d0 con 0x7f0eb00742f0 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb006bf50 con 0x7f0eb0074c10 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb006c250 con 0x7f0eb0112f40 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eaeffd640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0eb0074c10 0x7f0eb0073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eaffff640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0eb0112f40 0x7f0eb006d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eaf7fe640 1 -- 172.21.15.98:0/2450496300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0eb0112f40 msgr2=0x7f0eb006d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eaf7fe640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0eb0112f40 0x7f0eb006d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eaf7fe640 1 -- 172.21.15.98:0/2450496300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0eb0074c10 msgr2=0x7f0eb0073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eaf7fe640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0eb0074c10 0x7f0eb0073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eaf7fe640 1 -- 172.21.15.98:0/2450496300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb006de10 con 0x7f0eb00742f0 2024-04-25T17:48:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eaeffd640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0eb0074c10 0x7f0eb0073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:48:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.542+0000 7f0eaf7fe640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb00742f0 0x7f0eb0073570 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009850 tx=0x7f0ea0030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.543+0000 7f0eacff9640 1 -- 172.21.15.98:0/2450496300 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea003f070 con 0x7f0eb00742f0 2024-04-25T17:48:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.543+0000 7f0eacff9640 1 -- 172.21.15.98:0/2450496300 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ea0004460 con 0x7f0eb00742f0 2024-04-25T17:48:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.543+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb006e0a0 con 0x7f0eb00742f0 2024-04-25T17:48:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.543+0000 7f0eacff9640 1 -- 172.21.15.98:0/2450496300 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0035030 con 0x7f0eb00742f0 2024-04-25T17:48:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.543+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eb01b3e20 con 0x7f0eb00742f0 2024-04-25T17:48:36.546 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.545+0000 7f0eacff9640 1 -- 172.21.15.98:0/2450496300 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0ea0042070 con 0x7f0eb00742f0 2024-04-25T17:48:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.545+0000 7f0eb8d83740 1 Processor -- start 2024-04-25T17:48:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eb8d83740 1 -- start start 2024-04-25T17:48:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9ee4b0 0x559f7e9ee890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea0a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 0x559f7ea0cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eb8d83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559f7e953970 con 0x559f7e9ee4b0 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eb8d83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559f7e8f04a0 con 0x559f7ea0a7e0 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eb8d83740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559f7e94f930 con 0x559f7e9eedd0 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eaffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9ee4b0 0x559f7e9ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eaffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9ee4b0 0x559f7e9ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46628/0 (socket says 172.21.15.98:46628) 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eaffff640 1 -- 172.21.15.98:0/202582392 learned_addr learned my addr 172.21.15.98:0/202582392 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.546+0000 7f0eaeffd640 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 0x559f7ea0cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eaeffd640 1 -- 172.21.15.98:0/202582392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 msgr2=0x559f7ea0a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eaf7fe640 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea0a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eaeffd640 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea0a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eaeffd640 1 -- 172.21.15.98:0/202582392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9ee4b0 msgr2=0x559f7e9ee890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eaeffd640 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9ee4b0 0x559f7e9ee890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eaeffd640 1 -- 172.21.15.98:0/202582392 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f7e666210 con 0x559f7ea0a7e0 2024-04-25T17:48:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eaffff640 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9ee4b0 0x559f7e9ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:48:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eaf7fe640 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea0a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:48:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eacff9640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0e980a3e40 0x7f0e980a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eaeffd640 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 0x559f7ea0cbd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0eb006ee30 tx=0x7f0ea40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=576af174fe7a8e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eaffff640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0e980a3e40 0x7f0e980a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0eacff9640 1 -- 172.21.15.98:0/2450496300 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f0ea00f11b0 con 0x7f0eb00742f0 2024-04-25T17:48:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.547+0000 7f0e95ffb640 1 -- 172.21.15.98:0/202582392 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4004310 con 0x559f7ea0a7e0 2024-04-25T17:48:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.548+0000 7f0e95ffb640 1 -- 172.21.15.98:0/202582392 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ea40044b0 con 0x559f7ea0a7e0 2024-04-25T17:48:36.549 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.548+0000 7f0eaffff640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0e980a3e40 0x7f0e980a6300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009fd0 tx=0x7f0ea8009500 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.548+0000 7f0e95ffb640 1 -- 172.21.15.98:0/202582392 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40035f0 con 0x559f7ea0a7e0 2024-04-25T17:48:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.549+0000 7f0eb8d83740 1 -- 172.21.15.98:0/202582392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 msgr2=0x559f7ea0cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.549+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 0x559f7ea0cbd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0eb006ee30 tx=0x7f0ea40118d0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.549+0000 7f0eb8d83740 1 -- 172.21.15.98:0/202582392 shutdown_connections 2024-04-25T17:48:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.549+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 0x559f7ea0cbd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.549+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea0a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.549+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/202582392 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9ee4b0 0x559f7e9ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.549+0000 7f0eb8d83740 1 -- 172.21.15.98:0/202582392 >> 172.21.15.98:0/202582392 conn(0x559f7e9eab00 msgr2=0x559f7e9eaee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.549+0000 7f0eb8d83740 1 -- 172.21.15.98:0/202582392 shutdown_connections 2024-04-25T17:48:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.549+0000 7f0eb8d83740 1 -- 172.21.15.98:0/202582392 wait complete. 2024-04-25T17:48:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.549+0000 7f0eb8d83740 1 -- start start 2024-04-25T17:48:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.550+0000 7f0eb8d83740 1 Processor -- start 2024-04-25T17:48:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9ee4b0 0x559f7ea0a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9eedd0 0x559f7ea07400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 0x559f7ea07940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eaf7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9ee4b0 0x559f7ea0a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eaf7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9ee4b0 0x559f7ea0a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53116/0 (socket says 172.21.15.98:53116) 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eb8d83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559f7ea0d3b0 con 0x559f7e9eedd0 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eaeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9eedd0 0x559f7ea07400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eaeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9eedd0 0x559f7ea07400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46630/0 (socket says 172.21.15.98:46630) 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eaeffd640 1 -- 172.21.15.98:0/1622061693 learned_addr learned my addr 172.21.15.98:0/1622061693 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eb8d83740 1 -- 172.21.15.98:0/1622061693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559f7e8ee0b0 con 0x559f7ea0a7e0 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eb8d83740 1 -- 172.21.15.98:0/1622061693 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559f7ea0d530 con 0x559f7e9ee4b0 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eb8d83740 10 client.0 _ll_register_callbacks cb 0x559f7e8dbd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.551+0000 7f0eaffff640 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 0x559f7ea07940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eaeffd640 1 -- 172.21.15.98:0/1622061693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9ee4b0 msgr2=0x559f7ea0a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eaeffd640 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9ee4b0 0x559f7ea0a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eaeffd640 1 -- 172.21.15.98:0/1622061693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 msgr2=0x559f7ea07940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eaeffd640 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 0x559f7ea07940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eaeffd640 1 -- 172.21.15.98:0/1622061693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f7e732620 con 0x559f7e9eedd0 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eaf7fe640 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9ee4b0 0x559f7ea0a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eaffff640 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 0x559f7ea07940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eaeffd640 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9eedd0 0x559f7ea07400 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0ea401b040 tx=0x7f0ea4004420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83920e8bc5abe953 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0e8effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0e8effd640 1 -- 172.21.15.98:0/1622061693 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4029070 con 0x559f7e9eedd0 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0e8effd640 1 -- 172.21.15.98:0/1622061693 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f0ea4003b80 con 0x559f7e9eedd0 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eb8d83740 1 -- 172.21.15.98:0/1622061693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ea40044b0 con 0x559f7e9eedd0 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0e8effd640 1 -- 172.21.15.98:0/1622061693 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f0ea4035050 con 0x559f7e9eedd0 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eb8d83740 10 client.16686 fetch_fsmap learned FSMap version 48 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.552+0000 7f0eb8d83740 1 -- 172.21.15.98:0/1622061693 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x559f7ea192c0 con 0x559f7e9eedd0 2024-04-25T17:48:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.553+0000 7f0e8effd640 1 -- 172.21.15.98:0/1622061693 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ea40044b0 con 0x559f7e9eedd0 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.553+0000 7f0e8effd640 1 -- 172.21.15.98:0/1622061693 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 48) ==== 8346+0+0 (secure 0 0 0) 0x7f0ea400f510 con 0x559f7e9eedd0 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.553+0000 7f0eb8d83740 10 client.16686 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.553+0000 7f0eb8d83740 10 client.16686 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.553+0000 7f0eb8d83740 20 client.16686 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.553+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559f7ea1a670 0x559f7ea1ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.553+0000 7f0eb8d83740 4 client.16686 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.553+0000 7f0eb8d83740 1 -- 172.21.15.98:0/1622061693 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f7e961390 con 0x559f7ea1a670 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.553+0000 7f0eaf7fe640 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559f7ea1a670 0x559f7ea1ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.554+0000 7f0eaf7fe640 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559f7ea1a670 0x559f7ea1ca60 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f0d52d08b3cadfc server_cookie=6c29d362ab44392a in_seq=0 out_seq=0 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.554+0000 7f0e8effd640 10 client.16686 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.554+0000 7f0e8effd640 1 -- 172.21.15.98:0/1622061693 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f7e961390 con 0x559f7ea1a670 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0e8effd640 10 client.16686 handle_command_reply: tid=0 2024-04-25T17:48:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 client.16686 shutdown 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 -- 172.21.15.98:0/1622061693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9eedd0 msgr2=0x559f7ea07400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9eedd0 0x559f7ea07400 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0ea401b040 tx=0x7f0ea4004420 comp rx=0 tx=0).stop 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 -- 172.21.15.98:0/1622061693 shutdown_connections 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559f7ea1a670 0x559f7ea1ca60 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7ea0a7e0 0x559f7ea07940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7e9eedd0 0x559f7ea07400 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0e8effd640 0 client.16686 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/1622061693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9ee4b0 0x559f7ea0a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 -- 172.21.15.98:0/1622061693 >> 172.21.15.98:0/1622061693 conn(0x559f7e9eab00 msgr2=0x559f7e9f08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 -- 172.21.15.98:0/1622061693 shutdown_connections 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 1 -- 172.21.15.98:0/1622061693 wait complete. 2024-04-25T17:48:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.555+0000 7f0eb8d83740 20 client.16686 trim_cache size 0 max 16384 2024-04-25T17:48:36.583 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.577+0000 7f0eb8d83740 1 Processor -- start 2024-04-25T17:48:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.577+0000 7f0eb8d83740 1 -- start start 2024-04-25T17:48:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.577+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7e9ee4b0 0x559f7ea162e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.578+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea02ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.578+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea09c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.578+0000 7f0eb8d83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb01b6820 con 0x559f7ea0a7e0 2024-04-25T17:48:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.578+0000 7f0eb8d83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb01b4090 con 0x559f7e9ee4b0 2024-04-25T17:48:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.578+0000 7f0eb8d83740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb010c8e0 con 0x559f7e9eedd0 2024-04-25T17:48:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.578+0000 7f0eaffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea09c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.578+0000 7f0eaffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea09c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46642/0 (socket says 172.21.15.98:46642) 2024-04-25T17:48:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.578+0000 7f0eaffff640 1 -- 172.21.15.98:0/4016328755 learned_addr learned my addr 172.21.15.98:0/4016328755 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.579+0000 7f0eaffff640 1 -- 172.21.15.98:0/4016328755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 msgr2=0x559f7ea02ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T17:48:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.579+0000 7f0eaffff640 1 --2- 172.21.15.98:0/4016328755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea02ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.579+0000 7f0eaffff640 1 -- 172.21.15.98:0/4016328755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7e9ee4b0 msgr2=0x559f7ea162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.579+0000 7f0eaffff640 1 --2- 172.21.15.98:0/4016328755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7e9ee4b0 0x559f7ea162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.579+0000 7f0eaffff640 1 -- 172.21.15.98:0/4016328755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f7ea063b0 con 0x559f7ea0a7e0 2024-04-25T17:48:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.579+0000 7f0eaffff640 1 --2- 172.21.15.98:0/4016328755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea09c40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8004140 tx=0x7f0ea8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd45a31df8d0b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.584+0000 7f0e957fa640 1 -- 172.21.15.98:0/4016328755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea800f7c0 con 0x559f7ea0a7e0 2024-04-25T17:48:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.584+0000 7f0e957fa640 1 -- 172.21.15.98:0/4016328755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ea800fe10 con 0x559f7ea0a7e0 2024-04-25T17:48:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.585+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4016328755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 msgr2=0x559f7ea09c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.585+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/4016328755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea09c40 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8004140 tx=0x7f0ea8017000 comp rx=0 tx=0).stop 2024-04-25T17:48:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 725.0 s, 2390.5 tps, lat 42.257 ms stddev 49.325 2024-04-25T17:48:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.594+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4016328755 shutdown_connections 2024-04-25T17:48:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.594+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/4016328755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea09c40 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.594+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/4016328755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea02ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.594+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/4016328755 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7e9ee4b0 0x559f7ea162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.594+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4016328755 >> 172.21.15.98:0/4016328755 conn(0x559f7e9eab00 msgr2=0x559f7e9eaee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.597+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4016328755 shutdown_connections 2024-04-25T17:48:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.600+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4016328755 wait complete. 2024-04-25T17:48:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.600+0000 7f0eb8d83740 1 -- start start 2024-04-25T17:48:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.600+0000 7f0eb8d83740 1 Processor -- start 2024-04-25T17:48:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.601+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7e9ee4b0 0x559f7ea12890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.601+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea0ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.601+0000 7f0eb8d83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea0e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.601+0000 7f0eb8d83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb010cf30 con 0x559f7ea0a7e0 2024-04-25T17:48:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.601+0000 7f0eb8d83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb010e3a0 con 0x559f7e9ee4b0 2024-04-25T17:48:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.601+0000 7f0eb8d83740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb010d600 con 0x559f7e9eedd0 2024-04-25T17:48:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.601+0000 7f0eb8d83740 10 client.0 _ll_register_callbacks cb 0x7f0ea4004790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.601+0000 7f0eaeffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea0ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.601+0000 7f0eaeffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea0ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53132/0 (socket says 172.21.15.98:53132) 2024-04-25T17:48:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.601+0000 7f0eaeffd640 1 -- 172.21.15.98:0/4167257273 learned_addr learned my addr 172.21.15.98:0/4167257273 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.602+0000 7f0eaffff640 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea0e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.602+0000 7f0eaffff640 1 -- 172.21.15.98:0/4167257273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 msgr2=0x559f7ea0ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.602+0000 7f0eaffff640 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea0ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.602+0000 7f0eaffff640 1 -- 172.21.15.98:0/4167257273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7e9ee4b0 msgr2=0x559f7ea12890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T17:48:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.602+0000 7f0eaffff640 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7e9ee4b0 0x559f7ea12890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.602+0000 7f0eaffff640 1 -- 172.21.15.98:0/4167257273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f7ea0e830 con 0x559f7ea0a7e0 2024-04-25T17:48:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.603+0000 7f0eaffff640 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea0e2f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8019040 tx=0x7f0ea8005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8dcbcbd44660008 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.603+0000 7f0e8effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:48:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.603+0000 7f0e8effd640 1 -- 172.21.15.98:0/4167257273 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea805a060 con 0x559f7ea0a7e0 2024-04-25T17:48:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.604+0000 7f0e8effd640 1 -- 172.21.15.98:0/4167257273 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f0ea801f630 con 0x559f7ea0a7e0 2024-04-25T17:48:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.604+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4167257273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ea800fe10 con 0x559f7ea0a7e0 2024-04-25T17:48:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.604+0000 7f0e8effd640 1 -- 172.21.15.98:0/4167257273 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f0ea8051040 con 0x559f7ea0a7e0 2024-04-25T17:48:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.604+0000 7f0eb8d83740 10 client.16692 fetch_fsmap learned FSMap version 48 2024-04-25T17:48:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.604+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4167257273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x559f7ea807e0 con 0x559f7ea0a7e0 2024-04-25T17:48:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.605+0000 7f0e8effd640 1 -- 172.21.15.98:0/4167257273 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ea800fe10 con 0x559f7ea0a7e0 2024-04-25T17:48:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.605+0000 7f0e8effd640 1 -- 172.21.15.98:0/4167257273 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 48) ==== 8346+0+0 (secure 0 0 0) 0x7f0ea8054070 con 0x559f7ea0a7e0 2024-04-25T17:48:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.605+0000 7f0eb8d83740 10 client.16692 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T17:48:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.605+0000 7f0eb8d83740 10 client.16692 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:48:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.605+0000 7f0eb8d83740 20 client.16692 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.605+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559f7ea1a670 0x559f7ea82ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.605+0000 7f0eb8d83740 4 client.16692 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:48:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.605+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4167257273 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559f7e95b200 con 0x559f7ea1a670 2024-04-25T17:48:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.606+0000 7f0eaf7fe640 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559f7ea1a670 0x559f7ea82ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.607+0000 7f0eaf7fe640 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559f7ea1a670 0x559f7ea82ed0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b221e002d390642 server_cookie=9869a1009bee3386 in_seq=0 out_seq=0 2024-04-25T17:48:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.607+0000 7f0e8effd640 10 client.16692 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:36.623 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.624+0000 7f0e8effd640 1 -- 172.21.15.98:0/4167257273 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559f7e95b200 con 0x559f7ea1a670 2024-04-25T17:48:36.623 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.624+0000 7f0e8effd640 10 client.16692 handle_command_reply: tid=0 2024-04-25T17:48:36.623 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.624+0000 7f0eb8d83740 1 client.16692 shutdown 2024-04-25T17:48:36.624 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4167257273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 msgr2=0x559f7ea0e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea0e2f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8019040 tx=0x7f0ea8005bd0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4167257273 shutdown_connections 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x559f7ea1a670 0x559f7ea82ed0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559f7ea0a7e0 0x559f7ea0e2f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559f7e9eedd0 0x559f7ea0ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 1 --2- 172.21.15.98:0/4167257273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559f7e9ee4b0 0x559f7ea12890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4167257273 >> 172.21.15.98:0/4167257273 conn(0x559f7e9eab00 msgr2=0x559f7e9e93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0e8effd640 0 client.16692 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4167257273 shutdown_connections 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 1 -- 172.21.15.98:0/4167257273 wait complete. 2024-04-25T17:48:36.625 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb8d83740 20 client.16692 trim_cache size 0 max 16384 2024-04-25T17:48:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:48:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:48:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:48:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:48:36.626 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0e980a3e40 msgr2=0x7f0e980a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb5f95640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0e980a3e40 0x7f0e980a6300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8009fd0 tx=0x7f0ea8009500 comp rx=0 tx=0).stop 2024-04-25T17:48:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb00742f0 msgr2=0x7f0eb0073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb5f95640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb00742f0 0x7f0eb0073570 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009850 tx=0x7f0ea0030650 comp rx=0 tx=0).stop 2024-04-25T17:48:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 shutdown_connections 2024-04-25T17:48:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb5f95640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0e980a3e40 0x7f0e980a6300 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb5f95640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0eb0112f40 0x7f0eb006d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb5f95640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0eb0074c10 0x7f0eb0073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb5f95640 1 --2- 172.21.15.98:0/2450496300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0eb00742f0 0x7f0eb0073570 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.625+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 >> 172.21.15.98:0/2450496300 conn(0x7f0eb00fda00 msgr2=0x7f0eb00fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.626+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 shutdown_connections 2024-04-25T17:48:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:36.626+0000 7f0eb5f95640 1 -- 172.21.15.98:0/2450496300 wait complete. 2024-04-25T17:48:36.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:36 smithi098 ceph-mon[29192]: pgmap v399: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 4.9 MiB/s wr, 527 op/s 2024-04-25T17:48:36.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:36 smithi203 ceph-mon[32834]: pgmap v399: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 4.9 MiB/s wr, 527 op/s 2024-04-25T17:48:36.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:36 smithi173 ceph-mon[32856]: pgmap v399: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 4.9 MiB/s wr, 527 op/s 2024-04-25T17:48:38.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:38 smithi098 ceph-mon[29192]: pgmap v400: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 7.5 MiB/s wr, 800 op/s 2024-04-25T17:48:38.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:38 smithi203 ceph-mon[32834]: pgmap v400: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 7.5 MiB/s wr, 800 op/s 2024-04-25T17:48:38.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:38 smithi173 ceph-mon[32856]: pgmap v400: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 7.5 MiB/s wr, 800 op/s 2024-04-25T17:48:39.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:39 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:48:39.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:39 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:48:39.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:39 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:48:40.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:40 smithi098 ceph-mon[29192]: pgmap v401: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T17:48:40.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:40 smithi203 ceph-mon[32834]: pgmap v401: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T17:48:40.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:40 smithi173 ceph-mon[32856]: pgmap v401: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T17:48:41.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 730.0 s, 1924.5 tps, lat 52.000 ms stddev 56.888 2024-04-25T17:48:41.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:41 smithi098 ceph-mon[29192]: pgmap v402: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T17:48:41.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:41 smithi203 ceph-mon[32834]: pgmap v402: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T17:48:41.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:41 smithi173 ceph-mon[32856]: pgmap v402: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T17:48:43.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:43 smithi098 ceph-mon[29192]: pgmap v403: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 7.8 MiB/s wr, 827 op/s 2024-04-25T17:48:43.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:43 smithi203 ceph-mon[32834]: pgmap v403: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 7.8 MiB/s wr, 827 op/s 2024-04-25T17:48:44.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:43 smithi173 ceph-mon[32856]: pgmap v403: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 7.8 MiB/s wr, 827 op/s 2024-04-25T17:48:44.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:44 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:48:44.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:44 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:48:45.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:44 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:48:45.493 INFO:tasks.quiescer.fs.[cephfs]:Total duration (58.2) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-25T17:48:45.493 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.1 2024-04-25T17:48:45.494 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:48:45.838 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:48:45.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:45 smithi098 ceph-mon[29192]: pgmap v404: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 5.3 MiB/s wr, 562 op/s 2024-04-25T17:48:45.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:45 smithi203 ceph-mon[32834]: pgmap v404: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 5.3 MiB/s wr, 562 op/s 2024-04-25T17:48:46.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:45 smithi173 ceph-mon[32856]: pgmap v404: 97 pgs: 97 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 5.3 MiB/s wr, 562 op/s 2024-04-25T17:48:46.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 735.0 s, 2341.8 tps, lat 42.223 ms stddev 48.744 2024-04-25T17:48:47.871 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:47 smithi098 ceph-mon[29192]: pgmap v405: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.9 MiB/s wr, 829 op/s 2024-04-25T17:48:47.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:47 smithi203 ceph-mon[32834]: pgmap v405: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.9 MiB/s wr, 829 op/s 2024-04-25T17:48:48.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:47 smithi173 ceph-mon[32856]: pgmap v405: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 7.9 MiB/s wr, 829 op/s 2024-04-25T17:48:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.561+0000 7f39c7c4c700 1 -- 172.21.15.98:0/4238834546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39c0102ac0 msgr2=0x7f39c0102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.561+0000 7f39c7c4c700 1 --2- 172.21.15.98:0/4238834546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39c0102ac0 0x7f39c0102f40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f39b00098f0 tx=0x7f39b0009c00 comp rx=0 tx=0).stop 2024-04-25T17:48:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.561+0000 7f39c7c4c700 1 -- 172.21.15.98:0/4238834546 shutdown_connections 2024-04-25T17:48:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.561+0000 7f39c7c4c700 1 --2- 172.21.15.98:0/4238834546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39c0103480 0x7f39c010f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.561+0000 7f39c7c4c700 1 --2- 172.21.15.98:0/4238834546 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0108a70 0x7f39c0108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.561+0000 7f39c7c4c700 1 --2- 172.21.15.98:0/4238834546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39c0102ac0 0x7f39c0102f40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.561+0000 7f39c7c4c700 1 -- 172.21.15.98:0/4238834546 >> 172.21.15.98:0/4238834546 conn(0x7f39c0079bb0 msgr2=0x7f39c0079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.561+0000 7f39c7c4c700 1 -- 172.21.15.98:0/4238834546 shutdown_connections 2024-04-25T17:48:48.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.561+0000 7f39c7c4c700 1 -- 172.21.15.98:0/4238834546 wait complete. 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.562+0000 7f39c7c4c700 1 Processor -- start 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.562+0000 7f39c7c4c700 1 -- start start 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.562+0000 7f39c7c4c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0102ac0 0x7f39c0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.562+0000 7f39c7c4c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39c0103480 0x7f39c0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.562+0000 7f39c7c4c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39c0108a70 0x7f39c00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c7c4c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39c0112d40 con 0x7f39c0102ac0 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c7c4c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f39c0112bc0 con 0x7f39c0103480 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c7c4c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f39c0112ec0 con 0x7f39c0108a70 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c61e9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39c0108a70 0x7f39c00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c59e8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0102ac0 0x7f39c0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c59e8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0102ac0 0x7f39c0074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46158/0 (socket says 172.21.15.98:46158) 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c59e8700 1 -- 172.21.15.98:0/3376982769 learned_addr learned my addr 172.21.15.98:0/3376982769 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c51e7700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39c0103480 0x7f39c0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c61e9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39c0108a70 0x7f39c00757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:34338/0 (socket says 172.21.15.98:34338) 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c59e8700 1 -- 172.21.15.98:0/3376982769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39c0108a70 msgr2=0x7f39c00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c59e8700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39c0108a70 0x7f39c00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c59e8700 1 -- 172.21.15.98:0/3376982769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39c0103480 msgr2=0x7f39c0075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c59e8700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39c0103480 0x7f39c0075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.563+0000 7f39c59e8700 1 -- 172.21.15.98:0/3376982769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c01abbc0 con 0x7f39c0102ac0 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.564+0000 7f39c59e8700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0102ac0 0x7f39c0074d40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00ec10 tx=0x7f39bc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.564+0000 7f39b6ffd700 1 -- 172.21.15.98:0/3376982769 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00cc90 con 0x7f39c0102ac0 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.564+0000 7f39b6ffd700 1 -- 172.21.15.98:0/3376982769 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39bc00ce30 con 0x7f39c0102ac0 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.564+0000 7f39c7c4c700 1 -- 172.21.15.98:0/3376982769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c01abe50 con 0x7f39c0102ac0 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.564+0000 7f39b6ffd700 1 -- 172.21.15.98:0/3376982769 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc010430 con 0x7f39c0102ac0 2024-04-25T17:48:48.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.564+0000 7f39c7c4c700 1 -- 172.21.15.98:0/3376982769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c01ac250 con 0x7f39c0102ac0 2024-04-25T17:48:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.565+0000 7f39c7c4c700 1 -- 172.21.15.98:0/3376982769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c01041a0 con 0x7f39c0102ac0 2024-04-25T17:48:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.567+0000 7f39b6ffd700 1 -- 172.21.15.98:0/3376982769 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f39bc004000 con 0x7f39c0102ac0 2024-04-25T17:48:48.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.568+0000 7f39b6ffd700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f39ac0a3e70 0x7f39ac0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.569+0000 7f39b6ffd700 1 -- 172.21.15.98:0/3376982769 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f39bc0d0c30 con 0x7f39c0102ac0 2024-04-25T17:48:48.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.569+0000 7f39c51e7700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f39ac0a3e70 0x7f39ac0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:48.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.570+0000 7f39c51e7700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f39ac0a3e70 0x7f39ac0a6330 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009810 tx=0x7f39b0019040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:48.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.575+0000 7f39b6ffd700 1 -- 172.21.15.98:0/3376982769 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39bc09d270 con 0x7f39c0102ac0 2024-04-25T17:48:48.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.985+0000 7f39c7c4c700 1 -- 172.21.15.98:0/3376982769 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39c002df80 con 0x7f39c0102ac0 2024-04-25T17:48:48.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.988+0000 7f39b6ffd700 1 -- 172.21.15.98:0/3376982769 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+11877 (secure 0 0 0) 0x7f39bc09cb70 con 0x7f39c0102ac0 2024-04-25T17:48:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:48:48.989 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:48:42.740777+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26026":{"gid":26026,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/3925531495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":3925531495},{"type":"v1","addr":"172.21.15.203:6837","nonce":3925531495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:48:48.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 -- 172.21.15.98:0/3376982769 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f39ac0a3e70 msgr2=0x7f39ac0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:48.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f39ac0a3e70 0x7f39ac0a6330 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009810 tx=0x7f39b0019040 comp rx=0 tx=0).stop 2024-04-25T17:48:48.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 -- 172.21.15.98:0/3376982769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0102ac0 msgr2=0x7f39c0074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:48.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0102ac0 0x7f39c0074d40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f39bc00ec10 tx=0x7f39bc00ef20 comp rx=0 tx=0).stop 2024-04-25T17:48:48.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 -- 172.21.15.98:0/3376982769 shutdown_connections 2024-04-25T17:48:48.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f39ac0a3e70 0x7f39ac0a6330 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:48.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39c0108a70 0x7f39c00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39c0102ac0 0x7f39c0074d40 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 --2- 172.21.15.98:0/3376982769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39c0103480 0x7f39c0075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 -- 172.21.15.98:0/3376982769 >> 172.21.15.98:0/3376982769 conn(0x7f39c0079bb0 msgr2=0x7f39c0101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 -- 172.21.15.98:0/3376982769 shutdown_connections 2024-04-25T17:48:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:48.991+0000 7f39c7c4c700 1 -- 172.21.15.98:0/3376982769 wait complete. 2024-04-25T17:48:48.992 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 50 2024-04-25T17:48:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.759+0000 7f8bf92cf640 1 -- 172.21.15.98:0/844936564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4106680 msgr2=0x7f8bf4106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.759+0000 7f8bf92cf640 1 --2- 172.21.15.98:0/844936564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4106680 0x7f8bf4106a60 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009900 tx=0x7f8be402a410 comp rx=0 tx=0).stop 2024-04-25T17:48:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.759+0000 7f8bf92cf640 1 -- 172.21.15.98:0/844936564 shutdown_connections 2024-04-25T17:48:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.759+0000 7f8bf92cf640 1 --2- 172.21.15.98:0/844936564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bf4100f80 0x7f8bf410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.759+0000 7f8bf92cf640 1 --2- 172.21.15.98:0/844936564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf41005c0 0x7f8bf4100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.759+0000 7f8bf92cf640 1 --2- 172.21.15.98:0/844936564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4106680 0x7f8bf4106a60 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.759+0000 7f8bf92cf640 1 -- 172.21.15.98:0/844936564 >> 172.21.15.98:0/844936564 conn(0x7f8bf40773e0 msgr2=0x7f8bf40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.759+0000 7f8bf92cf640 1 -- 172.21.15.98:0/844936564 shutdown_connections 2024-04-25T17:48:49.760 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.759+0000 7f8bf92cf640 1 -- 172.21.15.98:0/844936564 wait complete. 2024-04-25T17:48:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.760+0000 7f8bf92cf640 1 Processor -- start 2024-04-25T17:48:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf92cf640 1 -- start start 2024-04-25T17:48:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf92cf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf41005c0 0x7f8bf410e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf92cf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bf4100f80 0x7f8bf410e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf92cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4106680 0x7f8bf410eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf92cf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf4112200 con 0x7f8bf4106680 2024-04-25T17:48:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf92cf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf4112080 con 0x7f8bf4100f80 2024-04-25T17:48:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf92cf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf4112380 con 0x7f8bf41005c0 2024-04-25T17:48:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4106680 0x7f8bf410eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4106680 0x7f8bf410eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46166/0 (socket says 172.21.15.98:46166) 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf37fe640 1 -- 172.21.15.98:0/1987015167 learned_addr learned my addr 172.21.15.98:0/1987015167 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.761+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bf4100f80 0x7f8bf410e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf41005c0 0x7f8bf410e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bf37fe640 1 -- 172.21.15.98:0/1987015167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf41005c0 msgr2=0x7f8bf410e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bf37fe640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf41005c0 0x7f8bf410e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bf37fe640 1 -- 172.21.15.98:0/1987015167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bf4100f80 msgr2=0x7f8bf410e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bf37fe640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bf4100f80 0x7f8bf410e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bf37fe640 1 -- 172.21.15.98:0/1987015167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf41a6ae0 con 0x7f8bf4106680 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf41005c0 0x7f8bf410e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bf37fe640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4106680 0x7f8bf410eee0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00ec60 tx=0x7f8bec00c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bdbfff640 1 -- 172.21.15.98:0/1987015167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bec004d20 con 0x7f8bf4106680 2024-04-25T17:48:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bdbfff640 1 -- 172.21.15.98:0/1987015167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bec00ba20 con 0x7f8bf4106680 2024-04-25T17:48:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bf92cf640 1 -- 172.21.15.98:0/1987015167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf41a6dd0 con 0x7f8bf4106680 2024-04-25T17:48:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bdbfff640 1 -- 172.21.15.98:0/1987015167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bec0056c0 con 0x7f8bf4106680 2024-04-25T17:48:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.762+0000 7f8bf92cf640 1 -- 172.21.15.98:0/1987015167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf41a72a0 con 0x7f8bf4106680 2024-04-25T17:48:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.765+0000 7f8bfbfe7740 1 Processor -- start 2024-04-25T17:48:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.765+0000 7f8bdbfff640 1 -- 172.21.15.98:0/1987015167 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8bec005860 con 0x7f8bf4106680 2024-04-25T17:48:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.765+0000 7f8bfbfe7740 1 -- start start 2024-04-25T17:48:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.765+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x56216701ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.765+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 0x5621670292c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.765+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562167029800 0x56216702bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.765+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562166f83970 con 0x56216701ef40 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.765+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562166f204a0 con 0x562167029800 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.765+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562166f7f930 con 0x56216701e620 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 0x5621670292c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 0x5621670292c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46174/0 (socket says 172.21.15.98:46174) 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/203679912 learned_addr learned my addr 172.21.15.98:0/203679912 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/203679912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x56216701ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bdbfff640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8bdc0a3dc0 0x7f8bdc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/203679912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562167029800 0x56216702bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bdbfff640 1 -- 172.21.15.98:0/1987015167 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f8bec0d0cc0 con 0x7f8bf4106680 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/203679912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 msgr2=0x56216701ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/203679912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x56216701ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/203679912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562167029800 msgr2=0x56216702bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf37fe640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8bdc0a3dc0 0x7f8bdc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/203679912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562167029800 0x56216702bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/203679912 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562166c96210 con 0x56216701ef40 2024-04-25T17:48:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/203679912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x56216701ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:48:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/203679912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 0x5621670292c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8be4000c00 tx=0x7f8be4032be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51715c088cde4e18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.766+0000 7f8bf37fe640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8bdc0a3dc0 0x7f8bdc0a6280 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0995d0 tx=0x7f8bec00ce50 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.767+0000 7f8bd97fa640 1 -- 172.21.15.98:0/203679912 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be403f070 con 0x56216701ef40 2024-04-25T17:48:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.767+0000 7f8bd97fa640 1 -- 172.21.15.98:0/203679912 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8be4004030 con 0x56216701ef40 2024-04-25T17:48:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.767+0000 7f8bd97fa640 1 -- 172.21.15.98:0/203679912 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4035070 con 0x56216701ef40 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/203679912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 msgr2=0x5621670292c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/203679912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 0x5621670292c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8be4000c00 tx=0x7f8be4032be0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/203679912 shutdown_connections 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/203679912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562167029800 0x56216702bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/203679912 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 0x5621670292c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/203679912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x56216701ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/203679912 >> 172.21.15.98:0/203679912 conn(0x56216701ab30 msgr2=0x56216701af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/203679912 shutdown_connections 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/203679912 wait complete. 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 -- start start 2024-04-25T17:48:49.773 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 Processor -- start 2024-04-25T17:48:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x5621670290c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ef40 0x5621670263b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562167029800 0x5621670268f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56216702c3d0 con 0x562167029800 2024-04-25T17:48:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562166f1e0b0 con 0x56216701ef40 2024-04-25T17:48:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56216702c550 con 0x56216701e620 2024-04-25T17:48:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.768+0000 7f8bfbfe7740 10 client.0 _ll_register_callbacks cb 0x562166f0bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562167029800 0x5621670268f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.774 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562167029800 0x5621670268f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46178/0 (socket says 172.21.15.98:46178) 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf27fc640 1 -- 172.21.15.98:0/4290208656 learned_addr learned my addr 172.21.15.98:0/4290208656 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x5621670290c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ef40 0x5621670263b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf27fc640 1 -- 172.21.15.98:0/4290208656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 msgr2=0x5621670290c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x5621670290c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf27fc640 1 -- 172.21.15.98:0/4290208656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ef40 msgr2=0x5621670263b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ef40 0x5621670263b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf27fc640 1 -- 172.21.15.98:0/4290208656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562166d62620 con 0x562167029800 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ef40 0x5621670263b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562167029800 0x5621670268f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8be8005fd0 tx=0x7f8be8016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a6be8b21b7f0d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bd6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/4290208656 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8003e10 con 0x562167029800 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/4290208656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8be4004030 con 0x562167029800 2024-04-25T17:48:49.775 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/4290208656 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f8be80109e0 con 0x562167029800 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/4290208656 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f8be8010be0 con 0x562167029800 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bfbfe7740 10 client.16710 fetch_fsmap learned FSMap version 50 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.769+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/4290208656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x562167038320 con 0x562167029800 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/4290208656 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8be4004030 con 0x562167029800 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/4290208656 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 50) ==== 8338+0+0 (secure 0 0 0) 0x7f8be80050b0 con 0x562167029800 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bfbfe7740 10 client.16710 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bfbfe7740 10 client.16710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bfbfe7740 20 client.16710 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562167039790 0x56216703bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bfbfe7740 4 client.16710 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/4290208656 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562166f91390 con 0x562167039790 2024-04-25T17:48:49.776 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562167039790 0x56216703bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bd6ff5640 10 client.16710 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.770+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562167039790 0x56216703bb80 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37c299e132ff9c2e server_cookie=76be19f91a3faf05 in_seq=0 out_seq=0 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.772+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/4290208656 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562166f91390 con 0x562167039790 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.772+0000 7f8bd6ff5640 10 client.16710 handle_command_reply: tid=0 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.772+0000 7f8bfbfe7740 1 client.16710 shutdown 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/4290208656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562167029800 msgr2=0x5621670268f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562167029800 0x5621670268f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8be8005fd0 tx=0x7f8be8016040 comp rx=0 tx=0).stop 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/4290208656 shutdown_connections 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562167039790 0x56216703bb80 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bd6ff5640 0 client.16710 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562167029800 0x5621670268f0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ef40 0x5621670263b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/4290208656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x5621670290c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/4290208656 >> 172.21.15.98:0/4290208656 conn(0x56216701ab30 msgr2=0x562167020860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:49.777 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/4290208656 shutdown_connections 2024-04-25T17:48:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/4290208656 wait complete. 2024-04-25T17:48:49.778 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.773+0000 7f8bfbfe7740 20 client.16710 trim_cache size 0 max 16384 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bfbfe7740 1 Processor -- start 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bfbfe7740 1 -- start start 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ab30 0x5621670352c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701e620 0x562167019b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701ef40 0x562167028bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf41b2610 con 0x56216701e620 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf41afee0 con 0x56216701ab30 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf41add50 con 0x56216701ef40 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bf2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701e620 0x562167019b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bf2ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701e620 0x562167019b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46190/0 (socket says 172.21.15.98:46190) 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.795+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/3197907951 learned_addr learned my addr 172.21.15.98:0/3197907951 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/3197907951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ab30 0x5621670352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/3197907951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701ef40 0x562167028bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/3197907951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701ef40 msgr2=0x562167028bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/3197907951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701ef40 0x562167028bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/3197907951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ab30 msgr2=0x5621670352c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/3197907951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ab30 0x5621670352c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/3197907951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562167024920 con 0x56216701e620 2024-04-25T17:48:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/3197907951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701e620 0x562167019b30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8be402a450 tx=0x7f8be40051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f497a803ced46a2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bd8ff9640 1 -- 172.21.15.98:0/3197907951 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4035040 con 0x56216701e620 2024-04-25T17:48:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bd8ff9640 1 -- 172.21.15.98:0/3197907951 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8be4053070 con 0x56216701e620 2024-04-25T17:48:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.796+0000 7f8bd8ff9640 1 -- 172.21.15.98:0/3197907951 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be403f040 con 0x56216701e620 2024-04-25T17:48:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.797+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/3197907951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701e620 msgr2=0x562167019b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.797+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/3197907951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701e620 0x562167019b30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8be402a450 tx=0x7f8be40051a0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.797+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/3197907951 shutdown_connections 2024-04-25T17:48:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.798+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/3197907951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701ef40 0x562167028bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.798+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/3197907951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701e620 0x562167019b30 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.798+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/3197907951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ab30 0x5621670352c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.798+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/3197907951 >> 172.21.15.98:0/3197907951 conn(0x562167026a50 msgr2=0x562167019420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.798+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/3197907951 shutdown_connections 2024-04-25T17:48:49.801 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.798+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/3197907951 wait complete. 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.798+0000 7f8bfbfe7740 1 -- start start 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bfbfe7740 1 Processor -- start 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ab30 0x562167023100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x562167023640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bfbfe7740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 0x562167023b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bf2ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x562167023640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf41ae3a0 con 0x56216701ef40 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf41af810 con 0x56216701ab30 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bf2ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x562167023640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:34388/0 (socket says 172.21.15.98:34388) 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bfbfe7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf41aea70 con 0x56216701e620 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/186700135 learned_addr learned my addr 172.21.15.98:0/186700135 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ab30 0x562167023100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.799+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 0x562167023b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bfbfe7740 10 client.0 _ll_register_callbacks cb 0x7f8be80036c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/186700135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ab30 msgr2=0x562167023100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ab30 0x562167023100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/186700135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 msgr2=0x562167023b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 0x562167023b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/186700135 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56216709edb0 con 0x56216701e620 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x562167023640 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009900 tx=0x7f8be4005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b5aa4576a4dfe28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bd6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/186700135 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be405b070 con 0x56216701e620 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/186700135 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f8be404e410 con 0x56216701e620 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/186700135 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8be4053070 con 0x56216701e620 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.800+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/186700135 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f8be4067050 con 0x56216701e620 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bfbfe7740 10 client.26182 fetch_fsmap learned FSMap version 50 2024-04-25T17:48:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/186700135 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5621670a00a0 con 0x56216701e620 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/186700135 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8be4053070 con 0x56216701e620 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/186700135 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 50) ==== 8338+0+0 (secure 0 0 0) 0x7f8be404d070 con 0x56216701e620 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bfbfe7740 10 client.26182 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bfbfe7740 10 client.26182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bfbfe7740 20 client.26182 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562167029800 0x5621670a2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bfbfe7740 4 client.26182 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/186700135 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562166f8b200 con 0x562167029800 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562167029800 0x5621670a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.801+0000 7f8bd6ff5640 10 client.26182 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.802+0000 7f8bf27fc640 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562167029800 0x5621670a2380 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dee16533c40f80e server_cookie=9d1a77c9f4f56b15 in_seq=0 out_seq=0 2024-04-25T17:48:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bd6ff5640 1 -- 172.21.15.98:0/186700135 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562166f8b200 con 0x562167029800 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bd6ff5640 10 client.26182 handle_command_reply: tid=0 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 client.26182 shutdown 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/186700135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 msgr2=0x562167023640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x562167023640 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009900 tx=0x7f8be4005780 comp rx=0 tx=0).stop 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/186700135 shutdown_connections 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x562167029800 0x5621670a2380 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bd6ff5640 0 client.26182 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56216701ef40 0x562167023b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56216701e620 0x562167023640 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 --2- 172.21.15.98:0/186700135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56216701ab30 0x562167023100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/186700135 >> 172.21.15.98:0/186700135 conn(0x562167026a50 msgr2=0x562167020860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/186700135 shutdown_connections 2024-04-25T17:48:49.811 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.811+0000 7f8bfbfe7740 1 -- 172.21.15.98:0/186700135 wait complete. 2024-04-25T17:48:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.812+0000 7f8bfbfe7740 20 client.26182 trim_cache size 0 max 16384 2024-04-25T17:48:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:48:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:48:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:48:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:48:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 -- 172.21.15.98:0/1987015167 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8bdc0a3dc0 msgr2=0x7f8bdc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8bdc0a3dc0 0x7f8bdc0a6280 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0995d0 tx=0x7f8bec00ce50 comp rx=0 tx=0).stop 2024-04-25T17:48:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 -- 172.21.15.98:0/1987015167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4106680 msgr2=0x7f8bf410eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4106680 0x7f8bf410eee0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f8bec00ec60 tx=0x7f8bec00c7a0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 -- 172.21.15.98:0/1987015167 shutdown_connections 2024-04-25T17:48:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8bdc0a3dc0 0x7f8bdc0a6280 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bf4106680 0x7f8bf410eee0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bf4100f80 0x7f8bf410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 --2- 172.21.15.98:0/1987015167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bf41005c0 0x7f8bf410e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 -- 172.21.15.98:0/1987015167 >> 172.21.15.98:0/1987015167 conn(0x7f8bf40773e0 msgr2=0x7f8bf40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 -- 172.21.15.98:0/1987015167 shutdown_connections 2024-04-25T17:48:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:48:49.813+0000 7f8bf92cf640 1 -- 172.21.15.98:0/1987015167 wait complete. 2024-04-25T17:48:49.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:49 smithi098 ceph-mon[29192]: pgmap v406: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.3 MiB/s wr, 556 op/s 2024-04-25T17:48:49.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:49 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3376982769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:48:49.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:49 smithi203 ceph-mon[32834]: pgmap v406: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.3 MiB/s wr, 556 op/s 2024-04-25T17:48:49.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:49 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3376982769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:48:50.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:49 smithi173 ceph-mon[32856]: pgmap v406: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.3 MiB/s wr, 556 op/s 2024-04-25T17:48:50.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:49 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3376982769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:48:50.347 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 89.1 --await-for 42.0' 2024-04-25T17:48:50.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 89.1 --await-for 42.0 2024-04-25T17:48:50.755 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:48:51.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 740.0 s, 1644.0 tps, lat 61.044 ms stddev 78.541 2024-04-25T17:48:51.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:51 smithi098 ceph-mon[29192]: pgmap v407: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.3 MiB/s wr, 556 op/s 2024-04-25T17:48:51.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:51 smithi203 ceph-mon[32834]: pgmap v407: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.3 MiB/s wr, 556 op/s 2024-04-25T17:48:52.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:51 smithi173 ceph-mon[32856]: pgmap v407: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.3 MiB/s wr, 556 op/s 2024-04-25T17:48:53.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:53 smithi098 ceph-mon[29192]: pgmap v408: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 9.8 MiB/s wr, 880 op/s 2024-04-25T17:48:53.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:53 smithi203 ceph-mon[32834]: pgmap v408: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 9.8 MiB/s wr, 880 op/s 2024-04-25T17:48:54.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:53 smithi173 ceph-mon[32856]: pgmap v408: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 9.8 MiB/s wr, 880 op/s 2024-04-25T17:48:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.223+0000 7f98c0cca700 1 -- 172.21.15.98:0/4167680580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98bc107ed0 msgr2=0x7f98bc06b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:54.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.223+0000 7f98c0cca700 1 --2- 172.21.15.98:0/4167680580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98bc107ed0 0x7f98bc06b650 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0098f0 tx=0x7f98ac009c00 comp rx=0 tx=0).stop 2024-04-25T17:48:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.226+0000 7f98c0cca700 1 -- 172.21.15.98:0/4167680580 shutdown_connections 2024-04-25T17:48:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.226+0000 7f98c0cca700 1 --2- 172.21.15.98:0/4167680580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98bc1054d0 0x7f98bc105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.226+0000 7f98c0cca700 1 --2- 172.21.15.98:0/4167680580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98bc107a60 0x7f98bc06b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.226+0000 7f98c0cca700 1 --2- 172.21.15.98:0/4167680580 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98bc107ed0 0x7f98bc06b650 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.226+0000 7f98c0cca700 1 -- 172.21.15.98:0/4167680580 >> 172.21.15.98:0/4167680580 conn(0x7f98bc0788f0 msgr2=0x7f98bc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:54.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.228+0000 7f98c0cca700 1 -- 172.21.15.98:0/4167680580 shutdown_connections 2024-04-25T17:48:54.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.228+0000 7f98c0cca700 1 -- 172.21.15.98:0/4167680580 wait complete. 2024-04-25T17:48:54.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.229+0000 7f98c0cca700 1 Processor -- start 2024-04-25T17:48:54.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.234+0000 7f98c0cca700 1 -- start start 2024-04-25T17:48:54.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.234+0000 7f98c0cca700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98bc1054d0 0x7f98bc19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.234+0000 7f98c0cca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98bc107a60 0x7f98bc19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.234+0000 7f98c0cca700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98bc107ed0 0x7f98bc196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.234+0000 7f98c0cca700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc10aa40 con 0x7f98bc107a60 2024-04-25T17:48:54.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.234+0000 7f98c0cca700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc10a8c0 con 0x7f98bc1054d0 2024-04-25T17:48:54.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.234+0000 7f98c0cca700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc10abc0 con 0x7f98bc107ed0 2024-04-25T17:48:54.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.235+0000 7f98b9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98bc107a60 0x7f98bc19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.235+0000 7f98b9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98bc107a60 0x7f98bc19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59886/0 (socket says 172.21.15.98:59886) 2024-04-25T17:48:54.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.235+0000 7f98b9d9b700 1 -- 172.21.15.98:0/2411201355 learned_addr learned my addr 172.21.15.98:0/2411201355 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:54.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.235+0000 7f98bad9d700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98bc107ed0 0x7f98bc196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.235+0000 7f98ba59c700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98bc1054d0 0x7f98bc19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.236+0000 7f98b9d9b700 1 -- 172.21.15.98:0/2411201355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98bc107ed0 msgr2=0x7f98bc196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:54.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.236+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98bc107ed0 0x7f98bc196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.236+0000 7f98b9d9b700 1 -- 172.21.15.98:0/2411201355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98bc1054d0 msgr2=0x7f98bc19ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:54.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.236+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98bc1054d0 0x7f98bc19ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.236+0000 7f98b9d9b700 1 -- 172.21.15.98:0/2411201355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98bc1973b0 con 0x7f98bc107a60 2024-04-25T17:48:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.236+0000 7f98bad9d700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98bc107ed0 0x7f98bc196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:48:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.236+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98bc107a60 0x7f98bc19cfa0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b840 tx=0x7f98ac00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.237+0000 7f98ab7fe700 1 -- 172.21.15.98:0/2411201355 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac025070 con 0x7f98bc107a60 2024-04-25T17:48:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.237+0000 7f98ab7fe700 1 -- 172.21.15.98:0/2411201355 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ac00bc50 con 0x7f98bc107a60 2024-04-25T17:48:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.237+0000 7f98ab7fe700 1 -- 172.21.15.98:0/2411201355 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac017660 con 0x7f98bc107a60 2024-04-25T17:48:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.238+0000 7f98c0cca700 1 -- 172.21.15.98:0/2411201355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98bc197640 con 0x7f98bc107a60 2024-04-25T17:48:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.238+0000 7f98c0cca700 1 -- 172.21.15.98:0/2411201355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98bc06c460 con 0x7f98bc107a60 2024-04-25T17:48:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.241+0000 7f98ab7fe700 1 -- 172.21.15.98:0/2411201355 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f98ac028070 con 0x7f98bc107a60 2024-04-25T17:48:54.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.242+0000 7f98ab7fe700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f98a40a4080 0x7f98a40a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.242+0000 7f98ab7fe700 1 -- 172.21.15.98:0/2411201355 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f98ac0d13a0 con 0x7f98bc107a60 2024-04-25T17:48:54.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.243+0000 7f98cb553b80 1 Processor -- start 2024-04-25T17:48:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.243+0000 7f98ba59c700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f98a40a4080 0x7f98a40a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.243+0000 7f98cb553b80 1 -- start start 2024-04-25T17:48:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340afa80 0x55e5340afe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 0x55e5340bad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340bd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98cb553b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e534002ec0 con 0x55e5340b03a0 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98cb553b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e533fa9af0 con 0x55e5340afa80 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98cb553b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e533ffee80 con 0x55e5340bb270 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98bad9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340bd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98ba59c700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f98a40a4080 0x7f98a40a6540 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f98b401ab40 tx=0x7f98b401a040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98bad9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340bd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53978/0 (socket says 172.21.15.98:53978) 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98ba59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340afa80 0x55e5340afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98bad9d700 1 -- 172.21.15.98:0/2545763679 learned_addr learned my addr 172.21.15.98:0/2545763679 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/2545763679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 0x55e5340bad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98bad9d700 1 -- 172.21.15.98:0/2545763679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340afa80 msgr2=0x55e5340afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98bad9d700 1 --2- 172.21.15.98:0/2545763679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340afa80 0x55e5340afe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98bad9d700 1 -- 172.21.15.98:0/2545763679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 msgr2=0x55e5340bad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98bad9d700 1 --2- 172.21.15.98:0/2545763679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 0x55e5340bad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98bad9d700 1 -- 172.21.15.98:0/2545763679 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e533f50a60 con 0x55e5340bb270 2024-04-25T17:48:54.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.244+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/2545763679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 0x55e5340bad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:48:54.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.245+0000 7f98bad9d700 1 --2- 172.21.15.98:0/2545763679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340bd660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f98b000f4c0 tx=0x7f98b000f7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f65d292fe30546b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:54.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.245+0000 7f98a8ff9700 1 -- 172.21.15.98:0/2545763679 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b0004a30 con 0x55e5340bb270 2024-04-25T17:48:54.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.245+0000 7f98a8ff9700 1 -- 172.21.15.98:0/2545763679 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98b000fe50 con 0x55e5340bb270 2024-04-25T17:48:54.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.245+0000 7f98a8ff9700 1 -- 172.21.15.98:0/2545763679 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b0010930 con 0x55e5340bb270 2024-04-25T17:48:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 -- 172.21.15.98:0/2545763679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 msgr2=0x55e5340bd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 --2- 172.21.15.98:0/2545763679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340bd660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f98b000f4c0 tx=0x7f98b000f7d0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 -- 172.21.15.98:0/2545763679 shutdown_connections 2024-04-25T17:48:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 --2- 172.21.15.98:0/2545763679 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340bd660 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 --2- 172.21.15.98:0/2545763679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340afa80 0x55e5340afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 --2- 172.21.15.98:0/2545763679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 0x55e5340bad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 -- 172.21.15.98:0/2545763679 >> 172.21.15.98:0/2545763679 conn(0x55e533fdd820 msgr2=0x55e5340abf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:54.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 -- 172.21.15.98:0/2545763679 shutdown_connections 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 -- 172.21.15.98:0/2545763679 wait complete. 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 -- start start 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.246+0000 7f98cb553b80 1 Processor -- start 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.247+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340afa80 0x55e5340ba990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.247+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 0x55e5340b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.247+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.247+0000 7f98cb553b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e5340bdc60 con 0x55e5340b03a0 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.247+0000 7f98cb553b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e533fa1d60 con 0x55e5340afa80 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.247+0000 7f98cb553b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e5340bdde0 con 0x55e5340bb270 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.247+0000 7f98bad9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.247+0000 7f98bad9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340b81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:53988/0 (socket says 172.21.15.98:53988) 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.247+0000 7f98bad9d700 1 -- 172.21.15.98:0/1843326020 learned_addr learned my addr 172.21.15.98:0/1843326020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.247+0000 7f98cb553b80 10 client.0 _ll_register_callbacks cb 0x55e533f9bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:54.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 0x55e5340b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98bad9d700 1 -- 172.21.15.98:0/1843326020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340afa80 msgr2=0x55e5340ba990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98bad9d700 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340afa80 0x55e5340ba990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98bad9d700 1 -- 172.21.15.98:0/1843326020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 msgr2=0x55e5340b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98bad9d700 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 0x55e5340b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98bad9d700 1 -- 172.21.15.98:0/1843326020 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e533fba400 con 0x55e5340bb270 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 0x55e5340b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98bad9d700 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340b81d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f98b00046e0 tx=0x7f98b00052e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79b6870995deaa56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98a1ffb700 1 -- 172.21.15.98:0/1843326020 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b0026070 con 0x55e5340bb270 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98cb553b80 1 -- 172.21.15.98:0/1843326020 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98b000fe50 con 0x55e5340bb270 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98a1ffb700 1 -- 172.21.15.98:0/1843326020 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f98b0010840 con 0x55e5340bb270 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98a1ffb700 1 -- 172.21.15.98:0/1843326020 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f98b0032050 con 0x55e5340bb270 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98cb553b80 10 client.26191 fetch_fsmap learned FSMap version 50 2024-04-25T17:48:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.248+0000 7f98cb553b80 1 -- 172.21.15.98:0/1843326020 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55e5340c8480 con 0x55e5340bb270 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.249+0000 7f98a1ffb700 1 -- 172.21.15.98:0/1843326020 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98b000fe50 con 0x55e5340bb270 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.249+0000 7f98a1ffb700 1 -- 172.21.15.98:0/1843326020 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 50) ==== 8338+0+0 (secure 0 0 0) 0x7f98b0018850 con 0x55e5340bb270 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.249+0000 7f98cb553b80 10 client.26191 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.249+0000 7f98cb553b80 10 client.26191 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.249+0000 7f98cb553b80 20 client.26191 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.249+0000 7f98cb553b80 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5340ca9b0 0x55e5340ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.249+0000 7f98cb553b80 4 client.26191 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.249+0000 7f98cb553b80 1 -- 172.21.15.98:0/1843326020 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5340106b0 con 0x55e5340ca9b0 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.250+0000 7f98ba59c700 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5340ca9b0 0x55e5340ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.250+0000 7f98a1ffb700 10 client.26191 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.250+0000 7f98ba59c700 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5340ca9b0 0x55e5340ccda0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a3df5828a8b7e43 server_cookie=4b1d51dbb75e4f98 in_seq=0 out_seq=0 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98a1ffb700 1 -- 172.21.15.98:0/1843326020 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5340106b0 con 0x55e5340ca9b0 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98a1ffb700 10 client.26191 handle_command_reply: tid=0 2024-04-25T17:48:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 client.26191 shutdown 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 -- 172.21.15.98:0/1843326020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 msgr2=0x55e5340b81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340b81d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f98b00046e0 tx=0x7f98b00052e0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 -- 172.21.15.98:0/1843326020 shutdown_connections 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e5340ca9b0 0x55e5340ccda0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340bb270 0x55e5340b81d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340afa80 0x55e5340ba990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 --2- 172.21.15.98:0/1843326020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340b03a0 0x55e5340b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98a1ffb700 0 client.26191 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 -- 172.21.15.98:0/1843326020 >> 172.21.15.98:0/1843326020 conn(0x55e533fdd820 msgr2=0x55e5340abb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 -- 172.21.15.98:0/1843326020 shutdown_connections 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 1 -- 172.21.15.98:0/1843326020 wait complete. 2024-04-25T17:48:54.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.251+0000 7f98cb553b80 20 client.26191 trim_cache size 0 max 16384 2024-04-25T17:48:54.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.279+0000 7f98cb553b80 1 Processor -- start 2024-04-25T17:48:54.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.279+0000 7f98cb553b80 1 -- start start 2024-04-25T17:48:54.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 0x55e5340d23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 0x55e5340d2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 0x55e5340c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98cb553b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc1b5f00 con 0x55e5340bb270 2024-04-25T17:48:54.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98cb553b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc073be0 con 0x55e5340b03a0 2024-04-25T17:48:54.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98cb553b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc071a50 con 0x55e5340afa80 2024-04-25T17:48:54.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98b9d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 0x55e5340d2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98b9d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 0x55e5340d2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:45048/0 (socket says 172.21.15.98:45048) 2024-04-25T17:48:54.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98b9d9b700 1 -- 172.21.15.98:0/4264413547 learned_addr learned my addr 172.21.15.98:0/4264413547 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:54.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98ba59c700 1 --2- 172.21.15.98:0/4264413547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 0x55e5340d23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98bad9d700 1 --2- 172.21.15.98:0/4264413547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 0x55e5340c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98b9d9b700 1 -- 172.21.15.98:0/4264413547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 msgr2=0x55e5340d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/4264413547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 0x55e5340d23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98b9d9b700 1 -- 172.21.15.98:0/4264413547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 msgr2=0x55e5340c3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/4264413547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 0x55e5340c3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98b9d9b700 1 -- 172.21.15.98:0/4264413547 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5340b68f0 con 0x55e5340b03a0 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.280+0000 7f98bad9d700 1 --2- 172.21.15.98:0/4264413547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 0x55e5340c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.281+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/4264413547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 0x55e5340d2930 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f98ac099d40 tx=0x7f98ac09cfe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36b00d70e5c5710f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.281+0000 7f98a3fff700 1 -- 172.21.15.98:0/4264413547 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0b5020 con 0x55e5340b03a0 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.281+0000 7f98a3fff700 1 -- 172.21.15.98:0/4264413547 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ac0a5f50 con 0x55e5340b03a0 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.281+0000 7f98a3fff700 1 -- 172.21.15.98:0/4264413547 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac09d830 con 0x55e5340b03a0 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 -- 172.21.15.98:0/4264413547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 msgr2=0x55e5340d2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 --2- 172.21.15.98:0/4264413547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 0x55e5340d2930 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f98ac099d40 tx=0x7f98ac09cfe0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 -- 172.21.15.98:0/4264413547 shutdown_connections 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 --2- 172.21.15.98:0/4264413547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 0x55e5340d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 --2- 172.21.15.98:0/4264413547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 0x55e5340c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 --2- 172.21.15.98:0/4264413547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 0x55e5340d2930 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 -- 172.21.15.98:0/4264413547 >> 172.21.15.98:0/4264413547 conn(0x55e533fdd820 msgr2=0x55e5340c5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 -- 172.21.15.98:0/4264413547 shutdown_connections 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 -- 172.21.15.98:0/4264413547 wait complete. 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 -- start start 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.282+0000 7f98cb553b80 1 Processor -- start 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 0x55e5340bfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 0x55e5340c0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98cb553b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 0x55e5340c0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98cb553b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc0720a0 con 0x55e5340bb270 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98cb553b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc073510 con 0x55e5340b03a0 2024-04-25T17:48:54.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98ba59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 0x55e5340bfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98cb553b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc072770 con 0x55e5340afa80 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98ba59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 0x55e5340bfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:54008/0 (socket says 172.21.15.98:54008) 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98ba59c700 1 -- 172.21.15.98:0/3493335465 learned_addr learned my addr 172.21.15.98:0/3493335465 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98cb553b80 10 client.0 _ll_register_callbacks cb 0x55e533f9bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 0x55e5340c0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98bad9d700 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 0x55e5340c0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98ba59c700 1 -- 172.21.15.98:0/3493335465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 msgr2=0x55e5340c0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98ba59c700 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 0x55e5340c0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98ba59c700 1 -- 172.21.15.98:0/3493335465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 msgr2=0x55e5340c0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98ba59c700 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 0x55e5340c0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.283+0000 7f98ba59c700 1 -- 172.21.15.98:0/3493335465 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e534130ed0 con 0x55e5340afa80 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.284+0000 7f98ba59c700 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 0x55e5340bfcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f98b4018c40 tx=0x7f98b4022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fead2df8bc998c59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.284+0000 7f98a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:48:54.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.284+0000 7f98a1ffb700 1 -- 172.21.15.98:0/3493335465 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b403b070 con 0x55e5340afa80 2024-04-25T17:48:54.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.284+0000 7f98cb553b80 1 -- 172.21.15.98:0/3493335465 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98ac0a5f50 con 0x55e5340afa80 2024-04-25T17:48:54.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.284+0000 7f98a1ffb700 1 -- 172.21.15.98:0/3493335465 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f98b4004bb0 con 0x55e5340afa80 2024-04-25T17:48:54.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.284+0000 7f98a1ffb700 1 -- 172.21.15.98:0/3493335465 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f98b4046050 con 0x55e5340afa80 2024-04-25T17:48:54.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.284+0000 7f98cb553b80 10 client.26197 fetch_fsmap learned FSMap version 50 2024-04-25T17:48:54.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.284+0000 7f98cb553b80 1 -- 172.21.15.98:0/3493335465 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55e534132140 con 0x55e5340afa80 2024-04-25T17:48:54.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.285+0000 7f98a1ffb700 1 -- 172.21.15.98:0/3493335465 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ac0a5f50 con 0x55e5340afa80 2024-04-25T17:48:54.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.285+0000 7f98a1ffb700 1 -- 172.21.15.98:0/3493335465 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 50) ==== 8338+0+0 (secure 0 0 0) 0x7f98b40263b0 con 0x55e5340afa80 2024-04-25T17:48:54.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.285+0000 7f98cb553b80 10 client.26197 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T17:48:54.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.285+0000 7f98cb553b80 10 client.26197 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:48:54.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.285+0000 7f98cb553b80 20 client.26197 populate_metadata read hostname 'smithi098' 2024-04-25T17:48:54.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.285+0000 7f98cb553b80 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e534132870 0x55e534134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:48:54.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.285+0000 7f98cb553b80 4 client.26197 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.1, "await_for": 42.0}] 2024-04-25T17:48:54.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.285+0000 7f98cb553b80 1 -- 172.21.15.98:0/3493335465 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.1, "await_for": 42.0}) -- 0x55e53400a750 con 0x55e534132870 2024-04-25T17:48:54.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.285+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e534132870 0x55e534134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:48:54.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.286+0000 7f98a1ffb700 10 client.26197 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:54.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.286+0000 7f98b9d9b700 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e534132870 0x55e534134c90 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79a305b7950e528e server_cookie=ecf37d7a979bba19 in_seq=0 out_seq=0 2024-04-25T17:48:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98a1ffb700 1 -- 172.21.15.98:0/3493335465 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55e53400a750 con 0x55e534132870 2024-04-25T17:48:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98a1ffb700 10 client.26197 handle_command_reply: tid=0 2024-04-25T17:48:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 client.26197 shutdown 2024-04-25T17:48:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 -- 172.21.15.98:0/3493335465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 msgr2=0x55e5340bfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 0x55e5340bfcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f98b4018c40 tx=0x7f98b4022000 comp rx=0 tx=0).stop 2024-04-25T17:48:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 -- 172.21.15.98:0/3493335465 shutdown_connections 2024-04-25T17:48:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e534132870 0x55e534134c90 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98a1ffb700 0 client.26197 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:48:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5340afa80 0x55e5340bfcd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5340bb270 0x55e5340c0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 --2- 172.21.15.98:0/3493335465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5340b03a0 0x55e5340c0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 -- 172.21.15.98:0/3493335465 >> 172.21.15.98:0/3493335465 conn(0x55e533fdd820 msgr2=0x55e5340c63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 -- 172.21.15.98:0/3493335465 shutdown_connections 2024-04-25T17:48:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 1 -- 172.21.15.98:0/3493335465 wait complete. 2024-04-25T17:48:54.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.523+0000 7f98cb553b80 20 client.26197 trim_cache size 0 max 16384 2024-04-25T17:48:54.526 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:48:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 50, 2024-04-25T17:48:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:48:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 38, 2024-04-25T17:48:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:48:54.526 INFO:teuthology.orchestra.run.smithi098.stdout: "a13e6163": { 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 38, 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 89.1, 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:48:54.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:48:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:48:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:48:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:48:54.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:48:54.528 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:48:54.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 -- 172.21.15.98:0/2411201355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f98a40a4080 msgr2=0x7f98a40a6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:54.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f98a40a4080 0x7f98a40a6540 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f98b401ab40 tx=0x7f98b401a040 comp rx=0 tx=0).stop 2024-04-25T17:48:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 -- 172.21.15.98:0/2411201355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98bc107a60 msgr2=0x7f98bc19cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:48:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98bc107a60 0x7f98bc19cfa0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b840 tx=0x7f98ac00b870 comp rx=0 tx=0).stop 2024-04-25T17:48:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 -- 172.21.15.98:0/2411201355 shutdown_connections 2024-04-25T17:48:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f98a40a4080 0x7f98a40a6540 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98bc107ed0 0x7f98bc196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98bc1054d0 0x7f98bc19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 --2- 172.21.15.98:0/2411201355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98bc107a60 0x7f98bc19cfa0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:48:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 -- 172.21.15.98:0/2411201355 >> 172.21.15.98:0/2411201355 conn(0x7f98bc0788f0 msgr2=0x7f98bc104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:48:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 -- 172.21.15.98:0/2411201355 shutdown_connections 2024-04-25T17:48:54.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:48:54.524+0000 7f98c0cca700 1 -- 172.21.15.98:0/2411201355 wait complete. 2024-04-25T17:48:55.230 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a13e6163', quiesce took 9.7 seconds. Will release after: 29.1 2024-04-25T17:48:55.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:55 smithi098 ceph-mon[29192]: pgmap v409: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 591 op/s 2024-04-25T17:48:55.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:55 smithi203 ceph-mon[32834]: pgmap v409: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 591 op/s 2024-04-25T17:48:56.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:55 smithi173 ceph-mon[32856]: pgmap v409: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 591 op/s 2024-04-25T17:48:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 745.0 s, 726.3 tps, lat 77.638 ms stddev 79.496 2024-04-25T17:48:57.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:57 smithi098 ceph-mon[29192]: pgmap v410: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 8.5 MiB/s wr, 754 op/s 2024-04-25T17:48:57.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:57 smithi203 ceph-mon[32834]: pgmap v410: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 8.5 MiB/s wr, 754 op/s 2024-04-25T17:48:58.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:57 smithi173 ceph-mon[32856]: pgmap v410: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 8.5 MiB/s wr, 754 op/s 2024-04-25T17:48:59.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:48:59 smithi203 ceph-mon[32834]: pgmap v411: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.9 MiB/s wr, 487 op/s 2024-04-25T17:48:59.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:48:59 smithi098 ceph-mon[29192]: pgmap v411: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.9 MiB/s wr, 487 op/s 2024-04-25T17:49:00.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:48:59 smithi173 ceph-mon[32856]: pgmap v411: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.9 MiB/s wr, 487 op/s 2024-04-25T17:49:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 750.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:49:01.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:01 smithi098 ceph-mon[29192]: pgmap v412: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.9 MiB/s wr, 487 op/s 2024-04-25T17:49:01.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:01 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:49:01.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:01 smithi203 ceph-mon[32834]: pgmap v412: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.9 MiB/s wr, 487 op/s 2024-04-25T17:49:01.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:01 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:49:02.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:01 smithi173 ceph-mon[32856]: pgmap v412: 97 pgs: 97 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.9 MiB/s wr, 487 op/s 2024-04-25T17:49:02.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:01 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:49:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.916+0000 7f6cf719b640 1 -- 172.21.15.98:0/757418050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf01021d0 msgr2=0x7f6cf01025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:02.918 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 --2- 172.21.15.98:0/757418050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf01021d0 0x7f6cf01025b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009900 tx=0x7f6ce402a410 comp rx=0 tx=0).stop 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 -- 172.21.15.98:0/757418050 shutdown_connections 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 --2- 172.21.15.98:0/757418050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf01072a0 0x7f6cf010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 --2- 172.21.15.98:0/757418050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf0102b80 0x7f6cf0106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 --2- 172.21.15.98:0/757418050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf01021d0 0x7f6cf01025b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 -- 172.21.15.98:0/757418050 >> 172.21.15.98:0/757418050 conn(0x7f6cf00793e0 msgr2=0x7f6cf00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 -- 172.21.15.98:0/757418050 shutdown_connections 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 -- 172.21.15.98:0/757418050 wait complete. 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 Processor -- start 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 -- start start 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf01021d0 0x7f6cf01986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf01072a0 0x7f6cf0198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf019cf60 0x7f6cf019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf01121e0 con 0x7f6cf01072a0 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf0112060 con 0x7f6cf01021d0 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.917+0000 7f6cf719b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf0112360 con 0x7f6cf019cf60 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6ceffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf01072a0 0x7f6cf0198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6ceffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf01072a0 0x7f6cf0198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59936/0 (socket says 172.21.15.98:59936) 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6ceffff640 1 -- 172.21.15.98:0/848095505 learned_addr learned my addr 172.21.15.98:0/848095505 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf01021d0 0x7f6cf01986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6cf5738640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf019cf60 0x7f6cf019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6ceffff640 1 -- 172.21.15.98:0/848095505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf019cf60 msgr2=0x7f6cf019d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6ceffff640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf019cf60 0x7f6cf019d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6ceffff640 1 -- 172.21.15.98:0/848095505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf01021d0 msgr2=0x7f6cf01986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6ceffff640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf01021d0 0x7f6cf01986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6ceffff640 1 -- 172.21.15.98:0/848095505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf019da50 con 0x7f6cf01072a0 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6ceffff640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf01072a0 0x7f6cf0198be0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00d770 tx=0x7f6cdc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6cedffb640 1 -- 172.21.15.98:0/848095505 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc017070 con 0x7f6cf01072a0 2024-04-25T17:49:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6cf719b640 1 -- 172.21.15.98:0/848095505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf019dd40 con 0x7f6cf01072a0 2024-04-25T17:49:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6cedffb640 1 -- 172.21.15.98:0/848095505 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cdc009c30 con 0x7f6cf01072a0 2024-04-25T17:49:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.918+0000 7f6cedffb640 1 -- 172.21.15.98:0/848095505 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc00f620 con 0x7f6cf01072a0 2024-04-25T17:49:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.919+0000 7f6cf719b640 1 -- 172.21.15.98:0/848095505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf01a5580 con 0x7f6cf01072a0 2024-04-25T17:49:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.921+0000 7f6cedffb640 1 -- 172.21.15.98:0/848095505 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6cdc01a070 con 0x7f6cf01072a0 2024-04-25T17:49:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.921+0000 7f6cf9f8c740 1 Processor -- start 2024-04-25T17:49:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.921+0000 7f6cf9f8c740 1 -- start start 2024-04-25T17:49:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.921+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c61aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.921+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c6252c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.921+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c627bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.921+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a11c57f970 con 0x55a11c625800 2024-04-25T17:49:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.921+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a11c51c4a0 con 0x55a11c61a620 2024-04-25T17:49:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.921+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a11c57b930 con 0x55a11c61af40 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cedffb640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6ce00a3dc0 0x7f6ce00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6ceffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c627bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf5738640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6ce00a3dc0 0x7f6ce00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6ceffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c627bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59940/0 (socket says 172.21.15.98:59940) 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6ceffff640 1 -- 172.21.15.98:0/4139008939 learned_addr learned my addr 172.21.15.98:0/4139008939 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c6252c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf5738640 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c61aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cedffb640 1 -- 172.21.15.98:0/848095505 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f6cdc0d02c0 con 0x7f6cf01072a0 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf4f37640 1 -- 172.21.15.98:0/4139008939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 msgr2=0x55a11c61aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c61aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf4f37640 1 -- 172.21.15.98:0/4139008939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 msgr2=0x55a11c627bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c627bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf4f37640 1 -- 172.21.15.98:0/4139008939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a11c292210 con 0x55a11c61af40 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6ceffff640 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c627bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf5738640 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c61aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf5738640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6ce00a3dc0 0x7f6ce00a6280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8005fd0 tx=0x7f6ce8012a50 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.922+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c6252c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009230 tx=0x7f6ce4032bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=203054940940d319 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cdaffd640 1 -- 172.21.15.98:0/4139008939 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4040070 con 0x55a11c61af40 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cdaffd640 1 -- 172.21.15.98:0/4139008939 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce4004030 con 0x55a11c61af40 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cdaffd640 1 -- 172.21.15.98:0/4139008939 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4035040 con 0x55a11c61af40 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/4139008939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 msgr2=0x55a11c6252c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c6252c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009230 tx=0x7f6ce4032bd0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/4139008939 shutdown_connections 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c627bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c6252c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/4139008939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c61aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/4139008939 >> 172.21.15.98:0/4139008939 conn(0x55a11c616b30 msgr2=0x55a11c616f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/4139008939 shutdown_connections 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.923+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/4139008939 wait complete. 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf9f8c740 1 -- start start 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf9f8c740 1 Processor -- start 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61a620 0x55a11c6251b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61af40 0x55a11c6224a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c622a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a11c6283d0 con 0x55a11c625800 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a11c51a0b0 con 0x55a11c61af40 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a11c628550 con 0x55a11c61a620 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf9f8c740 10 client.0 _ll_register_callbacks cb 0x55a11c507d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf5738640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c622a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf5738640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c622a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59946/0 (socket says 172.21.15.98:59946) 2024-04-25T17:49:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf5738640 1 -- 172.21.15.98:0/3981335398 learned_addr learned my addr 172.21.15.98:0/3981335398 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.924+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61a620 0x55a11c6251b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6ceffff640 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61af40 0x55a11c6224a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6cf5738640 1 -- 172.21.15.98:0/3981335398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61a620 msgr2=0x55a11c6251b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6cf5738640 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61a620 0x55a11c6251b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6cf5738640 1 -- 172.21.15.98:0/3981335398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61af40 msgr2=0x55a11c6224a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6cf5738640 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61af40 0x55a11c6224a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6cf5738640 1 -- 172.21.15.98:0/3981335398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a11c35e620 con 0x55a11c625800 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61a620 0x55a11c6251b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6cf5738640 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c622a40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8010b50 tx=0x7f6ce80095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95af0febd9c3050 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6ccffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6ccffff640 1 -- 172.21.15.98:0/3981335398 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8025070 con 0x55a11c625800 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6ccffff640 1 -- 172.21.15.98:0/3981335398 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f6ce801f420 con 0x55a11c625800 2024-04-25T17:49:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/3981335398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ce4004030 con 0x55a11c625800 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6ccffff640 1 -- 172.21.15.98:0/3981335398 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f6ce8031050 con 0x55a11c625800 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6cf9f8c740 10 client.16740 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/3981335398 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55a11c634390 con 0x55a11c625800 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.925+0000 7f6ccffff640 1 -- 172.21.15.98:0/3981335398 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce4004030 con 0x55a11c625800 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.926+0000 7f6ccffff640 1 -- 172.21.15.98:0/3981335398 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7f6ce801e580 con 0x55a11c625800 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.926+0000 7f6cf9f8c740 10 client.16740 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.926+0000 7f6cf9f8c740 10 client.16740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.926+0000 7f6cf9f8c740 20 client.16740 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.926+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a11c635850 0x55a11c637c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.926+0000 7f6cf9f8c740 4 client.16740 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.926+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/3981335398 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a11c58d390 con 0x55a11c635850 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.926+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a11c635850 0x55a11c637c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.926+0000 7f6ccffff640 10 client.16740 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.926+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a11c635850 0x55a11c637c40 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0103a9fb2b35264 server_cookie=1c4203b6f2fc1689 in_seq=0 out_seq=0 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6ccffff640 1 -- 172.21.15.98:0/3981335398 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a11c58d390 con 0x55a11c635850 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6ccffff640 10 client.16740 handle_command_reply: tid=0 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 client.16740 shutdown 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/3981335398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 msgr2=0x55a11c622a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c622a40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8010b50 tx=0x7f6ce80095f0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/3981335398 shutdown_connections 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a11c635850 0x55a11c637c40 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c622a40 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61af40 0x55a11c6224a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6ccffff640 0 client.16740 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/3981335398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61a620 0x55a11c6251b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/3981335398 >> 172.21.15.98:0/3981335398 conn(0x55a11c616b30 msgr2=0x55a11c61c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/3981335398 shutdown_connections 2024-04-25T17:49:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/3981335398 wait complete. 2024-04-25T17:49:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.927+0000 7f6cf9f8c740 20 client.16740 trim_cache size 0 max 16384 2024-04-25T17:49:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.942+0000 7f6cf9f8c740 1 Processor -- start 2024-04-25T17:49:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.942+0000 7f6cf9f8c740 1 -- start start 2024-04-25T17:49:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.942+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c631290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.942+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c615b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.942+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c63d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.942+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf01b0390 con 0x55a11c625800 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.942+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf01adc60 con 0x55a11c61a620 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.942+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf01abad0 con 0x55a11c61af40 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.942+0000 7f6cf5738640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c63d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cf5738640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c63d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59958/0 (socket says 172.21.15.98:59958) 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cf5738640 1 -- 172.21.15.98:0/1848172328 learned_addr learned my addr 172.21.15.98:0/1848172328 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6ceffff640 1 --2- 172.21.15.98:0/1848172328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c615b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/1848172328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c631290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cf5738640 1 -- 172.21.15.98:0/1848172328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 msgr2=0x55a11c615b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cf5738640 1 --2- 172.21.15.98:0/1848172328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c615b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cf5738640 1 -- 172.21.15.98:0/1848172328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 msgr2=0x55a11c631290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cf5738640 1 --2- 172.21.15.98:0/1848172328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c631290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cf5738640 1 -- 172.21.15.98:0/1848172328 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a11c631dc0 con 0x55a11c625800 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/1848172328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c631290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cf5738640 1 --2- 172.21.15.98:0/1848172328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c63d550 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009320 tx=0x7f6ce80104c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4bd22bda79a02c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cda7fc640 1 -- 172.21.15.98:0/1848172328 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce802e070 con 0x55a11c625800 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cda7fc640 1 -- 172.21.15.98:0/1848172328 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce801c800 con 0x55a11c625800 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.943+0000 7f6cda7fc640 1 -- 172.21.15.98:0/1848172328 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8025040 con 0x55a11c625800 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1848172328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 msgr2=0x55a11c63d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/1848172328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c63d550 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009320 tx=0x7f6ce80104c0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1848172328 shutdown_connections 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/1848172328 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c63d550 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/1848172328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c615b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/1848172328 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c631290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1848172328 >> 172.21.15.98:0/1848172328 conn(0x55a11c616b30 msgr2=0x55a11c615420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1848172328 shutdown_connections 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1848172328 wait complete. 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 -- start start 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 Processor -- start 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.944+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c62dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c62b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf9f8c740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c62b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf01ac120 con 0x55a11c625800 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf01ad590 con 0x55a11c61a620 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf9f8c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf01ac7f0 con 0x55a11c61af40 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf9f8c740 10 client.0 _ll_register_callbacks cb 0x7f6ce8010850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf5738640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c62b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf5738640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c62b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59974/0 (socket says 172.21.15.98:59974) 2024-04-25T17:49:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf5738640 1 -- 172.21.15.98:0/1796912735 learned_addr learned my addr 172.21.15.98:0/1796912735 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6ceffff640 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c62b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c62dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf5738640 1 -- 172.21.15.98:0/1796912735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 msgr2=0x55a11c62b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf5738640 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c62b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf5738640 1 -- 172.21.15.98:0/1796912735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 msgr2=0x55a11c62dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf5738640 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c62dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf5738640 1 -- 172.21.15.98:0/1796912735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a11c69f1e0 con 0x55a11c625800 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c62dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf5738640 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c62b780 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8010b50 tx=0x7f6ce801eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f659b26c1c3ea60c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6ccffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6ccffff640 1 -- 172.21.15.98:0/1796912735 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce802e030 con 0x55a11c625800 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6ccffff640 1 -- 172.21.15.98:0/1796912735 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f6ce801ee40 con 0x55a11c625800 2024-04-25T17:49:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1796912735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ce801c800 con 0x55a11c625800 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6ccffff640 1 -- 172.21.15.98:0/1796912735 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f6ce8025040 con 0x55a11c625800 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf9f8c740 10 client.16746 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.945+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1796912735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55a11c6a02e0 con 0x55a11c625800 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6ccffff640 1 -- 172.21.15.98:0/1796912735 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ce801c800 con 0x55a11c625800 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6ccffff640 1 -- 172.21.15.98:0/1796912735 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7f6ce80050d0 con 0x55a11c625800 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6cf9f8c740 10 client.16746 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6cf9f8c740 10 client.16746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6cf9f8c740 20 client.16746 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a11c6a08c0 0x55a11c6a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6cf9f8c740 4 client.16746 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1796912735 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a11c587200 con 0x55a11c6a08c0 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a11c6a08c0 0x55a11c6a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6cf4f37640 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a11c6a08c0 0x55a11c6a2ce0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2510df865942d80d server_cookie=9bfaffda877d3033 in_seq=0 out_seq=0 2024-04-25T17:49:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.946+0000 7f6ccffff640 10 client.16746 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6ccffff640 1 -- 172.21.15.98:0/1796912735 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a11c587200 con 0x55a11c6a08c0 2024-04-25T17:49:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6ccffff640 10 client.16746 handle_command_reply: tid=0 2024-04-25T17:49:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 client.16746 shutdown 2024-04-25T17:49:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1796912735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 msgr2=0x55a11c62b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c62b780 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8010b50 tx=0x7f6ce801eaf0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1796912735 shutdown_connections 2024-04-25T17:49:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55a11c6a08c0 0x55a11c6a2ce0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a11c625800 0x55a11c62b780 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a11c61af40 0x55a11c62b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 --2- 172.21.15.98:0/1796912735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a11c61a620 0x55a11c62dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6ccffff640 0 client.16746 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1796912735 >> 172.21.15.98:0/1796912735 conn(0x55a11c616b30 msgr2=0x55a11c615cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1796912735 shutdown_connections 2024-04-25T17:49:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 1 -- 172.21.15.98:0/1796912735 wait complete. 2024-04-25T17:49:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.957+0000 7f6cf9f8c740 20 client.16746 trim_cache size 0 max 16384 2024-04-25T17:49:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:49:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:49:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:49:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:49:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 -- 172.21.15.98:0/848095505 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6ce00a3dc0 msgr2=0x7f6ce00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6ce00a3dc0 0x7f6ce00a6280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8005fd0 tx=0x7f6ce8012a50 comp rx=0 tx=0).stop 2024-04-25T17:49:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 -- 172.21.15.98:0/848095505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf01072a0 msgr2=0x7f6cf0198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf01072a0 0x7f6cf0198be0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00d770 tx=0x7f6cdc00dc30 comp rx=0 tx=0).stop 2024-04-25T17:49:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 -- 172.21.15.98:0/848095505 shutdown_connections 2024-04-25T17:49:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f6ce00a3dc0 0x7f6ce00a6280 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf019cf60 0x7f6cf019d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cf01072a0 0x7f6cf0198be0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 --2- 172.21.15.98:0/848095505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6cf01021d0 0x7f6cf01986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 -- 172.21.15.98:0/848095505 >> 172.21.15.98:0/848095505 conn(0x7f6cf00793e0 msgr2=0x7f6cf00ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 -- 172.21.15.98:0/848095505 shutdown_connections 2024-04-25T17:49:02.960 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:02.958+0000 7f6cf719b640 1 -- 172.21.15.98:0/848095505 wait complete. 2024-04-25T17:49:03.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:03 smithi098 ceph-mon[29192]: pgmap v413: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 1.65k op/s 2024-04-25T17:49:03.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:03 smithi203 ceph-mon[32834]: pgmap v413: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 1.65k op/s 2024-04-25T17:49:04.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:03 smithi173 ceph-mon[32856]: pgmap v413: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 26 MiB/s wr, 1.65k op/s 2024-04-25T17:49:05.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:05 smithi098 ceph-mon[29192]: pgmap v414: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 22 MiB/s wr, 1.33k op/s 2024-04-25T17:49:05.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:05 smithi098 ceph-mon[29192]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T17:49:05.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:05 smithi203 ceph-mon[32834]: pgmap v414: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 22 MiB/s wr, 1.33k op/s 2024-04-25T17:49:05.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:05 smithi203 ceph-mon[32834]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T17:49:06.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:05 smithi173 ceph-mon[32856]: pgmap v414: 97 pgs: 97 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 22 MiB/s wr, 1.33k op/s 2024-04-25T17:49:06.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:05 smithi173 ceph-mon[32856]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T17:49:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 755.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:49:07.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:07 smithi098 ceph-mon[29192]: pgmap v416: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.43k op/s 2024-04-25T17:49:07.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:07 smithi203 ceph-mon[32834]: pgmap v416: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.43k op/s 2024-04-25T17:49:08.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:07 smithi173 ceph-mon[32856]: pgmap v416: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.43k op/s 2024-04-25T17:49:09.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:09 smithi098 ceph-mon[29192]: pgmap v417: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.43k op/s 2024-04-25T17:49:09.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:09 smithi203 ceph-mon[32834]: pgmap v417: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.43k op/s 2024-04-25T17:49:10.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:09 smithi173 ceph-mon[32856]: pgmap v417: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 24 MiB/s wr, 1.43k op/s 2024-04-25T17:49:11.457 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:11 smithi098 ceph-mon[29192]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T17:49:11.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:11 smithi203 ceph-mon[32834]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T17:49:11.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:11 smithi173 ceph-mon[32856]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T17:49:11.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 760.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:49:12.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:12 smithi098 ceph-mon[29192]: pgmap v419: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 218 KiB/s wr, 36 op/s 2024-04-25T17:49:12.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:12 smithi203 ceph-mon[32834]: pgmap v419: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 218 KiB/s wr, 36 op/s 2024-04-25T17:49:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:12 smithi173 ceph-mon[32856]: pgmap v419: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 218 KiB/s wr, 36 op/s 2024-04-25T17:49:14.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:14 smithi098 ceph-mon[29192]: pgmap v420: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 253 KiB/s wr, 42 op/s 2024-04-25T17:49:14.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:14 smithi203 ceph-mon[32834]: pgmap v420: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 253 KiB/s wr, 42 op/s 2024-04-25T17:49:14.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:14 smithi173 ceph-mon[32856]: pgmap v420: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 253 KiB/s wr, 42 op/s 2024-04-25T17:49:15.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:15 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:49:15.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:15 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:49:15.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:15 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:49:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.044+0000 7f64b0319640 1 -- 172.21.15.98:0/2850398455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a8106190 msgr2=0x7f64a810dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.044+0000 7f64b0319640 1 --2- 172.21.15.98:0/2850398455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a8106190 0x7f64a810dd60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009ca0 tx=0x7f64a002a8c0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.044+0000 7f64b0319640 1 -- 172.21.15.98:0/2850398455 shutdown_connections 2024-04-25T17:49:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.044+0000 7f64b0319640 1 --2- 172.21.15.98:0/2850398455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a8106190 0x7f64a810dd60 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.044+0000 7f64b0319640 1 --2- 172.21.15.98:0/2850398455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64a8101a70 0x7f64a8105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.044+0000 7f64b0319640 1 --2- 172.21.15.98:0/2850398455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64a81010c0 0x7f64a81014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.044+0000 7f64b0319640 1 -- 172.21.15.98:0/2850398455 >> 172.21.15.98:0/2850398455 conn(0x7f64a8077fd0 msgr2=0x7f64a80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.044+0000 7f64b0319640 1 -- 172.21.15.98:0/2850398455 shutdown_connections 2024-04-25T17:49:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.044+0000 7f64b0319640 1 -- 172.21.15.98:0/2850398455 wait complete. 2024-04-25T17:49:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64b0319640 1 Processor -- start 2024-04-25T17:49:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64b0319640 1 -- start start 2024-04-25T17:49:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64b0319640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a81010c0 0x7f64a8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64b0319640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64a8106190 0x7f64a8198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64b0319640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64a819cf00 0x7f64a819d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64b0319640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64a8112210 con 0x7f64a81010c0 2024-04-25T17:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64b0319640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f64a8112090 con 0x7f64a819cf00 2024-04-25T17:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64b0319640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64a8112390 con 0x7f64a8106190 2024-04-25T17:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64ae0b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a81010c0 0x7f64a8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64ae0b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a81010c0 0x7f64a8198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38402/0 (socket says 172.21.15.98:38402) 2024-04-25T17:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64ae0b5640 1 -- 172.21.15.98:0/3570924668 learned_addr learned my addr 172.21.15.98:0/3570924668 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64ae8b6640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64a819cf00 0x7f64a819d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64ad8b4640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64a8106190 0x7f64a8198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64ae0b5640 1 -- 172.21.15.98:0/3570924668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64a8106190 msgr2=0x7f64a8198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:16.048 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64a8106190 0x7f64a8198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64ae0b5640 1 -- 172.21.15.98:0/3570924668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64a819cf00 msgr2=0x7f64a819d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64a819cf00 0x7f64a819d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.045+0000 7f64ae0b5640 1 -- 172.21.15.98:0/3570924668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a819d9f0 con 0x7f64a81010c0 2024-04-25T17:49:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.046+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a81010c0 0x7f64a8198690 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f64a400c890 tx=0x7f64a400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.046+0000 7f649f7fe640 1 -- 172.21.15.98:0/3570924668 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4004220 con 0x7f64a81010c0 2024-04-25T17:49:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.046+0000 7f649f7fe640 1 -- 172.21.15.98:0/3570924668 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a40043c0 con 0x7f64a81010c0 2024-04-25T17:49:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.046+0000 7f64b0319640 1 -- 172.21.15.98:0/3570924668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a819dce0 con 0x7f64a81010c0 2024-04-25T17:49:16.049 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.046+0000 7f649f7fe640 1 -- 172.21.15.98:0/3570924668 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4004a00 con 0x7f64a81010c0 2024-04-25T17:49:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.046+0000 7f64b0319640 1 -- 172.21.15.98:0/3570924668 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a81a5520 con 0x7f64a81010c0 2024-04-25T17:49:16.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.048+0000 7f649f7fe640 1 -- 172.21.15.98:0/3570924668 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f64a40079b0 con 0x7f64a81010c0 2024-04-25T17:49:16.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.048+0000 7f64b302b740 1 Processor -- start 2024-04-25T17:49:16.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.048+0000 7f64b302b740 1 -- start start 2024-04-25T17:49:16.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.048+0000 7f64b302b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3ed1620 0x5576c3ed1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64b302b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3edc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64b302b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3edc800 0x5576c3edebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64b302b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576c3e36970 con 0x5576c3ed1f40 2024-04-25T17:49:16.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64b302b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5576c3dd34a0 con 0x5576c3edc800 2024-04-25T17:49:16.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64b302b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5576c3e32930 con 0x5576c3ed1620 2024-04-25T17:49:16.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f649f7fe640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f64940a3e40 0x7f64940a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae0b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3edc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae0b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3edc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38418/0 (socket says 172.21.15.98:38418) 2024-04-25T17:49:16.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae0b5640 1 -- 172.21.15.98:0/1007443995 learned_addr learned my addr 172.21.15.98:0/1007443995 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:16.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae8b6640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f64940a3e40 0x7f64940a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae8b6640 1 --2- 172.21.15.98:0/1007443995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3ed1620 0x5576c3ed1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ad8b4640 1 --2- 172.21.15.98:0/1007443995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3edc800 0x5576c3edebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.059 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f649f7fe640 1 -- 172.21.15.98:0/3570924668 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7f64a40d0420 con 0x7f64a81010c0 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae0b5640 1 -- 172.21.15.98:0/1007443995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3ed1620 msgr2=0x5576c3ed1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/1007443995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3ed1620 0x5576c3ed1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae0b5640 1 -- 172.21.15.98:0/1007443995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3edc800 msgr2=0x5576c3edebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/1007443995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3edc800 0x5576c3edebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae0b5640 1 -- 172.21.15.98:0/1007443995 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576c3b49210 con 0x5576c3ed1f40 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae8b6640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f64940a3e40 0x7f64940a6300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f64a002f500 tx=0x7f64a002f310 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/1007443995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3edc2c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f64a40988d0 tx=0x7f64a400a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402c6cffc89f88c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f649cbfe640 1 -- 172.21.15.98:0/1007443995 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a401d050 con 0x5576c3ed1f40 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.049+0000 7f649cbfe640 1 -- 172.21.15.98:0/1007443995 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a4005290 con 0x5576c3ed1f40 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.050+0000 7f649cbfe640 1 -- 172.21.15.98:0/1007443995 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4098ae0 con 0x5576c3ed1f40 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.050+0000 7f64b302b740 1 -- 172.21.15.98:0/1007443995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 msgr2=0x5576c3edc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.050+0000 7f64b302b740 1 --2- 172.21.15.98:0/1007443995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3edc2c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f64a40988d0 tx=0x7f64a400a5f0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.050+0000 7f64b302b740 1 -- 172.21.15.98:0/1007443995 shutdown_connections 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.050+0000 7f64b302b740 1 --2- 172.21.15.98:0/1007443995 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3edc800 0x5576c3edebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.050+0000 7f64b302b740 1 --2- 172.21.15.98:0/1007443995 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3edc2c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.050+0000 7f64b302b740 1 --2- 172.21.15.98:0/1007443995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3ed1620 0x5576c3ed1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.050+0000 7f64b302b740 1 -- 172.21.15.98:0/1007443995 >> 172.21.15.98:0/1007443995 conn(0x5576c3ecdb30 msgr2=0x5576c3ecdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.050+0000 7f64b302b740 1 -- 172.21.15.98:0/1007443995 shutdown_connections 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.050+0000 7f64b302b740 1 -- 172.21.15.98:0/1007443995 wait complete. 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64b302b740 1 -- start start 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64b302b740 1 Processor -- start 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64b302b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 0x5576c3edc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64b302b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3ed1f40 0x5576c3ed94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64b302b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3edc800 0x5576c3ed9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64b302b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5576c3edf370 con 0x5576c3edc800 2024-04-25T17:49:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64b302b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5576c3dd10b0 con 0x5576c3ed1620 2024-04-25T17:49:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64b302b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5576c3edf4f0 con 0x5576c3ed1f40 2024-04-25T17:49:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64b302b740 10 client.0 _ll_register_callbacks cb 0x5576c3dbed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae8b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3edc800 0x5576c3ed9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae8b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3edc800 0x5576c3ed9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38432/0 (socket says 172.21.15.98:38432) 2024-04-25T17:49:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae8b6640 1 -- 172.21.15.98:0/3559239045 learned_addr learned my addr 172.21.15.98:0/3559239045 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ad8b4640 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3ed1f40 0x5576c3ed94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 0x5576c3edc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae8b6640 1 -- 172.21.15.98:0/3559239045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3ed1f40 msgr2=0x5576c3ed94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae8b6640 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3ed1f40 0x5576c3ed94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae8b6640 1 -- 172.21.15.98:0/3559239045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 msgr2=0x5576c3edc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae8b6640 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 0x5576c3edc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae8b6640 1 -- 172.21.15.98:0/3559239045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576c3c15620 con 0x5576c3edc800 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 0x5576c3edc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f64ae8b6640 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3edc800 0x5576c3ed9ae0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f64a0035000 tx=0x7f64a002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13190fbf584de119 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.051+0000 7f6491ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f6491ffb640 1 -- 172.21.15.98:0/3559239045 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0052070 con 0x5576c3edc800 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f6491ffb640 1 -- 172.21.15.98:0/3559239045 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7f64a0003d00 con 0x5576c3edc800 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f64b302b740 1 -- 172.21.15.98:0/3559239045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5576c3e44390 con 0x5576c3edc800 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f6491ffb640 1 -- 172.21.15.98:0/3559239045 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f64a005e050 con 0x5576c3edc800 2024-04-25T17:49:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f64b302b740 10 client.16758 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f64b302b740 1 -- 172.21.15.98:0/3559239045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x5576c3eeb320 con 0x5576c3edc800 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f6491ffb640 1 -- 172.21.15.98:0/3559239045 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5576c3e44390 con 0x5576c3edc800 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f6491ffb640 1 -- 172.21.15.98:0/3559239045 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7f64a004c040 con 0x5576c3edc800 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f64b302b740 10 client.16758 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f64b302b740 10 client.16758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f64b302b740 20 client.16758 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f64b302b740 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5576c3eec7e0 0x5576c3eeebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f64b302b740 4 client.16758 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.052+0000 7f64b302b740 1 -- 172.21.15.98:0/3559239045 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576c3e3e200 con 0x5576c3eec7e0 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.053+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5576c3eec7e0 0x5576c3eeebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.053+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5576c3eec7e0 0x5576c3eeebd0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdf5742903675ee4 server_cookie=793a1e8d487101ff in_seq=0 out_seq=0 2024-04-25T17:49:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.053+0000 7f6491ffb640 10 client.16758 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f6491ffb640 1 -- 172.21.15.98:0/3559239045 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576c3e3e200 con 0x5576c3eec7e0 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f6491ffb640 10 client.16758 handle_command_reply: tid=0 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 client.16758 shutdown 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 -- 172.21.15.98:0/3559239045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3edc800 msgr2=0x5576c3ed9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3edc800 0x5576c3ed9ae0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f64a0035000 tx=0x7f64a002a8c0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 -- 172.21.15.98:0/3559239045 shutdown_connections 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5576c3eec7e0 0x5576c3eeebd0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3edc800 0x5576c3ed9ae0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f6491ffb640 0 client.16758 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3ed1f40 0x5576c3ed94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 --2- 172.21.15.98:0/3559239045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 0x5576c3edc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 -- 172.21.15.98:0/3559239045 >> 172.21.15.98:0/3559239045 conn(0x5576c3ecdb30 msgr2=0x5576c3ed3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 -- 172.21.15.98:0/3559239045 shutdown_connections 2024-04-25T17:49:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 1 -- 172.21.15.98:0/3559239045 wait complete. 2024-04-25T17:49:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.054+0000 7f64b302b740 20 client.16758 trim_cache size 0 max 16384 2024-04-25T17:49:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.068+0000 7f64b302b740 1 Processor -- start 2024-04-25T17:49:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.068+0000 7f64b302b740 1 -- start start 2024-04-25T17:49:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.068+0000 7f64b302b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 0x5576c3ed8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.068+0000 7f64b302b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3ef44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64b302b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 0x5576c3ef4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64b302b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81b0440 con 0x5576c3ed1f40 2024-04-25T17:49:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64b302b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81add10 con 0x5576c3ed1620 2024-04-25T17:49:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64b302b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81abb80 con 0x5576c3edc800 2024-04-25T17:49:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ad8b4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3ef44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ad8b4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3ef44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38442/0 (socket says 172.21.15.98:38442) 2024-04-25T17:49:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ad8b4640 1 -- 172.21.15.98:0/68804420 learned_addr learned my addr 172.21.15.98:0/68804420 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/68804420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 0x5576c3ed8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ae8b6640 1 --2- 172.21.15.98:0/68804420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 0x5576c3ef4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.080 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ad8b4640 1 -- 172.21.15.98:0/68804420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 msgr2=0x5576c3ef4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ad8b4640 1 --2- 172.21.15.98:0/68804420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 0x5576c3ef4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ad8b4640 1 -- 172.21.15.98:0/68804420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 msgr2=0x5576c3ed8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ad8b4640 1 --2- 172.21.15.98:0/68804420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 0x5576c3ed8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ad8b4640 1 -- 172.21.15.98:0/68804420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576c3ed2810 con 0x5576c3ed1f40 2024-04-25T17:49:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f64ad8b4640 1 --2- 172.21.15.98:0/68804420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3ef44c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6498005fd0 tx=0x7f6498011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444fe2571feb3ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f6493fff640 1 -- 172.21.15.98:0/68804420 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649801b3f0 con 0x5576c3ed1f40 2024-04-25T17:49:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f6493fff640 1 -- 172.21.15.98:0/68804420 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6498020070 con 0x5576c3ed1f40 2024-04-25T17:49:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.069+0000 7f6493fff640 1 -- 172.21.15.98:0/68804420 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6498010940 con 0x5576c3ed1f40 2024-04-25T17:49:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 -- 172.21.15.98:0/68804420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 msgr2=0x5576c3ef44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 --2- 172.21.15.98:0/68804420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3ef44c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6498005fd0 tx=0x7f6498011810 comp rx=0 tx=0).stop 2024-04-25T17:49:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 -- 172.21.15.98:0/68804420 shutdown_connections 2024-04-25T17:49:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 --2- 172.21.15.98:0/68804420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 0x5576c3ef4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 --2- 172.21.15.98:0/68804420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1f40 0x5576c3ef44c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 --2- 172.21.15.98:0/68804420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1620 0x5576c3ed8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 -- 172.21.15.98:0/68804420 >> 172.21.15.98:0/68804420 conn(0x5576c3ecdb30 msgr2=0x5576c3ecc420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 -- 172.21.15.98:0/68804420 shutdown_connections 2024-04-25T17:49:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 -- 172.21.15.98:0/68804420 wait complete. 2024-04-25T17:49:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 -- start start 2024-04-25T17:49:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 Processor -- start 2024-04-25T17:49:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1620 0x5576c3ee44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1f40 0x5576c3ee4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 0x5576c3ee1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81ac1d0 con 0x5576c3ed1620 2024-04-25T17:49:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81ad640 con 0x5576c3ed1f40 2024-04-25T17:49:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64a81ac8a0 con 0x5576c3edc800 2024-04-25T17:49:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64b302b740 10 client.0 _ll_register_callbacks cb 0x7f64a0031e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64ae0b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1620 0x5576c3ee44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64ae0b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1620 0x5576c3ee44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38444/0 (socket says 172.21.15.98:38444) 2024-04-25T17:49:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.070+0000 7f64ae0b5640 1 -- 172.21.15.98:0/2404479901 learned_addr learned my addr 172.21.15.98:0/2404479901 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:16.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64ad8b4640 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1f40 0x5576c3ee4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64ae8b6640 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 0x5576c3ee1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64ae0b5640 1 -- 172.21.15.98:0/2404479901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 msgr2=0x5576c3ee1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 0x5576c3ee1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64ae0b5640 1 -- 172.21.15.98:0/2404479901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1f40 msgr2=0x5576c3ee4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:16.086 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1f40 0x5576c3ee4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64ae0b5640 1 -- 172.21.15.98:0/2404479901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576c3ee2110 con 0x5576c3ed1620 2024-04-25T17:49:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64ae8b6640 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 0x5576c3ee1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:49:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64ae0b5640 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1620 0x5576c3ee44f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f64a4007f50 tx=0x7f64a409beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286f31d596a554fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f6491ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:16.087 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f6491ffb640 1 -- 172.21.15.98:0/2404479901 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a401d040 con 0x5576c3ed1620 2024-04-25T17:49:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f6491ffb640 1 -- 172.21.15.98:0/2404479901 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7f64a4098ae0 con 0x5576c3ed1620 2024-04-25T17:49:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64b302b740 1 -- 172.21.15.98:0/2404479901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6498020070 con 0x5576c3ed1620 2024-04-25T17:49:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f6491ffb640 1 -- 172.21.15.98:0/2404479901 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f64a40b0e60 con 0x5576c3ed1620 2024-04-25T17:49:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64b302b740 10 client.16764 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64b302b740 1 -- 172.21.15.98:0/2404479901 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x5576c3f527d0 con 0x5576c3ed1620 2024-04-25T17:49:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f6491ffb640 1 -- 172.21.15.98:0/2404479901 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6498020070 con 0x5576c3ed1620 2024-04-25T17:49:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f6491ffb640 1 -- 172.21.15.98:0/2404479901 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7f64a409c6d0 con 0x5576c3ed1620 2024-04-25T17:49:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64b302b740 10 client.16764 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64b302b740 10 client.16764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:49:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64b302b740 20 client.16764 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:16.089 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64b302b740 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5576c3f52a60 0x5576c3f54e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64b302b740 4 client.16764 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:49:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.071+0000 7f64b302b740 1 -- 172.21.15.98:0/2404479901 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5576c3dba5f0 con 0x5576c3f52a60 2024-04-25T17:49:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.072+0000 7f64ad8b4640 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5576c3f52a60 0x5576c3f54e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.072+0000 7f6491ffb640 10 client.16764 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.072+0000 7f64ad8b4640 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5576c3f52a60 0x5576c3f54e50 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=602a81d60cbbb4bf server_cookie=c35633c1936a534a in_seq=0 out_seq=0 2024-04-25T17:49:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.079+0000 7f6491ffb640 1 -- 172.21.15.98:0/2404479901 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5576c3dba5f0 con 0x5576c3f52a60 2024-04-25T17:49:16.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.079+0000 7f6491ffb640 10 client.16764 handle_command_reply: tid=0 2024-04-25T17:49:16.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.079+0000 7f64b302b740 1 client.16764 shutdown 2024-04-25T17:49:16.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.079+0000 7f64b302b740 1 -- 172.21.15.98:0/2404479901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1620 msgr2=0x5576c3ee44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:16.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.079+0000 7f64b302b740 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1620 0x5576c3ee44f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f64a4007f50 tx=0x7f64a409beb0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b302b740 1 -- 172.21.15.98:0/2404479901 shutdown_connections 2024-04-25T17:49:16.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b302b740 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5576c3f52a60 0x5576c3f54e50 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b302b740 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5576c3edc800 0x5576c3ee1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.093 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b302b740 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5576c3ed1f40 0x5576c3ee4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f6491ffb640 0 client.16764 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:16.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b302b740 1 --2- 172.21.15.98:0/2404479901 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5576c3ed1620 0x5576c3ee44f0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b302b740 1 -- 172.21.15.98:0/2404479901 >> 172.21.15.98:0/2404479901 conn(0x5576c3ecdb30 msgr2=0x5576c3ed8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:16.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b302b740 1 -- 172.21.15.98:0/2404479901 shutdown_connections 2024-04-25T17:49:16.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b302b740 1 -- 172.21.15.98:0/2404479901 wait complete. 2024-04-25T17:49:16.094 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b302b740 20 client.16764 trim_cache size 0 max 16384 2024-04-25T17:49:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:49:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:49:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:49:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:49:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 -- 172.21.15.98:0/3570924668 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f64940a3e40 msgr2=0x7f64940a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f64940a3e40 0x7f64940a6300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f64a002f500 tx=0x7f64a002f310 comp rx=0 tx=0).stop 2024-04-25T17:49:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 -- 172.21.15.98:0/3570924668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a81010c0 msgr2=0x7f64a8198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a81010c0 0x7f64a8198690 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f64a400c890 tx=0x7f64a400cd50 comp rx=0 tx=0).stop 2024-04-25T17:49:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 -- 172.21.15.98:0/3570924668 shutdown_connections 2024-04-25T17:49:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f64940a3e40 0x7f64940a6300 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64a819cf00 0x7f64a819d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64a8106190 0x7f64a8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 --2- 172.21.15.98:0/3570924668 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64a81010c0 0x7f64a8198690 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 -- 172.21.15.98:0/3570924668 >> 172.21.15.98:0/3570924668 conn(0x7f64a8077fd0 msgr2=0x7f64a80feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 -- 172.21.15.98:0/3570924668 shutdown_connections 2024-04-25T17:49:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:16.080+0000 7f64b0319640 1 -- 172.21.15.98:0/3570924668 wait complete. 2024-04-25T17:49:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:16 smithi098 ceph-mon[29192]: pgmap v421: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 205 KiB/s wr, 34 op/s 2024-04-25T17:49:16.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:16 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:49:16.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:16 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:49:16.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:16 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:49:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:16 smithi203 ceph-mon[32834]: pgmap v421: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 205 KiB/s wr, 34 op/s 2024-04-25T17:49:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:16 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:49:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:16 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:49:16.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:16 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:49:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:16 smithi173 ceph-mon[32856]: pgmap v421: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 205 KiB/s wr, 34 op/s 2024-04-25T17:49:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:16 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:49:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:16 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:49:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:16 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:49:16.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 765.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:49:18.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:18 smithi098 ceph-mon[29192]: pgmap v422: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T17:49:18.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:18 smithi203 ceph-mon[32834]: pgmap v422: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T17:49:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:18 smithi173 ceph-mon[32856]: pgmap v422: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T17:49:20.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:20 smithi098 ceph-mon[29192]: pgmap v423: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T17:49:20.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:20 smithi203 ceph-mon[32834]: pgmap v423: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T17:49:20.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:20 smithi173 ceph-mon[32856]: pgmap v423: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 37 KiB/s wr, 10 op/s 2024-04-25T17:49:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 770.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:49:22.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:22 smithi098 ceph-mon[29192]: pgmap v424: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 36 KiB/s wr, 10 op/s 2024-04-25T17:49:22.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:22 smithi203 ceph-mon[32834]: pgmap v424: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 36 KiB/s wr, 10 op/s 2024-04-25T17:49:22.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:22 smithi173 ceph-mon[32856]: pgmap v424: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 36 KiB/s wr, 10 op/s 2024-04-25T17:49:24.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:49:24.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:24 smithi098 ceph-mon[29192]: pgmap v425: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-04-25T17:49:24.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:24 smithi203 ceph-mon[32834]: pgmap v425: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-04-25T17:49:24.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:24 smithi173 ceph-mon[32856]: pgmap v425: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-04-25T17:49:24.586 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:49:26.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:26 smithi098 ceph-mon[29192]: pgmap v426: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-25T17:49:26.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:26 smithi203 ceph-mon[32834]: pgmap v426: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-25T17:49:26.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:26 smithi173 ceph-mon[32856]: pgmap v426: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.561+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3472350239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2c108a70 msgr2=0x7f3d2c108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.561+0000 7f3d32e2e700 1 --2- 172.21.15.98:0/3472350239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2c108a70 0x7f3d2c108e50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009990 tx=0x7f3d20009ca0 comp rx=0 tx=0).stop 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.562+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3472350239 shutdown_connections 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.562+0000 7f3d32e2e700 1 --2- 172.21.15.98:0/3472350239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d2c103480 0x7f3d2c10f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.562+0000 7f3d32e2e700 1 --2- 172.21.15.98:0/3472350239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2c108a70 0x7f3d2c108e50 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.562+0000 7f3d32e2e700 1 --2- 172.21.15.98:0/3472350239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d2c102ac0 0x7f3d2c102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.562+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3472350239 >> 172.21.15.98:0/3472350239 conn(0x7f3d2c079bb0 msgr2=0x7f3d2c079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.562+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3472350239 shutdown_connections 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.562+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3472350239 wait complete. 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.562+0000 7f3d32e2e700 1 Processor -- start 2024-04-25T17:49:26.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d32e2e700 1 -- start start 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d32e2e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d2c102ac0 0x7f3d2c199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d32e2e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2c103480 0x7f3d2c1996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d32e2e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d2c108a70 0x7f3d2c19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d32e2e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c112c80 con 0x7f3d2c103480 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d32e2e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c112b00 con 0x7f3d2c108a70 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d32e2e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c112e00 con 0x7f3d2c102ac0 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2c103480 0x7f3d2c1996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d2bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2c103480 0x7f3d2c1996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43790/0 (socket says 172.21.15.98:43790) 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d2bfff700 1 -- 172.21.15.98:0/3917779794 learned_addr learned my addr 172.21.15.98:0/3917779794 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d30bca700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d2c102ac0 0x7f3d2c199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d313cb700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d2c108a70 0x7f3d2c19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d2bfff700 1 -- 172.21.15.98:0/3917779794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d2c102ac0 msgr2=0x7f3d2c199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d2bfff700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d2c102ac0 0x7f3d2c199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:26.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d2bfff700 1 -- 172.21.15.98:0/3917779794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d2c108a70 msgr2=0x7f3d2c19da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d2bfff700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d2c108a70 0x7f3d2c19da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d2bfff700 1 -- 172.21.15.98:0/3917779794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2c19e1a0 con 0x7f3d2c103480 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d313cb700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d2c108a70 0x7f3d2c19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.563+0000 7f3d2bfff700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2c103480 0x7f3d2c1996d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800b530 tx=0x7f3d1800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.564+0000 7f3d29ffb700 1 -- 172.21.15.98:0/3917779794 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18015020 con 0x7f3d2c103480 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.564+0000 7f3d29ffb700 1 -- 172.21.15.98:0/3917779794 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d18011ce0 con 0x7f3d2c103480 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.564+0000 7f3d29ffb700 1 -- 172.21.15.98:0/3917779794 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18010e40 con 0x7f3d2c103480 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.564+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3917779794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2c19e490 con 0x7f3d2c103480 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.564+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3917779794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2c1a5d10 con 0x7f3d2c103480 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.565+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3917779794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d2c069730 con 0x7f3d2c103480 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.566+0000 7f3d29ffb700 1 -- 172.21.15.98:0/3917779794 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3d18018020 con 0x7f3d2c103480 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.567+0000 7f3d29ffb700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3d1c0a3e30 0x7f3d1c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:26.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.567+0000 7f3d30bca700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3d1c0a3e30 0x7f3d1c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:26.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.567+0000 7f3d29ffb700 1 -- 172.21.15.98:0/3917779794 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7f3d180cff10 con 0x7f3d2c103480 2024-04-25T17:49:26.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.568+0000 7f3d30bca700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3d1c0a3e30 0x7f3d1c0a62f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000b5c0 tx=0x7f3d20005960 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.574+0000 7f3d29ffb700 1 -- 172.21.15.98:0/3917779794 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d1801f020 con 0x7f3d2c103480 2024-04-25T17:49:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 775.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:49:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.820+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3917779794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d2c02df80 con 0x7f3d2c103480 2024-04-25T17:49:26.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.821+0000 7f3d29ffb700 1 -- 172.21.15.98:0/3917779794 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+11878 (secure 0 0 0) 0x7f3d1809c6c0 con 0x7f3d2c103480 2024-04-25T17:49:26.821 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:49:26.821 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":51,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:48:59.915105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26026":{"gid":26026,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/3925531495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":3925531495},{"type":"v1","addr":"172.21.15.203:6837","nonce":3925531495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:49:26.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3917779794 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3d1c0a3e30 msgr2=0x7f3d1c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3d1c0a3e30 0x7f3d1c0a62f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000b5c0 tx=0x7f3d20005960 comp rx=0 tx=0).stop 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3917779794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2c103480 msgr2=0x7f3d2c1996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2c103480 0x7f3d2c1996d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800b530 tx=0x7f3d1800b840 comp rx=0 tx=0).stop 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3917779794 shutdown_connections 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3d1c0a3e30 0x7f3d1c0a62f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d2c102ac0 0x7f3d2c199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d2c108a70 0x7f3d2c19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 --2- 172.21.15.98:0/3917779794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d2c103480 0x7f3d2c1996d0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3917779794 >> 172.21.15.98:0/3917779794 conn(0x7f3d2c079bb0 msgr2=0x7f3d2c1018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3917779794 shutdown_connections 2024-04-25T17:49:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:26.823+0000 7f3d32e2e700 1 -- 172.21.15.98:0/3917779794 wait complete. 2024-04-25T17:49:26.824 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 51 2024-04-25T17:49:27.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:27 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3917779794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:49:27.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:27 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3917779794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:49:27.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:27 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3917779794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:49:27.598 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id a13e6163 --release --await' 2024-04-25T17:49:27.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id a13e6163 --release --await 2024-04-25T17:49:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:49:28.219 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:28 smithi098 ceph-mon[29192]: pgmap v427: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-25T17:49:28.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:28 smithi203 ceph-mon[32834]: pgmap v427: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-25T17:49:28.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:28 smithi173 ceph-mon[32856]: pgmap v427: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-25T17:49:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.177+0000 7fed91472640 1 -- 172.21.15.98:0/1264920550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c1031c0 msgr2=0x7fed8c1035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.177+0000 7fed91472640 1 --2- 172.21.15.98:0/1264920550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c1031c0 0x7fed8c1035a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fed7c009900 tx=0x7fed7c02a480 comp rx=0 tx=0).stop 2024-04-25T17:49:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.177+0000 7fed91472640 1 -- 172.21.15.98:0/1264920550 shutdown_connections 2024-04-25T17:49:29.192 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.177+0000 7fed91472640 1 --2- 172.21.15.98:0/1264920550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed8c10c590 0x7fed8c10e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.193 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.177+0000 7fed91472640 1 --2- 172.21.15.98:0/1264920550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c103ae0 0x7fed8c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.193 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.177+0000 7fed91472640 1 --2- 172.21.15.98:0/1264920550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c1031c0 0x7fed8c1035a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.193 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.177+0000 7fed91472640 1 -- 172.21.15.98:0/1264920550 >> 172.21.15.98:0/1264920550 conn(0x7fed8c078060 msgr2=0x7fed8c078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.193 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.177+0000 7fed91472640 1 -- 172.21.15.98:0/1264920550 shutdown_connections 2024-04-25T17:49:29.193 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.177+0000 7fed91472640 1 -- 172.21.15.98:0/1264920550 wait complete. 2024-04-25T17:49:29.193 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed91472640 1 Processor -- start 2024-04-25T17:49:29.193 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed91472640 1 -- start start 2024-04-25T17:49:29.193 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed91472640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c1031c0 0x7fed8c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.193 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed91472640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed8c103ae0 0x7fed8c198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed91472640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c10c590 0x7fed8c19cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed91472640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1121e0 con 0x7fed8c10c590 2024-04-25T17:49:29.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed91472640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c112060 con 0x7fed8c1031c0 2024-04-25T17:49:29.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed91472640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c112360 con 0x7fed8c103ae0 2024-04-25T17:49:29.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed8b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c10c590 0x7fed8c19cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.194 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed8b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c10c590 0x7fed8c19cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43804/0 (socket says 172.21.15.98:43804) 2024-04-25T17:49:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed8b7fe640 1 -- 172.21.15.98:0/233951292 learned_addr learned my addr 172.21.15.98:0/233951292 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed8affd640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c1031c0 0x7fed8c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed8a7fc640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed8c103ae0 0x7fed8c198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed8b7fe640 1 -- 172.21.15.98:0/233951292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed8c103ae0 msgr2=0x7fed8c198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed8c103ae0 0x7fed8c198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed8b7fe640 1 -- 172.21.15.98:0/233951292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c1031c0 msgr2=0x7fed8c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.195 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c1031c0 0x7fed8c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.178+0000 7fed8b7fe640 1 -- 172.21.15.98:0/233951292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8c19d6a0 con 0x7fed8c10c590 2024-04-25T17:49:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.179+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c10c590 0x7fed8c19cf70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fed8400eb10 tx=0x7fed8400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.179+0000 7fed73fff640 1 -- 172.21.15.98:0/233951292 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84004b20 con 0x7fed8c10c590 2024-04-25T17:49:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.179+0000 7fed73fff640 1 -- 172.21.15.98:0/233951292 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed84004cc0 con 0x7fed8c10c590 2024-04-25T17:49:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.179+0000 7fed91472640 1 -- 172.21.15.98:0/233951292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed8c19d990 con 0x7fed8c10c590 2024-04-25T17:49:29.196 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.179+0000 7fed73fff640 1 -- 172.21.15.98:0/233951292 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed84010670 con 0x7fed8c10c590 2024-04-25T17:49:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.179+0000 7fed91472640 1 -- 172.21.15.98:0/233951292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed8c1a5220 con 0x7fed8c10c590 2024-04-25T17:49:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.181+0000 7fed73fff640 1 -- 172.21.15.98:0/233951292 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fed84010810 con 0x7fed8c10c590 2024-04-25T17:49:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.181+0000 7fed94184740 1 Processor -- start 2024-04-25T17:49:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.181+0000 7fed94184740 1 -- start start 2024-04-25T17:49:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed94184740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc498620 0x55cdfc498a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed73fff640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fed740a3dc0 0x7fed740a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed94184740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498f40 0x55cdfc4a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed94184740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed94184740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cdfc3fd970 con 0x55cdfc4a3800 2024-04-25T17:49:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed94184740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cdfc39a4a0 con 0x55cdfc498f40 2024-04-25T17:49:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed94184740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cdfc3f9930 con 0x55cdfc498620 2024-04-25T17:49:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8affd640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fed740a3dc0 0x7fed740a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8a7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498f40 0x55cdfc4a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43814/0 (socket says 172.21.15.98:43814) 2024-04-25T17:49:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8b7fe640 1 -- 172.21.15.98:0/421820902 learned_addr learned my addr 172.21.15.98:0/421820902 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8a7fc640 1 --2- 172.21.15.98:0/421820902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc498620 0x55cdfc498a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed73fff640 1 -- 172.21.15.98:0/233951292 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7fed840d0e70 con 0x7fed8c10c590 2024-04-25T17:49:29.199 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8b7fe640 1 -- 172.21.15.98:0/421820902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc498620 msgr2=0x55cdfc498a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/421820902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc498620 0x55cdfc498a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8b7fe640 1 -- 172.21.15.98:0/421820902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498f40 msgr2=0x55cdfc4a32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/421820902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498f40 0x55cdfc4a32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8b7fe640 1 -- 172.21.15.98:0/421820902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdfc110210 con 0x55cdfc4a3800 2024-04-25T17:49:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.182+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/421820902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a5bf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fed840049d0 tx=0x7fed84013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfaee1dfe65a0ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.183+0000 7fed8affd640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fed740a3dc0 0x7fed740a6280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fed7c031440 tx=0x7fed7c0095f0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.183+0000 7fed717fa640 1 -- 172.21.15.98:0/421820902 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8409c950 con 0x55cdfc4a3800 2024-04-25T17:49:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.183+0000 7fed717fa640 1 -- 172.21.15.98:0/421820902 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed8409d510 con 0x55cdfc4a3800 2024-04-25T17:49:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.183+0000 7fed717fa640 1 -- 172.21.15.98:0/421820902 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8407d000 con 0x55cdfc4a3800 2024-04-25T17:49:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 -- 172.21.15.98:0/421820902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 msgr2=0x55cdfc4a5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 --2- 172.21.15.98:0/421820902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a5bf0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fed840049d0 tx=0x7fed84013000 comp rx=0 tx=0).stop 2024-04-25T17:49:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 -- 172.21.15.98:0/421820902 shutdown_connections 2024-04-25T17:49:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 --2- 172.21.15.98:0/421820902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a5bf0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 --2- 172.21.15.98:0/421820902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498f40 0x55cdfc4a32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 --2- 172.21.15.98:0/421820902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc498620 0x55cdfc498a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 -- 172.21.15.98:0/421820902 >> 172.21.15.98:0/421820902 conn(0x55cdfc494b30 msgr2=0x55cdfc494f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 -- 172.21.15.98:0/421820902 shutdown_connections 2024-04-25T17:49:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 -- 172.21.15.98:0/421820902 wait complete. 2024-04-25T17:49:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 -- start start 2024-04-25T17:49:29.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.184+0000 7fed94184740 1 Processor -- start 2024-04-25T17:49:29.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed94184740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 0x55cdfc4a3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed94184740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4a0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed94184740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed94184740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cdfc4aa840 con 0x55cdfc498f40 2024-04-25T17:49:29.203 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed94184740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cdfc3980b0 con 0x55cdfc498620 2024-04-25T17:49:29.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed94184740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cdfc4aa9c0 con 0x55cdfc4a3800 2024-04-25T17:49:29.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed94184740 10 client.0 _ll_register_callbacks cb 0x55cdfc385d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:29.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43828/0 (socket says 172.21.15.98:43828) 2024-04-25T17:49:29.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8a7fc640 1 -- 172.21.15.98:0/2213363300 learned_addr learned my addr 172.21.15.98:0/2213363300 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:29.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8affd640 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 0x55cdfc4a3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8a7fc640 1 -- 172.21.15.98:0/2213363300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 msgr2=0x55cdfc4a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.204 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8a7fc640 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8a7fc640 1 -- 172.21.15.98:0/2213363300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 msgr2=0x55cdfc4a3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8a7fc640 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 0x55cdfc4a3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8a7fc640 1 -- 172.21.15.98:0/2213363300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdfc1dc620 con 0x55cdfc498f40 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed8a7fc640 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4a0420 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fed80010b60 tx=0x7fed8000eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a4ae1710b3e6c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed6eff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed6eff5640 1 -- 172.21.15.98:0/2213363300 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed80010e50 con 0x55cdfc498f40 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed6eff5640 1 -- 172.21.15.98:0/2213363300 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7fed8001e480 con 0x55cdfc498f40 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.185+0000 7fed94184740 1 -- 172.21.15.98:0/2213363300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed8409d510 con 0x55cdfc498f40 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed6eff5640 1 -- 172.21.15.98:0/2213363300 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fed8001e680 con 0x55cdfc498f40 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed94184740 10 client.16782 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed94184740 1 -- 172.21.15.98:0/2213363300 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55cdfc4b67b0 con 0x55cdfc498f40 2024-04-25T17:49:29.205 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed6eff5640 1 -- 172.21.15.98:0/2213363300 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed8409d510 con 0x55cdfc498f40 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed6eff5640 1 -- 172.21.15.98:0/2213363300 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7fed8000f040 con 0x55cdfc498f40 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed94184740 10 client.16782 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed94184740 10 client.16782 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed94184740 20 client.16782 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed94184740 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cdfc4b7c20 0x55cdfc4ba010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed94184740 4 client.16782 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed94184740 1 -- 172.21.15.98:0/2213363300 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdfc40b390 con 0x55cdfc4b7c20 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.186+0000 7fed8affd640 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cdfc4b7c20 0x55cdfc4ba010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.187+0000 7fed8affd640 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cdfc4b7c20 0x55cdfc4ba010 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d747b1126a591429 server_cookie=1bd95ad9c6522f3c in_seq=0 out_seq=0 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.187+0000 7fed6eff5640 10 client.16782 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.187+0000 7fed6eff5640 1 -- 172.21.15.98:0/2213363300 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cdfc40b390 con 0x55cdfc4b7c20 2024-04-25T17:49:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.187+0000 7fed6eff5640 10 client.16782 handle_command_reply: tid=0 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.187+0000 7fed94184740 1 client.16782 shutdown 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 1 -- 172.21.15.98:0/2213363300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 msgr2=0x55cdfc4a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4a0420 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fed80010b60 tx=0x7fed8000eab0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 1 -- 172.21.15.98:0/2213363300 shutdown_connections 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cdfc4b7c20 0x55cdfc4ba010 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 0x55cdfc4a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4a0420 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 1 --2- 172.21.15.98:0/2213363300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 0x55cdfc4a3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 1 -- 172.21.15.98:0/2213363300 >> 172.21.15.98:0/2213363300 conn(0x55cdfc494b30 msgr2=0x55cdfc49a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed6eff5640 0 client.16782 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 1 -- 172.21.15.98:0/2213363300 shutdown_connections 2024-04-25T17:49:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 1 -- 172.21.15.98:0/2213363300 wait complete. 2024-04-25T17:49:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.188+0000 7fed94184740 20 client.16782 trim_cache size 0 max 16384 2024-04-25T17:49:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.202+0000 7fed94184740 1 Processor -- start 2024-04-25T17:49:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.202+0000 7fed94184740 1 -- start start 2024-04-25T17:49:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.202+0000 7fed94184740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc498620 0x55cdfc4b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.202+0000 7fed94184740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498f40 0x55cdfc493b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.202+0000 7fed94184740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4bf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.202+0000 7fed94184740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1ae2b0 con 0x55cdfc4a3800 2024-04-25T17:49:29.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.202+0000 7fed94184740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1abb80 con 0x55cdfc498f40 2024-04-25T17:49:29.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.202+0000 7fed94184740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1a9a10 con 0x55cdfc498620 2024-04-25T17:49:29.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4bf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43836/0 (socket says 172.21.15.98:43836) 2024-04-25T17:49:29.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8b7fe640 1 -- 172.21.15.98:0/352930399 learned_addr learned my addr 172.21.15.98:0/352930399 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:29.214 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8affd640 1 --2- 172.21.15.98:0/352930399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc498620 0x55cdfc4b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8a7fc640 1 --2- 172.21.15.98:0/352930399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498f40 0x55cdfc493b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8b7fe640 1 -- 172.21.15.98:0/352930399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc498620 msgr2=0x55cdfc4b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/352930399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc498620 0x55cdfc4b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8b7fe640 1 -- 172.21.15.98:0/352930399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498f40 msgr2=0x55cdfc493b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/352930399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498f40 0x55cdfc493b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8b7fe640 1 -- 172.21.15.98:0/352930399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdfc4902e0 con 0x55cdfc4a3800 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/352930399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4bf920 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fed840997e0 tx=0x7fed8400efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d0852b0e6f3b3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed70ff9640 1 -- 172.21.15.98:0/352930399 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed840aa070 con 0x55cdfc4a3800 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed70ff9640 1 -- 172.21.15.98:0/352930399 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed8409ce10 con 0x55cdfc4a3800 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.203+0000 7fed70ff9640 1 -- 172.21.15.98:0/352930399 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed840afed0 con 0x55cdfc4a3800 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 -- 172.21.15.98:0/352930399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 msgr2=0x55cdfc4bf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 --2- 172.21.15.98:0/352930399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4bf920 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fed840997e0 tx=0x7fed8400efd0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 -- 172.21.15.98:0/352930399 shutdown_connections 2024-04-25T17:49:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 --2- 172.21.15.98:0/352930399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc4a3800 0x55cdfc4bf920 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 --2- 172.21.15.98:0/352930399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498f40 0x55cdfc493b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 --2- 172.21.15.98:0/352930399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc498620 0x55cdfc4b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 -- 172.21.15.98:0/352930399 >> 172.21.15.98:0/352930399 conn(0x55cdfc494b30 msgr2=0x55cdfc493420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 -- 172.21.15.98:0/352930399 shutdown_connections 2024-04-25T17:49:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 -- 172.21.15.98:0/352930399 wait complete. 2024-04-25T17:49:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 -- start start 2024-04-25T17:49:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 Processor -- start 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 0x55cdfc4ace30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4ad370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 0x55cdfc51c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1aa060 con 0x55cdfc498f40 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1ab4b0 con 0x55cdfc498620 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c1aa710 con 0x55cdfc4a3800 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed94184740 10 client.0 _ll_register_callbacks cb 0x7fed80010860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.204+0000 7fed8a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4ad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4ad370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43840/0 (socket says 172.21.15.98:43840) 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8a7fc640 1 -- 172.21.15.98:0/3349171219 learned_addr learned my addr 172.21.15.98:0/3349171219 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 0x55cdfc51c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8affd640 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 0x55cdfc4ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8a7fc640 1 -- 172.21.15.98:0/3349171219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 msgr2=0x55cdfc51c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8a7fc640 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 0x55cdfc51c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8a7fc640 1 -- 172.21.15.98:0/3349171219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 msgr2=0x55cdfc4ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8a7fc640 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 0x55cdfc4ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8a7fc640 1 -- 172.21.15.98:0/3349171219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdfc51c7a0 con 0x55cdfc498f40 2024-04-25T17:49:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8b7fe640 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 0x55cdfc51c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:49:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8affd640 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 0x55cdfc4ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:49:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed8a7fc640 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4ad370 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fed80010b60 tx=0x7fed8000f2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b723ba0238a24c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed6eff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed6eff5640 1 -- 172.21.15.98:0/3349171219 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8000fe70 con 0x55cdfc498f40 2024-04-25T17:49:29.218 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed6eff5640 1 -- 172.21.15.98:0/3349171219 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7fed80027a70 con 0x55cdfc498f40 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed94184740 1 -- 172.21.15.98:0/3349171219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cdfc405200 con 0x55cdfc498f40 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed6eff5640 1 -- 172.21.15.98:0/3349171219 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fed80005850 con 0x55cdfc498f40 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed94184740 10 client.16788 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed94184740 1 -- 172.21.15.98:0/3349171219 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55cdfc51dab0 con 0x55cdfc498f40 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.205+0000 7fed6eff5640 1 -- 172.21.15.98:0/3349171219 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55cdfc405200 con 0x55cdfc498f40 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.206+0000 7fed6eff5640 1 -- 172.21.15.98:0/3349171219 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7fed8002f030 con 0x55cdfc498f40 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.206+0000 7fed94184740 10 client.16788 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.206+0000 7fed94184740 10 client.16788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.206+0000 7fed94184740 20 client.16788 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.206+0000 7fed94184740 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cdfc51e240 0x55cdfc520660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.206+0000 7fed94184740 4 client.16788 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.206+0000 7fed94184740 1 -- 172.21.15.98:0/3349171219 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cdfc3815f0 con 0x55cdfc51e240 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.206+0000 7fed8affd640 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cdfc51e240 0x55cdfc520660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.206+0000 7fed6eff5640 10 client.16788 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:29.219 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.206+0000 7fed8affd640 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cdfc51e240 0x55cdfc520660 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=313816f16df4ec18 server_cookie=5b854bb2aa45df38 in_seq=0 out_seq=0 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.214+0000 7fed6eff5640 1 -- 172.21.15.98:0/3349171219 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cdfc3815f0 con 0x55cdfc51e240 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.214+0000 7fed6eff5640 10 client.16788 handle_command_reply: tid=0 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 client.16788 shutdown 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 -- 172.21.15.98:0/3349171219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 msgr2=0x55cdfc4ad370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4ad370 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fed80010b60 tx=0x7fed8000f2b0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 -- 172.21.15.98:0/3349171219 shutdown_connections 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cdfc51e240 0x55cdfc520660 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cdfc4a3800 0x55cdfc51c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cdfc498f40 0x55cdfc4ad370 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed6eff5640 0 client.16788 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 --2- 172.21.15.98:0/3349171219 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cdfc498620 0x55cdfc4ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 -- 172.21.15.98:0/3349171219 >> 172.21.15.98:0/3349171219 conn(0x55cdfc494b30 msgr2=0x55cdfc493cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 -- 172.21.15.98:0/3349171219 shutdown_connections 2024-04-25T17:49:29.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 1 -- 172.21.15.98:0/3349171219 wait complete. 2024-04-25T17:49:29.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed94184740 20 client.16788 trim_cache size 0 max 16384 2024-04-25T17:49:29.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:49:29.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:49:29.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:49:29.222 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:49:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed91472640 1 -- 172.21.15.98:0/233951292 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fed740a3dc0 msgr2=0x7fed740a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed91472640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fed740a3dc0 0x7fed740a6280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fed7c031440 tx=0x7fed7c0095f0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed91472640 1 -- 172.21.15.98:0/233951292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c10c590 msgr2=0x7fed8c19cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.223 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed91472640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c10c590 0x7fed8c19cf70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fed8400eb10 tx=0x7fed8400c2d0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed91472640 1 -- 172.21.15.98:0/233951292 shutdown_connections 2024-04-25T17:49:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed91472640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fed740a3dc0 0x7fed740a6280 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed91472640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed8c10c590 0x7fed8c19cf70 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed91472640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed8c103ae0 0x7fed8c198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed91472640 1 --2- 172.21.15.98:0/233951292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed8c1031c0 0x7fed8c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.215+0000 7fed91472640 1 -- 172.21.15.98:0/233951292 >> 172.21.15.98:0/233951292 conn(0x7fed8c078060 msgr2=0x7fed8c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.216+0000 7fed91472640 1 -- 172.21.15.98:0/233951292 shutdown_connections 2024-04-25T17:49:29.224 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:29.216+0000 7fed91472640 1 -- 172.21.15.98:0/233951292 wait complete. 2024-04-25T17:49:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.638+0000 7fde4314f700 1 -- 172.21.15.98:0/3325762801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c103450 msgr2=0x7fde3c10f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.638+0000 7fde4314f700 1 --2- 172.21.15.98:0/3325762801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c103450 0x7fde3c10f8b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fde3400b0a0 tx=0x7fde3400b3b0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.638+0000 7fde4314f700 1 -- 172.21.15.98:0/3325762801 shutdown_connections 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.638+0000 7fde4314f700 1 --2- 172.21.15.98:0/3325762801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde3c108a50 0x7fde3c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.638+0000 7fde4314f700 1 --2- 172.21.15.98:0/3325762801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c103450 0x7fde3c10f8b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.638+0000 7fde4314f700 1 --2- 172.21.15.98:0/3325762801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde3c102a90 0x7fde3c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.638+0000 7fde4314f700 1 -- 172.21.15.98:0/3325762801 >> 172.21.15.98:0/3325762801 conn(0x7fde3c079ba0 msgr2=0x7fde3c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.638+0000 7fde4314f700 1 -- 172.21.15.98:0/3325762801 shutdown_connections 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.638+0000 7fde4314f700 1 -- 172.21.15.98:0/3325762801 wait complete. 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde4314f700 1 Processor -- start 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde4314f700 1 -- start start 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde4314f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c102a90 0x7fde3c198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde4314f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde3c103450 0x7fde3c1994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde4314f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde3c108a50 0x7fde3c19d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde4314f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c112be0 con 0x7fde3c102a90 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde4314f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c112a60 con 0x7fde3c108a50 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde4314f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c112d60 con 0x7fde3c103450 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde40eeb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c102a90 0x7fde3c198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde40eeb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c102a90 0x7fde3c198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43846/0 (socket says 172.21.15.98:43846) 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde40eeb700 1 -- 172.21.15.98:0/2610022986 learned_addr learned my addr 172.21.15.98:0/2610022986 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde33fff700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde3c103450 0x7fde3c1994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde40eeb700 1 -- 172.21.15.98:0/2610022986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde3c103450 msgr2=0x7fde3c1994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde40eeb700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde3c103450 0x7fde3c1994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde40eeb700 1 -- 172.21.15.98:0/2610022986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde3c108a50 msgr2=0x7fde3c19d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde40eeb700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde3c108a50 0x7fde3c19d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde40eeb700 1 -- 172.21.15.98:0/2610022986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c19df50 con 0x7fde3c102a90 2024-04-25T17:49:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.639+0000 7fde33fff700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde3c103450 0x7fde3c1994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:49:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.640+0000 7fde40eeb700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c102a90 0x7fde3c198f90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fde3800c9b0 tx=0x7fde3800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.640+0000 7fde31ffb700 1 -- 172.21.15.98:0/2610022986 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38007b00 con 0x7fde3c102a90 2024-04-25T17:49:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.640+0000 7fde4314f700 1 -- 172.21.15.98:0/2610022986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3c19e240 con 0x7fde3c102a90 2024-04-25T17:49:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.640+0000 7fde31ffb700 1 -- 172.21.15.98:0/2610022986 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde38007ca0 con 0x7fde3c102a90 2024-04-25T17:49:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.640+0000 7fde31ffb700 1 -- 172.21.15.98:0/2610022986 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38018730 con 0x7fde3c102a90 2024-04-25T17:49:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.640+0000 7fde4314f700 1 -- 172.21.15.98:0/2610022986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3c1a5ac0 con 0x7fde3c102a90 2024-04-25T17:49:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.642+0000 7fde31ffb700 1 -- 172.21.15.98:0/2610022986 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fde3800f450 con 0x7fde3c102a90 2024-04-25T17:49:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.643+0000 7fde31ffb700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fde240a3db0 0x7fde240a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.643+0000 7fde33fff700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fde240a3db0 0x7fde240a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.643+0000 7fde31ffb700 1 -- 172.21.15.98:0/2610022986 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7fde380d0800 con 0x7fde3c102a90 2024-04-25T17:49:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.643+0000 7fde33fff700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fde240a3db0 0x7fde240a6270 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fde2c005fd0 tx=0x7fde2c005e20 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde4d9d8b80 1 Processor -- start 2024-04-25T17:49:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde4d9d8b80 1 -- start start 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bb650 0x55b1d20bba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bbf70 0x55b1d20c6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b1d200ed90 con 0x55b1d20c6e70 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b1d1fb5a30 con 0x55b1d20bb650 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b1d200ad50 con 0x55b1d20bbf70 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde416ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde416ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43854/0 (socket says 172.21.15.98:43854) 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde416ec700 1 -- 172.21.15.98:0/3484617961 learned_addr learned my addr 172.21.15.98:0/3484617961 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde40eeb700 1 --2- 172.21.15.98:0/3484617961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bb650 0x55b1d20bba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde33fff700 1 --2- 172.21.15.98:0/3484617961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bbf70 0x55b1d20c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde416ec700 1 -- 172.21.15.98:0/3484617961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bbf70 msgr2=0x55b1d20c6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde416ec700 1 --2- 172.21.15.98:0/3484617961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bbf70 0x55b1d20c6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde416ec700 1 -- 172.21.15.98:0/3484617961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bb650 msgr2=0x55b1d20bba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde416ec700 1 --2- 172.21.15.98:0/3484617961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bb650 0x55b1d20bba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.644+0000 7fde416ec700 1 -- 172.21.15.98:0/3484617961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1d1f79840 con 0x55b1d20c6e70 2024-04-25T17:49:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.645+0000 7fde416ec700 1 --2- 172.21.15.98:0/3484617961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c9260 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fde3400b070 tx=0x7fde34009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d4a84a03994804 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.645+0000 7fde22ffd700 1 -- 172.21.15.98:0/3484617961 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34009350 con 0x55b1d20c6e70 2024-04-25T17:49:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.645+0000 7fde22ffd700 1 -- 172.21.15.98:0/3484617961 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde340074e0 con 0x55b1d20c6e70 2024-04-25T17:49:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.645+0000 7fde22ffd700 1 -- 172.21.15.98:0/3484617961 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3401d690 con 0x55b1d20c6e70 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.645+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3484617961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 msgr2=0x55b1d20c9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.645+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3484617961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c9260 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fde3400b070 tx=0x7fde34009d80 comp rx=0 tx=0).stop 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.645+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3484617961 shutdown_connections 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.645+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3484617961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c9260 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.645+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3484617961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bb650 0x55b1d20bba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.645+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3484617961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bbf70 0x55b1d20c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3484617961 >> 172.21.15.98:0/3484617961 conn(0x55b1d1fe8990 msgr2=0x55b1d20b7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3484617961 shutdown_connections 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3484617961 wait complete. 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 -- start start 2024-04-25T17:49:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 Processor -- start 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bb650 0x55b1d20c6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d20c3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b1d20c98a0 con 0x55b1d20c6e70 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b1d1fc86b0 con 0x55b1d20bbf70 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b1d20c9a20 con 0x55b1d20bb650 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde4d9d8b80 10 client.0 _ll_register_callbacks cb 0x55b1d1fa7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde416ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde416ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43868/0 (socket says 172.21.15.98:43868) 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.646+0000 7fde416ec700 1 -- 172.21.15.98:0/3418689841 learned_addr learned my addr 172.21.15.98:0/3418689841 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde40eeb700 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bb650 0x55b1d20c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde33fff700 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d20c3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde416ec700 1 -- 172.21.15.98:0/3418689841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bb650 msgr2=0x55b1d20c6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde416ec700 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bb650 0x55b1d20c6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde416ec700 1 -- 172.21.15.98:0/3418689841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 msgr2=0x55b1d20c3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde416ec700 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d20c3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde416ec700 1 -- 172.21.15.98:0/3418689841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1d1f5c9d0 con 0x55b1d20c6e70 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde33fff700 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d20c3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde416ec700 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c3e90 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fde3400ba70 tx=0x7fde340060d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae72015dc3a9fda7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde1bfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde1bfff700 1 -- 172.21.15.98:0/3418689841 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34029030 con 0x55b1d20c6e70 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3418689841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde340074e0 con 0x55b1d20c6e70 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde1bfff700 1 -- 172.21.15.98:0/3418689841 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7fde3400f070 con 0x55b1d20c6e70 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde1bfff700 1 -- 172.21.15.98:0/3418689841 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fde34035050 con 0x55b1d20c6e70 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde4d9d8b80 10 client.16800 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3418689841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55b1d20d4120 con 0x55b1d20c6e70 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.647+0000 7fde1bfff700 1 -- 172.21.15.98:0/3418689841 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde340074e0 con 0x55b1d20c6e70 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.648+0000 7fde1bfff700 1 -- 172.21.15.98:0/3418689841 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7fde34011070 con 0x55b1d20c6e70 2024-04-25T17:49:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.648+0000 7fde4d9d8b80 10 client.16800 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.648+0000 7fde4d9d8b80 10 client.16800 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.648+0000 7fde4d9d8b80 20 client.16800 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.648+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b1d20d65f0 0x55b1d20d89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.648+0000 7fde4d9d8b80 4 client.16800 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.648+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3418689841 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1d201c580 con 0x55b1d20d65f0 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.648+0000 7fde40eeb700 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b1d20d65f0 0x55b1d20d89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.648+0000 7fde40eeb700 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b1d20d65f0 0x55b1d20d89e0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97a6f04f433c8c95 server_cookie=2665155cd3272ceb in_seq=0 out_seq=0 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.648+0000 7fde1bfff700 10 client.16800 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde1bfff700 1 -- 172.21.15.98:0/3418689841 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b1d201c580 con 0x55b1d20d65f0 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde1bfff700 10 client.16800 handle_command_reply: tid=0 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 client.16800 shutdown 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3418689841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 msgr2=0x55b1d20c3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c3e90 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fde3400ba70 tx=0x7fde340060d0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3418689841 shutdown_connections 2024-04-25T17:49:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b1d20d65f0 0x55b1d20d89e0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bb650 0x55b1d20c6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20c3e90 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3418689841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d20c3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3418689841 >> 172.21.15.98:0/3418689841 conn(0x55b1d1fe8990 msgr2=0x55b1d20bdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde1bfff700 0 client.16800 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3418689841 shutdown_connections 2024-04-25T17:49:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3418689841 wait complete. 2024-04-25T17:49:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.649+0000 7fde4d9d8b80 20 client.16800 trim_cache size 0 max 16384 2024-04-25T17:49:29.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.668+0000 7fde4d9d8b80 1 Processor -- start 2024-04-25T17:49:29.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde4d9d8b80 1 -- start start 2024-04-25T17:49:29.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bb650 0x55b1d20c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d20db630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20dbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1afa60 con 0x55b1d20c6e70 2024-04-25T17:49:29.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1ad2f0 con 0x55b1d20bbf70 2024-04-25T17:49:29.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1ab160 con 0x55b1d20bb650 2024-04-25T17:49:29.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde416ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20dbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde416ec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20dbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43880/0 (socket says 172.21.15.98:43880) 2024-04-25T17:49:29.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde416ec700 1 -- 172.21.15.98:0/2609659509 learned_addr learned my addr 172.21.15.98:0/2609659509 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:29.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde33fff700 1 --2- 172.21.15.98:0/2609659509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d20db630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde40eeb700 1 --2- 172.21.15.98:0/2609659509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bb650 0x55b1d20c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde416ec700 1 -- 172.21.15.98:0/2609659509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bb650 msgr2=0x55b1d20c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde416ec700 1 --2- 172.21.15.98:0/2609659509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bb650 0x55b1d20c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde416ec700 1 -- 172.21.15.98:0/2609659509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 msgr2=0x55b1d20db630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde416ec700 1 --2- 172.21.15.98:0/2609659509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d20db630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde416ec700 1 -- 172.21.15.98:0/2609659509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1d20bd0b0 con 0x55b1d20c6e70 2024-04-25T17:49:29.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde416ec700 1 --2- 172.21.15.98:0/2609659509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20dbb70 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fde3400ba40 tx=0x7fde34007b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b962f6af3dc905 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde227fc700 1 -- 172.21.15.98:0/2609659509 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34029070 con 0x55b1d20c6e70 2024-04-25T17:49:29.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde227fc700 1 -- 172.21.15.98:0/2609659509 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde340074a0 con 0x55b1d20c6e70 2024-04-25T17:49:29.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.669+0000 7fde227fc700 1 -- 172.21.15.98:0/2609659509 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde340123f0 con 0x55b1d20c6e70 2024-04-25T17:49:29.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.670+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/2609659509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 msgr2=0x55b1d20dbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.670+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/2609659509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20dbb70 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fde3400ba40 tx=0x7fde34007b50 comp rx=0 tx=0).stop 2024-04-25T17:49:29.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.670+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/2609659509 shutdown_connections 2024-04-25T17:49:29.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.670+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/2609659509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20bb650 0x55b1d20c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.670+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/2609659509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20c6e70 0x55b1d20dbb70 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.670+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/2609659509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d20db630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.670+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/2609659509 >> 172.21.15.98:0/2609659509 conn(0x55b1d1fe8990 msgr2=0x55b1d20c5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.670+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/2609659509 shutdown_connections 2024-04-25T17:49:29.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.670+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/2609659509 wait complete. 2024-04-25T17:49:29.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.670+0000 7fde4d9d8b80 1 -- start start 2024-04-25T17:49:29.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde4d9d8b80 1 Processor -- start 2024-04-25T17:49:29.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20bb650 0x55b1d2130940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d2130e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde4d9d8b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20c6e70 0x55b1d2135900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1ab7b0 con 0x55b1d20bb650 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1acc20 con 0x55b1d20bbf70 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde4d9d8b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1abe80 con 0x55b1d20c6e70 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde4d9d8b80 10 client.0 _ll_register_callbacks cb 0x55b1d1fa7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde40eeb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20bb650 0x55b1d2130940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde40eeb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20bb650 0x55b1d2130940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43890/0 (socket says 172.21.15.98:43890) 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde40eeb700 1 -- 172.21.15.98:0/3925685931 learned_addr learned my addr 172.21.15.98:0/3925685931 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde33fff700 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d2130e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde40eeb700 1 -- 172.21.15.98:0/3925685931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20c6e70 msgr2=0x55b1d2135900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde40eeb700 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20c6e70 0x55b1d2135900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde40eeb700 1 -- 172.21.15.98:0/3925685931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 msgr2=0x55b1d2130e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde40eeb700 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d2130e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.671+0000 7fde40eeb700 1 -- 172.21.15.98:0/3925685931 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1d2135ef0 con 0x55b1d20bb650 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde40eeb700 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20bb650 0x55b1d2130940 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fde3800ffd0 tx=0x7fde38005980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ba84a74dcf7cb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde1bfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde1bfff700 1 -- 172.21.15.98:0/3925685931 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde380a5410 con 0x55b1d20bb650 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde1bfff700 1 -- 172.21.15.98:0/3925685931 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7fde3809aaa0 con 0x55b1d20bb650 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3925685931 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde340074a0 con 0x55b1d20bb650 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde1bfff700 1 -- 172.21.15.98:0/3925685931 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fde380870b0 con 0x55b1d20bb650 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde4d9d8b80 10 client.16806 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3925685931 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55b1d2141720 con 0x55b1d20bb650 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde1bfff700 1 -- 172.21.15.98:0/3925685931 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde340074a0 con 0x55b1d20bb650 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde1bfff700 1 -- 172.21.15.98:0/3925685931 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7fde38085ce0 con 0x55b1d20bb650 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde4d9d8b80 10 client.16806 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde4d9d8b80 10 client.16806 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:49:29.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde4d9d8b80 20 client.16806 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:29.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b1d2141db0 0x55b1d21441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:29.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde4d9d8b80 4 client.16806 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a13e6163", "release": true, "await": true}] 2024-04-25T17:49:29.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.672+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3925685931 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a13e6163", "release": true, "await": true}) -- 0x55b1d2016620 con 0x55b1d2141db0 2024-04-25T17:49:29.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.673+0000 7fde33fff700 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b1d2141db0 0x55b1d21441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:29.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.673+0000 7fde33fff700 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b1d2141db0 0x55b1d21441f0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=859449847e7e625d server_cookie=28e356a4270da4a5 in_seq=0 out_seq=0 2024-04-25T17:49:29.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.673+0000 7fde1bfff700 10 client.16806 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:29.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde1bfff700 1 -- 172.21.15.98:0/3925685931 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55b1d2016620 con 0x55b1d2141db0 2024-04-25T17:49:29.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde1bfff700 10 client.16806 handle_command_reply: tid=0 2024-04-25T17:49:29.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde4d9d8b80 1 client.16806 shutdown 2024-04-25T17:49:29.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3925685931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20bb650 msgr2=0x55b1d2130940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20bb650 0x55b1d2130940 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fde3800ffd0 tx=0x7fde38005980 comp rx=0 tx=0).stop 2024-04-25T17:49:29.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3925685931 shutdown_connections 2024-04-25T17:49:29.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55b1d2141db0 0x55b1d21441f0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b1d20c6e70 0x55b1d2135900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b1d20bb650 0x55b1d2130940 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde4d9d8b80 1 --2- 172.21.15.98:0/3925685931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b1d20bbf70 0x55b1d2130e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3925685931 >> 172.21.15.98:0/3925685931 conn(0x55b1d1fe8990 msgr2=0x55b1d20c5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.677+0000 7fde1bfff700 0 client.16806 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3925685931 shutdown_connections 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4d9d8b80 1 -- 172.21.15.98:0/3925685931 wait complete. 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4d9d8b80 20 client.16806 trim_cache size 0 max 16384 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 51, 2024-04-25T17:49:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 40, 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "a13e6163": { 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 40, 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:49:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 89.1, 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RELEASED", 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:49:29.682 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:49:29.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 -- 172.21.15.98:0/2610022986 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fde240a3db0 msgr2=0x7fde240a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fde240a3db0 0x7fde240a6270 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fde2c005fd0 tx=0x7fde2c005e20 comp rx=0 tx=0).stop 2024-04-25T17:49:29.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 -- 172.21.15.98:0/2610022986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c102a90 msgr2=0x7fde3c198f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:29.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c102a90 0x7fde3c198f90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fde3800c9b0 tx=0x7fde3800cd70 comp rx=0 tx=0).stop 2024-04-25T17:49:29.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 -- 172.21.15.98:0/2610022986 shutdown_connections 2024-04-25T17:49:29.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fde240a3db0 0x7fde240a6270 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fde3c108a50 0x7fde3c19d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde3c102a90 0x7fde3c198f90 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 --2- 172.21.15.98:0/2610022986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde3c103450 0x7fde3c1994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:29.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 -- 172.21.15.98:0/2610022986 >> 172.21.15.98:0/2610022986 conn(0x7fde3c079ba0 msgr2=0x7fde3c101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:29.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.678+0000 7fde4314f700 1 -- 172.21.15.98:0/2610022986 shutdown_connections 2024-04-25T17:49:29.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:29.679+0000 7fde4314f700 1 -- 172.21.15.98:0/2610022986 wait complete. 2024-04-25T17:49:30.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:30 smithi203 ceph-mon[32834]: pgmap v428: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-04-25T17:49:30.469 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:30 smithi098 ceph-mon[29192]: pgmap v428: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-04-25T17:49:30.471 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a13e6163', seconds elapsed: 45.0 2024-04-25T17:49:30.472 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.1 2024-04-25T17:49:30.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:49:30.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:30 smithi173 ceph-mon[32856]: pgmap v428: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-04-25T17:49:30.839 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:49:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 780.0 s, 557.7 tps, lat 1335.817 ms stddev 6606.461 2024-04-25T17:49:32.384 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:32 smithi098 ceph-mon[29192]: pgmap v429: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-04-25T17:49:32.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:32 smithi173 ceph-mon[32856]: pgmap v429: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-04-25T17:49:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.704+0000 7fcdaef70700 1 -- 172.21.15.98:0/2323638078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda811c220 msgr2=0x7fcda811e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:32.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.704+0000 7fcdaef70700 1 --2- 172.21.15.98:0/2323638078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda811c220 0x7fcda811e610 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c009990 tx=0x7fcd9c009ca0 comp rx=0 tx=0).stop 2024-04-25T17:49:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.707+0000 7fcdaef70700 1 -- 172.21.15.98:0/2323638078 shutdown_connections 2024-04-25T17:49:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.707+0000 7fcdaef70700 1 --2- 172.21.15.98:0/2323638078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda811c220 0x7fcda811e610 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.707+0000 7fcdaef70700 1 --2- 172.21.15.98:0/2323638078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcda8111220 0x7fcda8109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.707+0000 7fcdaef70700 1 --2- 172.21.15.98:0/2323638078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcda8109f80 0x7fcda810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.707+0000 7fcdaef70700 1 -- 172.21.15.98:0/2323638078 >> 172.21.15.98:0/2323638078 conn(0x7fcda806dbb0 msgr2=0x7fcda806dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.709+0000 7fcdaef70700 1 -- 172.21.15.98:0/2323638078 shutdown_connections 2024-04-25T17:49:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.709+0000 7fcdaef70700 1 -- 172.21.15.98:0/2323638078 wait complete. 2024-04-25T17:49:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.709+0000 7fcdaef70700 1 Processor -- start 2024-04-25T17:49:32.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:32 smithi203 ceph-mon[32834]: pgmap v429: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2024-04-25T17:49:32.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.710+0000 7fcdaef70700 1 -- start start 2024-04-25T17:49:32.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.711+0000 7fcdaef70700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcda8109f80 0x7fcda811b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:32.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.711+0000 7fcdaef70700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda8111220 0x7fcda81158f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:32.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.711+0000 7fcdaef70700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcda811c220 0x7fcda8115e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:32.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.711+0000 7fcdaef70700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcda8112300 con 0x7fcda8111220 2024-04-25T17:49:32.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.711+0000 7fcdaef70700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcda8112180 con 0x7fcda811c220 2024-04-25T17:49:32.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.711+0000 7fcdaef70700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcda8112480 con 0x7fcda8109f80 2024-04-25T17:49:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.712+0000 7fcda7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda8111220 0x7fcda81158f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.712+0000 7fcda7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda8111220 0x7fcda81158f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43900/0 (socket says 172.21.15.98:43900) 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.712+0000 7fcda7fff700 1 -- 172.21.15.98:0/76846697 learned_addr learned my addr 172.21.15.98:0/76846697 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.712+0000 7fcdacd0c700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcda8109f80 0x7fcda811b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.713+0000 7fcda7fff700 1 -- 172.21.15.98:0/76846697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcda8109f80 msgr2=0x7fcda811b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.713+0000 7fcda7fff700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcda8109f80 0x7fcda811b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.713+0000 7fcda7fff700 1 -- 172.21.15.98:0/76846697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcda811c220 msgr2=0x7fcda8115e30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.713+0000 7fcda7fff700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcda811c220 0x7fcda8115e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.713+0000 7fcda7fff700 1 -- 172.21.15.98:0/76846697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda8116470 con 0x7fcda8111220 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.713+0000 7fcda7fff700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda8111220 0x7fcda81158f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fcda000b840 tx=0x7fcda000bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.713+0000 7fcda5ffb700 1 -- 172.21.15.98:0/76846697 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda000d890 con 0x7fcda8111220 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.714+0000 7fcdaef70700 1 -- 172.21.15.98:0/76846697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda8116700 con 0x7fcda8111220 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.714+0000 7fcda5ffb700 1 -- 172.21.15.98:0/76846697 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcda000e430 con 0x7fcda8111220 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.714+0000 7fcda5ffb700 1 -- 172.21.15.98:0/76846697 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda0017660 con 0x7fcda8111220 2024-04-25T17:49:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.714+0000 7fcdaef70700 1 -- 172.21.15.98:0/76846697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda8121630 con 0x7fcda8111220 2024-04-25T17:49:32.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.720+0000 7fcdaef70700 1 -- 172.21.15.98:0/76846697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcda806f140 con 0x7fcda8111220 2024-04-25T17:49:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.722+0000 7fcda5ffb700 1 -- 172.21.15.98:0/76846697 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcda000e5e0 con 0x7fcda8111220 2024-04-25T17:49:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.723+0000 7fcda5ffb700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd900c5fc0 0x7fcd900c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:32.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.723+0000 7fcda5ffb700 1 -- 172.21.15.98:0/76846697 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7fcda00d0270 con 0x7fcda8111220 2024-04-25T17:49:32.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.728+0000 7fcdacd0c700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd900c5fc0 0x7fcd900c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:32.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.729+0000 7fcdacd0c700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd900c5fc0 0x7fcd900c8480 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800a850 tx=0x7fcd98008040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:32.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:32.733+0000 7fcda5ffb700 1 -- 172.21.15.98:0/76846697 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcda0013030 con 0x7fcda8111220 2024-04-25T17:49:33.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.096+0000 7fcdaef70700 1 -- 172.21.15.98:0/76846697 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcda8121950 con 0x7fcda8111220 2024-04-25T17:49:33.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.097+0000 7fcda5ffb700 1 -- 172.21.15.98:0/76846697 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+11878 (secure 0 0 0) 0x7fcda009c8b0 con 0x7fcda8111220 2024-04-25T17:49:33.096 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:49:33.097 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":51,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:48:59.915105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26026":{"gid":26026,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/3925531495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":3925531495},{"type":"v1","addr":"172.21.15.203:6837","nonce":3925531495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 -- 172.21.15.98:0/76846697 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd900c5fc0 msgr2=0x7fcd900c8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd900c5fc0 0x7fcd900c8480 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800a850 tx=0x7fcd98008040 comp rx=0 tx=0).stop 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 -- 172.21.15.98:0/76846697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda8111220 msgr2=0x7fcda81158f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda8111220 0x7fcda81158f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fcda000b840 tx=0x7fcda000bc00 comp rx=0 tx=0).stop 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 -- 172.21.15.98:0/76846697 shutdown_connections 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcd900c5fc0 0x7fcd900c8480 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcda8109f80 0x7fcda811b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcda811c220 0x7fcda8115e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 --2- 172.21.15.98:0/76846697 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda8111220 0x7fcda81158f0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 -- 172.21.15.98:0/76846697 >> 172.21.15.98:0/76846697 conn(0x7fcda806dbb0 msgr2=0x7fcda811d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 -- 172.21.15.98:0/76846697 shutdown_connections 2024-04-25T17:49:33.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:33.100+0000 7fcdaef70700 1 -- 172.21.15.98:0/76846697 wait complete. 2024-04-25T17:49:33.101 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 51 2024-04-25T17:49:33.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:33 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/76846697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:49:33.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:33 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/76846697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:49:33.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:33 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/76846697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:49:34.107 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db / --timeout 60 --expiration 89.1 --await-for 42.0' 2024-04-25T17:49:34.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db / --timeout 60 --expiration 89.1 --await-for 42.0 2024-04-25T17:49:34.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:34 smithi098 ceph-mon[29192]: pgmap v430: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 674 KiB/s wr, 83 op/s 2024-04-25T17:49:34.465 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:49:34.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:34 smithi173 ceph-mon[32856]: pgmap v430: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 674 KiB/s wr, 83 op/s 2024-04-25T17:49:34.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:34 smithi203 ceph-mon[32834]: pgmap v430: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 674 KiB/s wr, 83 op/s 2024-04-25T17:49:36.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.294+0000 7f0ac16ed700 1 -- 172.21.15.98:0/3850072343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0abc107930 msgr2=0x7f0abc107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:36.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.294+0000 7f0ac16ed700 1 --2- 172.21.15.98:0/3850072343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0abc107930 0x7f0abc107d10 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0aac009990 tx=0x7f0aac009ca0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.294+0000 7f0ac16ed700 1 -- 172.21.15.98:0/3850072343 shutdown_connections 2024-04-25T17:49:36.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.294+0000 7f0ac16ed700 1 --2- 172.21.15.98:0/3850072343 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0abc102300 0x7f0abc10e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.294+0000 7f0ac16ed700 1 --2- 172.21.15.98:0/3850072343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0abc107930 0x7f0abc107d10 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.294+0000 7f0ac16ed700 1 --2- 172.21.15.98:0/3850072343 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0abc101940 0x7f0abc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.294+0000 7f0ac16ed700 1 -- 172.21.15.98:0/3850072343 >> 172.21.15.98:0/3850072343 conn(0x7f0abc0788e0 msgr2=0x7f0abc078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:36.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.294+0000 7f0ac16ed700 1 -- 172.21.15.98:0/3850072343 shutdown_connections 2024-04-25T17:49:36.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.294+0000 7f0ac16ed700 1 -- 172.21.15.98:0/3850072343 wait complete. 2024-04-25T17:49:36.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.294+0000 7f0ac16ed700 1 Processor -- start 2024-04-25T17:49:36.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.295+0000 7f0ac16ed700 1 -- start start 2024-04-25T17:49:36.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.295+0000 7f0ac16ed700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0abc101940 0x7f0abc197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.295+0000 7f0ac16ed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0abc102300 0x7f0abc198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.295+0000 7f0ac16ed700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0abc107930 0x7f0abc19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.295+0000 7f0abaffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0abc101940 0x7f0abc197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.296+0000 7f0abaffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0abc101940 0x7f0abc197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43952/0 (socket says 172.21.15.98:43952) 2024-04-25T17:49:36.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.296+0000 7f0abaffd700 1 -- 172.21.15.98:0/617910387 learned_addr learned my addr 172.21.15.98:0/617910387 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:36.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.296+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0abc107930 0x7f0abc19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.296+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0abc111b40 con 0x7f0abc102300 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.296+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0abc1119c0 con 0x7f0abc101940 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.296+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0abc111cc0 con 0x7f0abc107930 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.302+0000 7f0abb7fe700 1 -- 172.21.15.98:0/617910387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0abc101940 msgr2=0x7f0abc197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.302+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0abc101940 0x7f0abc197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.302+0000 7f0abb7fe700 1 -- 172.21.15.98:0/617910387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0abc102300 msgr2=0x7f0abc198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.302+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0abc102300 0x7f0abc198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.302+0000 7f0abb7fe700 1 -- 172.21.15.98:0/617910387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0abc19cf40 con 0x7f0abc107930 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.305+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0abc107930 0x7f0abc19c800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400efc0 tx=0x7f0ab400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.305+0000 7f0aa3fff700 1 -- 172.21.15.98:0/617910387 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab401a070 con 0x7f0abc107930 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.305+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0abc19d230 con 0x7f0abc107930 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.305+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0abc06a720 con 0x7f0abc107930 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.306+0000 7f0aa3fff700 1 -- 172.21.15.98:0/617910387 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ab4004bb0 con 0x7f0abc107930 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.306+0000 7f0aa3fff700 1 -- 172.21.15.98:0/617910387 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4015a30 con 0x7f0abc107930 2024-04-25T17:49:36.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.310+0000 7f0aa3fff700 1 -- 172.21.15.98:0/617910387 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0ab4015bd0 con 0x7f0abc107930 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.311+0000 7f0aa3fff700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0aa40a3ed0 0x7f0aa40a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.311+0000 7f0aa3fff700 1 -- 172.21.15.98:0/617910387 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7f0ab40d55f0 con 0x7f0abc107930 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.314+0000 7f0abaffd700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0aa40a3ed0 0x7f0aa40a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.316+0000 7f0acbf76b80 1 Processor -- start 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.319+0000 7f0acbf76b80 1 -- start start 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.319+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 0x5623e555aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.320+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e55659c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.320+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e55682f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.320+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5623e54ade20 con 0x5623e5565f00 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.320+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5623e5454ac0 con 0x5623e555a6e0 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.320+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5623e54a9de0 con 0x5623e555b000 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.320+0000 7f0abaffd700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0aa40a3ed0 0x7f0aa40a6390 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00b2e0 tx=0x7f0aac01a040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.321+0000 7f0abaffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 0x5623e555aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.321+0000 7f0abaffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 0x5623e555aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43956/0 (socket says 172.21.15.98:43956) 2024-04-25T17:49:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.321+0000 7f0abaffd700 1 -- 172.21.15.98:0/3238280153 learned_addr learned my addr 172.21.15.98:0/3238280153 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.322+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/3238280153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e55682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.323+0000 7f0aba7fc700 1 --2- 172.21.15.98:0/3238280153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e55659c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.324+0000 7f0abb7fe700 1 -- 172.21.15.98:0/3238280153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 msgr2=0x5623e55659c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.324+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/3238280153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e55659c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.324+0000 7f0abb7fe700 1 -- 172.21.15.98:0/3238280153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 msgr2=0x5623e555aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.324+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/3238280153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 0x5623e555aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.324+0000 7f0abb7fe700 1 -- 172.21.15.98:0/3238280153 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623e54188d0 con 0x5623e5565f00 2024-04-25T17:49:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.325+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/3238280153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e55682f0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0ab40d2d40 tx=0x7f0ab40a1490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeeb3d4ef061e673 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:36.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.327+0000 7f0aba7fc700 1 --2- 172.21.15.98:0/3238280153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e55659c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.327+0000 7f0aa17fa700 1 -- 172.21.15.98:0/3238280153 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab40aa200 con 0x5623e5565f00 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.327+0000 7f0aa17fa700 1 -- 172.21.15.98:0/3238280153 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ab4004d60 con 0x5623e5565f00 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.327+0000 7f0aa17fa700 1 -- 172.21.15.98:0/3238280153 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab40a9fd0 con 0x5623e5565f00 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.329+0000 7f0acbf76b80 1 -- 172.21.15.98:0/3238280153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 msgr2=0x5623e55682f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.329+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/3238280153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e55682f0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0ab40d2d40 tx=0x7f0ab40a1490 comp rx=0 tx=0).stop 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.331+0000 7f0acbf76b80 1 -- 172.21.15.98:0/3238280153 shutdown_connections 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.331+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/3238280153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e55682f0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.331+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/3238280153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 0x5623e555aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.331+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/3238280153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e55659c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.331+0000 7f0acbf76b80 1 -- 172.21.15.98:0/3238280153 >> 172.21.15.98:0/3238280153 conn(0x5623e5487a20 msgr2=0x5623e5556ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.333+0000 7f0acbf76b80 1 -- 172.21.15.98:0/3238280153 shutdown_connections 2024-04-25T17:49:36.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.336+0000 7f0acbf76b80 1 -- 172.21.15.98:0/3238280153 wait complete. 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.337+0000 7f0acbf76b80 1 -- start start 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.337+0000 7f0acbf76b80 1 Processor -- start 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.338+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 0x5623e55656e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.338+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e5562a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.338+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e5562f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.338+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5623e5568930 con 0x5623e5565f00 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.338+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5623e5467740 con 0x5623e555a6e0 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.338+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5623e5568ab0 con 0x5623e555b000 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.338+0000 7f0acbf76b80 10 client.0 _ll_register_callbacks cb 0x5623e5446ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.339+0000 7f0aba7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e5562a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.339+0000 7f0aba7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e5562a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:38016/0 (socket says 172.21.15.98:38016) 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.339+0000 7f0aba7fc700 1 -- 172.21.15.98:0/2739461882 learned_addr learned my addr 172.21.15.98:0/2739461882 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.340+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e5562f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.340+0000 7f0abaffd700 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 0x5623e55656e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.340+0000 7f0aba7fc700 1 -- 172.21.15.98:0/2739461882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 msgr2=0x5623e55656e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.340+0000 7f0aba7fc700 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 0x5623e55656e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.340+0000 7f0aba7fc700 1 -- 172.21.15.98:0/2739461882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 msgr2=0x5623e5562f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.340+0000 7f0aba7fc700 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e5562f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.340+0000 7f0aba7fc700 1 -- 172.21.15.98:0/2739461882 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623e53fba60 con 0x5623e555b000 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.340+0000 7f0abaffd700 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 0x5623e55656e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.341+0000 7f0aba7fc700 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e5562a50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000e370 tx=0x7f0ab000e680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1d62ca320c3a76f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.341+0000 7f0a9eff5700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.341+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2739461882 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0016070 con 0x5623e555b000 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.341+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2739461882 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab4004d60 con 0x5623e555b000 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.341+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2739461882 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7f0ab0004a30 con 0x5623e555b000 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.342+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2739461882 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0ab0021050 con 0x5623e555b000 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.342+0000 7f0acbf76b80 10 client.26263 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:36.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.342+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2739461882 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x5623e557bb20 con 0x5623e555b000 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.342+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2739461882 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ab4004d60 con 0x5623e555b000 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.342+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2739461882 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7f0ab000f040 con 0x5623e555b000 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.343+0000 7f0acbf76b80 10 client.26263 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.343+0000 7f0acbf76b80 10 client.26263 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.343+0000 7f0acbf76b80 20 client.26263 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.343+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5623e557e4f0 0x5623e5580910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.343+0000 7f0acbf76b80 4 client.26263 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.343+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2739461882 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623e54bb610 con 0x5623e557e4f0 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.343+0000 7f0abaffd700 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5623e557e4f0 0x5623e5580910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.345+0000 7f0abaffd700 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5623e557e4f0 0x5623e5580910 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=648f5429e9813fe3 server_cookie=15c37b14d27dab0c in_seq=0 out_seq=0 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.345+0000 7f0a9eff5700 10 client.26263 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2739461882 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623e54bb610 con 0x5623e557e4f0 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0a9eff5700 10 client.26263 handle_command_reply: tid=0 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 client.26263 shutdown 2024-04-25T17:49:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2739461882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 msgr2=0x5623e5562a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e5562a50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000e370 tx=0x7f0ab000e680 comp rx=0 tx=0).stop 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2739461882 shutdown_connections 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5623e557e4f0 0x5623e5580910 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e5562f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e555a6e0 0x5623e55656e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2739461882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555b000 0x5623e5562a50 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000e370 tx=0x7f0ab000e680 comp rx=0 tx=0).stop 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2739461882 >> 172.21.15.98:0/2739461882 conn(0x5623e5487a20 msgr2=0x5623e555d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0a9eff5700 0 client.26263 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2739461882 shutdown_connections 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.346+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2739461882 wait complete. 2024-04-25T17:49:36.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.347+0000 7f0acbf76b80 20 client.26263 trim_cache size 0 max 16384 2024-04-25T17:49:36.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0acbf76b80 1 Processor -- start 2024-04-25T17:49:36.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0acbf76b80 1 -- start start 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e555a6e0 0x5623e55861e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5565f00 0x5623e5586720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e5564d50 0x5623e555fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa800fe60 con 0x5623e555a6e0 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa800d6f0 con 0x5623e5565f00 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa800b560 con 0x5623e5564d50 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0aba7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5565f00 0x5623e5586720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0aba7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5565f00 0x5623e5586720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43970/0 (socket says 172.21.15.98:43970) 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0abaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e555a6e0 0x5623e55861e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0aba7fc700 1 -- 172.21.15.98:0/4080083517 learned_addr learned my addr 172.21.15.98:0/4080083517 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/4080083517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e5564d50 0x5623e555fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0aba7fc700 1 -- 172.21.15.98:0/4080083517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e5564d50 msgr2=0x5623e555fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0aba7fc700 1 --2- 172.21.15.98:0/4080083517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e5564d50 0x5623e555fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0aba7fc700 1 -- 172.21.15.98:0/4080083517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e555a6e0 msgr2=0x5623e55861e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:36.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0aba7fc700 1 --2- 172.21.15.98:0/4080083517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e555a6e0 0x5623e55861e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0aba7fc700 1 -- 172.21.15.98:0/4080083517 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623e5568690 con 0x5623e5565f00 2024-04-25T17:49:36.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.373+0000 7f0abaffd700 1 --2- 172.21.15.98:0/4080083517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e555a6e0 0x5623e55861e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:49:36.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.374+0000 7f0aba7fc700 1 --2- 172.21.15.98:0/4080083517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5565f00 0x5623e5586720 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0001eb0 tx=0x7f0ab0005ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7bb2e59c4d5eb9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:36.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.374+0000 7f0aa0ff9700 1 -- 172.21.15.98:0/4080083517 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0016050 con 0x5623e5565f00 2024-04-25T17:49:36.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.374+0000 7f0aa0ff9700 1 -- 172.21.15.98:0/4080083517 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ab00044f0 con 0x5623e5565f00 2024-04-25T17:49:36.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.374+0000 7f0aa0ff9700 1 -- 172.21.15.98:0/4080083517 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0004a90 con 0x5623e5565f00 2024-04-25T17:49:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.375+0000 7f0acbf76b80 1 -- 172.21.15.98:0/4080083517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5565f00 msgr2=0x5623e5586720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.375+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/4080083517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5565f00 0x5623e5586720 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0001eb0 tx=0x7f0ab0005ba0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.375+0000 7f0acbf76b80 1 -- 172.21.15.98:0/4080083517 shutdown_connections 2024-04-25T17:49:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.375+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/4080083517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e5564d50 0x5623e555fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.375+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/4080083517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e555a6e0 0x5623e55861e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.375+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/4080083517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5565f00 0x5623e5586720 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.375+0000 7f0acbf76b80 1 -- 172.21.15.98:0/4080083517 >> 172.21.15.98:0/4080083517 conn(0x5623e5487a20 msgr2=0x5623e5555cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.375+0000 7f0acbf76b80 1 -- 172.21.15.98:0/4080083517 shutdown_connections 2024-04-25T17:49:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.375+0000 7f0acbf76b80 1 -- 172.21.15.98:0/4080083517 wait complete. 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.375+0000 7f0acbf76b80 1 -- start start 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0acbf76b80 1 Processor -- start 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555a6e0 0x5623e5576990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5564d50 0x5623e5574d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0acbf76b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e55752c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa800bbb0 con 0x5623e5565f00 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa800d020 con 0x5623e5564d50 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0acbf76b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa800c280 con 0x5623e555a6e0 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0aba7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5564d50 0x5623e5574d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0acbf76b80 10 client.0 _ll_register_callbacks cb 0x5623e5446ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0abb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e55752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0aba7fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5564d50 0x5623e5574d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:43986/0 (socket says 172.21.15.98:43986) 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0aba7fc700 1 -- 172.21.15.98:0/2482903574 learned_addr learned my addr 172.21.15.98:0/2482903574 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:36.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0abaffd700 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555a6e0 0x5623e5576990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0abb7fe700 1 -- 172.21.15.98:0/2482903574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555a6e0 msgr2=0x5623e5576990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555a6e0 0x5623e5576990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0abb7fe700 1 -- 172.21.15.98:0/2482903574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5564d50 msgr2=0x5623e5574d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5564d50 0x5623e5574d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.376+0000 7f0abb7fe700 1 -- 172.21.15.98:0/2482903574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623e5575b00 con 0x5623e5565f00 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0abb7fe700 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e55752c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0ab409d9f0 tx=0x7f0ab40098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da324120c9de26d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0a9eff5700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2482903574 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab40a9d70 con 0x5623e5565f00 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2482903574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab00044f0 con 0x5623e5565f00 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2482903574 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7f0ab40a9f10 con 0x5623e5565f00 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2482903574 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0ab40a09e0 con 0x5623e5565f00 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0acbf76b80 10 client.16827 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2482903574 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x5623e55e5b00 con 0x5623e5565f00 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2482903574 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ab00044f0 con 0x5623e5565f00 2024-04-25T17:49:36.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2482903574 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7f0ab408aad0 con 0x5623e5565f00 2024-04-25T17:49:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0acbf76b80 10 client.16827 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0acbf76b80 10 client.16827 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:49:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0acbf76b80 20 client.16827 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.377+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5623e55e5de0 0x5623e55e81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.378+0000 7f0acbf76b80 4 client.16827 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.1, "await_for": 42.0}] 2024-04-25T17:49:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.378+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2482903574 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 89.1, "await_for": 42.0}) -- 0x5623e54b56b0 con 0x5623e55e5de0 2024-04-25T17:49:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.378+0000 7f0abaffd700 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5623e55e5de0 0x5623e55e81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.378+0000 7f0a9eff5700 10 client.16827 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:36.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.378+0000 7f0abaffd700 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5623e55e5de0 0x5623e55e81d0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d40a7546a50c63de server_cookie=26361e663d33a5fa in_seq=0 out_seq=0 2024-04-25T17:49:36.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:36 smithi098 ceph-mon[29192]: pgmap v431: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 671 KiB/s wr, 79 op/s 2024-04-25T17:49:36.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:36 smithi173 ceph-mon[32856]: pgmap v431: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 671 KiB/s wr, 79 op/s 2024-04-25T17:49:36.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0a9eff5700 1 -- 172.21.15.98:0/2482903574 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5623e54b56b0 con 0x5623e55e5de0 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0a9eff5700 10 client.16827 handle_command_reply: tid=0 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 client.16827 shutdown 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2482903574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 msgr2=0x5623e55752c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e55752c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0ab409d9f0 tx=0x7f0ab40098a0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2482903574 shutdown_connections 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5623e55e5de0 0x5623e55e81d0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5623e555a6e0 0x5623e5576990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5623e5565f00 0x5623e55752c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 --2- 172.21.15.98:0/2482903574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5623e5564d50 0x5623e5574d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2482903574 >> 172.21.15.98:0/2482903574 conn(0x5623e5487a20 msgr2=0x5623e5561fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:36.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0a9eff5700 0 client.16827 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:36.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2482903574 shutdown_connections 2024-04-25T17:49:36.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 1 -- 172.21.15.98:0/2482903574 wait complete. 2024-04-25T17:49:36.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.561+0000 7f0acbf76b80 20 client.16827 trim_cache size 0 max 16384 2024-04-25T17:49:36.564 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:49:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 51, 2024-04-25T17:49:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:49:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 42, 2024-04-25T17:49:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:49:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "d4deafbb": { 2024-04-25T17:49:36.564 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 42, 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.0, 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 89.1, 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:49:36.565 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:49:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:49:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:49:36.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:49:36.566 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:49:36.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0aa40a3ed0 msgr2=0x7f0aa40a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:36.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0aa40a3ed0 0x7f0aa40a6390 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00b2e0 tx=0x7f0aac01a040 comp rx=0 tx=0).stop 2024-04-25T17:49:36.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0abc107930 msgr2=0x7f0abc19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:36.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0abc107930 0x7f0abc19c800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400efc0 tx=0x7f0ab400c3a0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 shutdown_connections 2024-04-25T17:49:36.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0aa40a3ed0 0x7f0aa40a6390 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0abc107930 0x7f0abc19c800 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0abc101940 0x7f0abc197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 --2- 172.21.15.98:0/617910387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0abc102300 0x7f0abc198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:36.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 >> 172.21.15.98:0/617910387 conn(0x7f0abc0788e0 msgr2=0x7f0abc0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:36.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 shutdown_connections 2024-04-25T17:49:36.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:49:36.562+0000 7f0ac16ed700 1 -- 172.21.15.98:0/617910387 wait complete. 2024-04-25T17:49:36.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 785.0 s, 2109.4 tps, lat 46.048 ms stddev 51.464 2024-04-25T17:49:36.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:36 smithi203 ceph-mon[32834]: pgmap v431: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 671 KiB/s wr, 79 op/s 2024-04-25T17:49:37.591 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'd4deafbb', quiesce took 7.1 seconds. Will release after: 29.1 2024-04-25T17:49:38.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:38 smithi173 ceph-mon[32856]: pgmap v432: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 3.2 MiB/s wr, 351 op/s 2024-04-25T17:49:38.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:38 smithi098 ceph-mon[29192]: pgmap v432: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 3.2 MiB/s wr, 351 op/s 2024-04-25T17:49:38.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:38 smithi203 ceph-mon[32834]: pgmap v432: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 3.2 MiB/s wr, 351 op/s 2024-04-25T17:49:40.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:40 smithi173 ceph-mon[32856]: pgmap v433: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 3.2 MiB/s wr, 349 op/s 2024-04-25T17:49:40.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:40 smithi098 ceph-mon[29192]: pgmap v433: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 3.2 MiB/s wr, 349 op/s 2024-04-25T17:49:40.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:40 smithi203 ceph-mon[32834]: pgmap v433: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 3.2 MiB/s wr, 349 op/s 2024-04-25T17:49:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 790.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:49:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.313+0000 7f7835fc6640 1 -- 172.21.15.98:0/4233079998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7830101aa0 msgr2=0x7f7830105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.313+0000 7f7835fc6640 1 --2- 172.21.15.98:0/4233079998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7830101aa0 0x7f7830105a90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f7818009900 tx=0x7f781802a410 comp rx=0 tx=0).stop 2024-04-25T17:49:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.313+0000 7f7835fc6640 1 -- 172.21.15.98:0/4233079998 shutdown_connections 2024-04-25T17:49:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.313+0000 7f7835fc6640 1 --2- 172.21.15.98:0/4233079998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7830106170 0x7f783010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.313+0000 7f7835fc6640 1 --2- 172.21.15.98:0/4233079998 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7830101aa0 0x7f7830105a90 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.313+0000 7f7835fc6640 1 --2- 172.21.15.98:0/4233079998 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78301010f0 0x7f78301014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.313+0000 7f7835fc6640 1 -- 172.21.15.98:0/4233079998 >> 172.21.15.98:0/4233079998 conn(0x7f7830077fd0 msgr2=0x7f78300783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.313+0000 7f7835fc6640 1 -- 172.21.15.98:0/4233079998 shutdown_connections 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.313+0000 7f7835fc6640 1 -- 172.21.15.98:0/4233079998 wait complete. 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f7835fc6640 1 Processor -- start 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f7835fc6640 1 -- start start 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f7835fc6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78301010f0 0x7f78301986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f7835fc6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7830106170 0x7f7830198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f7835fc6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783019cf60 0x7f783019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f7835fc6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7830111090 con 0x7f783019cf60 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f7835fc6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7830110f10 con 0x7f7830106170 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f7835fc6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7830111210 con 0x7f78301010f0 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783019cf60 0x7f783019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783019cf60 0x7f783019d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45420/0 (socket says 172.21.15.98:45420) 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782ffff640 1 -- 172.21.15.98:0/3171625815 learned_addr learned my addr 172.21.15.98:0/3171625815 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782f7fe640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78301010f0 0x7f78301986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f7827fff640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7830106170 0x7f7830198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782ffff640 1 -- 172.21.15.98:0/3171625815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78301010f0 msgr2=0x7f78301986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782ffff640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78301010f0 0x7f78301986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782ffff640 1 -- 172.21.15.98:0/3171625815 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7830106170 msgr2=0x7f7830198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782ffff640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7830106170 0x7f7830198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782ffff640 1 -- 172.21.15.98:0/3171625815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783019da50 con 0x7f783019cf60 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782f7fe640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78301010f0 0x7f78301986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.314+0000 7f782ffff640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783019cf60 0x7f783019d410 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f782800eaa0 tx=0x7f782800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.315+0000 7f782d7fa640 1 -- 172.21.15.98:0/3171625815 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782800cdb0 con 0x7f783019cf60 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.315+0000 7f782d7fa640 1 -- 172.21.15.98:0/3171625815 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7828005020 con 0x7f783019cf60 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.315+0000 7f7835fc6640 1 -- 172.21.15.98:0/3171625815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783019dd40 con 0x7f783019cf60 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.315+0000 7f782d7fa640 1 -- 172.21.15.98:0/3171625815 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78280116d0 con 0x7f783019cf60 2024-04-25T17:49:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.315+0000 7f7835fc6640 1 -- 172.21.15.98:0/3171625815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78301a5530 con 0x7f783019cf60 2024-04-25T17:49:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.317+0000 7f782d7fa640 1 -- 172.21.15.98:0/3171625815 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f78280051d0 con 0x7f783019cf60 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.317+0000 7f7838dba740 1 Processor -- start 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.317+0000 7f7838dba740 1 -- start start 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7838dba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5627f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7838dba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1f40 0x55e5627fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7838dba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627fc800 0x55e5627febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7838dba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e562756970 con 0x55e5627f1620 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7838dba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e5626f34a0 con 0x55e5627f1f40 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7838dba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e562752930 con 0x55e5627fc800 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f782d7fa640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f781c0a3e40 0x7f781c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7827fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5627f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7827fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5627f1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45422/0 (socket says 172.21.15.98:45422) 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7827fff640 1 -- 172.21.15.98:0/4282666547 learned_addr learned my addr 172.21.15.98:0/4282666547 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f782f7fe640 1 --2- 172.21.15.98:0/4282666547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1f40 0x55e5627fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f782ffff640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f781c0a3e40 0x7f781c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7827fff640 1 --2- 172.21.15.98:0/4282666547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627fc800 0x55e5627febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f782d7fa640 1 -- 172.21.15.98:0/3171625815 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7f78280d0a10 con 0x7f783019cf60 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7827fff640 1 -- 172.21.15.98:0/4282666547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627fc800 msgr2=0x55e5627febf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7827fff640 1 --2- 172.21.15.98:0/4282666547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627fc800 0x55e5627febf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7827fff640 1 -- 172.21.15.98:0/4282666547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1f40 msgr2=0x55e5627fc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7827fff640 1 --2- 172.21.15.98:0/4282666547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1f40 0x55e5627fc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f7827fff640 1 -- 172.21.15.98:0/4282666547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e562469210 con 0x55e5627f1620 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.318+0000 7f782ffff640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f781c0a3e40 0x7f781c0a6300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f782809c8e0 tx=0x7f782808e040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7827fff640 1 --2- 172.21.15.98:0/4282666547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5627f1a00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x55e5627eecc0 tx=0x7f7818005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6899ecc200a7699e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7825ffb640 1 -- 172.21.15.98:0/4282666547 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818035050 con 0x55e5627f1620 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7825ffb640 1 -- 172.21.15.98:0/4282666547 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78180044d0 con 0x55e5627f1620 2024-04-25T17:49:42.327 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7825ffb640 1 -- 172.21.15.98:0/4282666547 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78180046c0 con 0x55e5627f1620 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7838dba740 1 -- 172.21.15.98:0/4282666547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 msgr2=0x55e5627f1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7838dba740 1 --2- 172.21.15.98:0/4282666547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5627f1a00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x55e5627eecc0 tx=0x7f7818005a20 comp rx=0 tx=0).stop 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7838dba740 1 -- 172.21.15.98:0/4282666547 shutdown_connections 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7838dba740 1 --2- 172.21.15.98:0/4282666547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627fc800 0x55e5627febf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7838dba740 1 --2- 172.21.15.98:0/4282666547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1f40 0x55e5627fc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7838dba740 1 --2- 172.21.15.98:0/4282666547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5627f1a00 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7838dba740 1 -- 172.21.15.98:0/4282666547 >> 172.21.15.98:0/4282666547 conn(0x55e5627edb30 msgr2=0x55e5627edf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7838dba740 1 -- 172.21.15.98:0/4282666547 shutdown_connections 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.319+0000 7f7838dba740 1 -- 172.21.15.98:0/4282666547 wait complete. 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7838dba740 1 -- start start 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7838dba740 1 Processor -- start 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7838dba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1620 0x55e5627fc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7838dba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627f1f40 0x55e5627f9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7838dba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627fc800 0x55e5627f9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7838dba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e5627ff3d0 con 0x55e5627fc800 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7838dba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e5626f10b0 con 0x55e5627f1620 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7838dba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e5627ff550 con 0x55e5627f1f40 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7838dba740 10 client.0 _ll_register_callbacks cb 0x55e5626ded30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7827fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627fc800 0x55e5627f9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7827fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627fc800 0x55e5627f9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45424/0 (socket says 172.21.15.98:45424) 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.320+0000 7f7827fff640 1 -- 172.21.15.98:0/3174900331 learned_addr learned my addr 172.21.15.98:0/3174900331 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f782f7fe640 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627f1f40 0x55e5627f9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f7827fff640 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1620 0x55e5627fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f7827fff640 1 -- 172.21.15.98:0/3174900331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627f1f40 msgr2=0x55e5627f9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f7827fff640 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627f1f40 0x55e5627f9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f7827fff640 1 -- 172.21.15.98:0/3174900331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1620 msgr2=0x55e5627fc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f7827fff640 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1620 0x55e5627fc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f7827fff640 1 -- 172.21.15.98:0/3174900331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e562535620 con 0x55e5627fc800 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f7827fff640 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627fc800 0x55e5627f9a00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7818009900 tx=0x7f7818005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad62944a9063bc13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f780effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f780effd640 1 -- 172.21.15.98:0/3174900331 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818053060 con 0x55e5627fc800 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f780effd640 1 -- 172.21.15.98:0/3174900331 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7f7818005040 con 0x55e5627fc800 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f7838dba740 1 -- 172.21.15.98:0/3174900331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78180044d0 con 0x55e5627fc800 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f780effd640 1 -- 172.21.15.98:0/3174900331 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f7818035070 con 0x55e5627fc800 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f7838dba740 10 client.16839 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f7838dba740 1 -- 172.21.15.98:0/3174900331 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55e56280b320 con 0x55e5627fc800 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.321+0000 7f780effd640 1 -- 172.21.15.98:0/3174900331 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78180044d0 con 0x55e5627fc800 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.322+0000 7f780effd640 1 -- 172.21.15.98:0/3174900331 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7f78180035f0 con 0x55e5627fc800 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.322+0000 7f7838dba740 10 client.16839 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.322+0000 7f7838dba740 10 client.16839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.322+0000 7f7838dba740 20 client.16839 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.322+0000 7f7838dba740 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e56280c670 0x55e56280ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.322+0000 7f7838dba740 4 client.16839 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.322+0000 7f7838dba740 1 -- 172.21.15.98:0/3174900331 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e562764390 con 0x55e56280c670 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.322+0000 7f782f7fe640 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e56280c670 0x55e56280ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.322+0000 7f782f7fe640 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e56280c670 0x55e56280ea60 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=960ef7cc09236a6f server_cookie=b8cd651006193160 in_seq=0 out_seq=0 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.322+0000 7f780effd640 10 client.16839 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f780effd640 1 -- 172.21.15.98:0/3174900331 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e562764390 con 0x55e56280c670 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f780effd640 10 client.16839 handle_command_reply: tid=0 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 client.16839 shutdown 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 -- 172.21.15.98:0/3174900331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627fc800 msgr2=0x55e5627f9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627fc800 0x55e5627f9a00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7818009900 tx=0x7f7818005bd0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 -- 172.21.15.98:0/3174900331 shutdown_connections 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e56280c670 0x55e56280ea60 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627fc800 0x55e5627f9a00 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f780effd640 0 client.16839 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627f1f40 0x55e5627f9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 --2- 172.21.15.98:0/3174900331 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1620 0x55e5627fc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 -- 172.21.15.98:0/3174900331 >> 172.21.15.98:0/3174900331 conn(0x55e5627edb30 msgr2=0x55e5627f38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 -- 172.21.15.98:0/3174900331 shutdown_connections 2024-04-25T17:49:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 1 -- 172.21.15.98:0/3174900331 wait complete. 2024-04-25T17:49:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.323+0000 7f7838dba740 20 client.16839 trim_cache size 0 max 16384 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f7838dba740 1 Processor -- start 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f7838dba740 1 -- start start 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f7838dba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1620 0x55e562808290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f7838dba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1f40 0x55e5627f8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f7838dba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627fc800 0x55e5627fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f7838dba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78301ae0d0 con 0x55e5627f1f40 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f7838dba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f78301ab9a0 con 0x55e5627f1620 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f7838dba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f78301a9810 con 0x55e5627fc800 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f782f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1f40 0x55e5627f8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f782f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1f40 0x55e5627f8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45428/0 (socket says 172.21.15.98:45428) 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.338+0000 7f782f7fe640 1 -- 172.21.15.98:0/2166254181 learned_addr learned my addr 172.21.15.98:0/2166254181 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f7827fff640 1 --2- 172.21.15.98:0/2166254181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1620 0x55e562808290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f7827fff640 1 --2- 172.21.15.98:0/2166254181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627fc800 0x55e5627fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f782f7fe640 1 -- 172.21.15.98:0/2166254181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627fc800 msgr2=0x55e5627fbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f782f7fe640 1 --2- 172.21.15.98:0/2166254181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627fc800 0x55e5627fbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f782f7fe640 1 -- 172.21.15.98:0/2166254181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1620 msgr2=0x55e562808290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f782f7fe640 1 --2- 172.21.15.98:0/2166254181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1620 0x55e562808290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f782f7fe640 1 -- 172.21.15.98:0/2166254181 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5627f1d30 con 0x55e5627f1f40 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f782f7fe640 1 --2- 172.21.15.98:0/2166254181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1f40 0x55e5627f8600 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f7820005fd0 tx=0x7f7820011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c709c8058a35d78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f78257fa640 1 -- 172.21.15.98:0/2166254181 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820011e60 con 0x55e5627f1f40 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f78257fa640 1 -- 172.21.15.98:0/2166254181 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7820022070 con 0x55e5627f1f40 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.339+0000 7f78257fa640 1 -- 172.21.15.98:0/2166254181 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820010c00 con 0x55e5627f1f40 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 -- 172.21.15.98:0/2166254181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1f40 msgr2=0x55e5627f8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 --2- 172.21.15.98:0/2166254181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1f40 0x55e5627f8600 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f7820005fd0 tx=0x7f7820011b30 comp rx=0 tx=0).stop 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 -- 172.21.15.98:0/2166254181 shutdown_connections 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 --2- 172.21.15.98:0/2166254181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627fc800 0x55e5627fbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.344 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 --2- 172.21.15.98:0/2166254181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1f40 0x55e5627f8600 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 --2- 172.21.15.98:0/2166254181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627f1620 0x55e562808290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 -- 172.21.15.98:0/2166254181 >> 172.21.15.98:0/2166254181 conn(0x55e5627edb30 msgr2=0x55e5627f4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 -- 172.21.15.98:0/2166254181 shutdown_connections 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 -- 172.21.15.98:0/2166254181 wait complete. 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 -- start start 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 Processor -- start 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5628047a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627f1f40 0x55e562804ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627fc800 0x55e5627f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78301a9e60 con 0x55e5627f1620 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f78301ab2d0 con 0x55e5627fc800 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f78301aa530 con 0x55e5627f1f40 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.340+0000 7f7838dba740 10 client.0 _ll_register_callbacks cb 0x7f7818032c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:49:42.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7827fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5628047a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7827fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5628047a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45444/0 (socket says 172.21.15.98:45444) 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7827fff640 1 -- 172.21.15.98:0/3940180686 learned_addr learned my addr 172.21.15.98:0/3940180686 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f782f7fe640 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627f1f40 0x55e562804ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7827fff640 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627fc800 0x55e5627f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7827fff640 1 -- 172.21.15.98:0/3940180686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627f1f40 msgr2=0x55e562804ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7827fff640 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627f1f40 0x55e562804ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7827fff640 1 -- 172.21.15.98:0/3940180686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627fc800 msgr2=0x55e5627f58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7827fff640 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627fc800 0x55e5627f58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7827fff640 1 -- 172.21.15.98:0/3940180686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5627f6060 con 0x55e5627f1620 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7827fff640 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5628047a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7818003a90 tx=0x7f781802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42540304a9d5776f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f780effd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:49:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f780effd640 1 -- 172.21.15.98:0/3940180686 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818049040 con 0x55e5627f1620 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f780effd640 1 -- 172.21.15.98:0/3940180686 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8513+0+0 (secure 0 0 0) 0x7f78180317e0 con 0x55e5627f1620 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7838dba740 1 -- 172.21.15.98:0/3940180686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7820022070 con 0x55e5627f1620 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f780effd640 1 -- 172.21.15.98:0/3940180686 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f78180319e0 con 0x55e5627f1620 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7838dba740 10 client.16845 fetch_fsmap learned FSMap version 51 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f7838dba740 1 -- 172.21.15.98:0/3940180686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55e5628723e0 con 0x55e5627f1620 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.341+0000 7f780effd640 1 -- 172.21.15.98:0/3940180686 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7820022070 con 0x55e5627f1620 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.342+0000 7f780effd640 1 -- 172.21.15.98:0/3940180686 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7f7818061a40 con 0x55e5627f1620 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.342+0000 7f7838dba740 10 client.16845 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.342+0000 7f7838dba740 10 client.16845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.342+0000 7f7838dba740 20 client.16845 populate_metadata read hostname 'smithi098' 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.342+0000 7f7838dba740 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e56280c670 0x55e562874b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.342+0000 7f7838dba740 4 client.16845 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:49:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.342+0000 7f7838dba740 1 -- 172.21.15.98:0/3940180686 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e56275e200 con 0x55e56280c670 2024-04-25T17:49:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.342+0000 7f782f7fe640 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e56280c670 0x55e562874b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:49:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.342+0000 7f780effd640 10 client.16845 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:49:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:49:42.342+0000 7f782f7fe640 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e56280c670 0x55e562874b10 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15ef5dd454353514 server_cookie=72f358f440de9255 in_seq=0 out_seq=0 2024-04-25T17:49:42.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:42 smithi173 ceph-mon[32856]: pgmap v434: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 3.2 MiB/s wr, 349 op/s 2024-04-25T17:49:42.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:42 smithi098 ceph-mon[29192]: pgmap v434: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 3.2 MiB/s wr, 349 op/s 2024-04-25T17:49:42.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:42 smithi203 ceph-mon[32834]: pgmap v434: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 3.2 MiB/s wr, 349 op/s 2024-04-25T17:49:44.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:44 smithi173 ceph-mon[32856]: pgmap v435: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 3.4 MiB/s wr, 366 op/s 2024-04-25T17:49:44.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:44 smithi098 ceph-mon[29192]: pgmap v435: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 3.4 MiB/s wr, 366 op/s 2024-04-25T17:49:44.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:44 smithi203 ceph-mon[32834]: pgmap v435: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 3.4 MiB/s wr, 366 op/s 2024-04-25T17:49:46.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:46 smithi173 ceph-mon[32856]: pgmap v436: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 2.7 MiB/s wr, 289 op/s 2024-04-25T17:49:46.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 795.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:49:46.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:46 smithi098 ceph-mon[29192]: pgmap v436: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 2.7 MiB/s wr, 289 op/s 2024-04-25T17:49:46.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:46 smithi203 ceph-mon[32834]: pgmap v436: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 2.7 MiB/s wr, 289 op/s 2024-04-25T17:49:48.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:48 smithi173 ceph-mon[32856]: pgmap v437: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-04-25T17:49:48.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:48 smithi098 ceph-mon[29192]: pgmap v437: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-04-25T17:49:48.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:48 smithi203 ceph-mon[32834]: pgmap v437: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-04-25T17:49:50.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:50 smithi098 ceph-mon[29192]: pgmap v438: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 154 KiB/s wr, 22 op/s 2024-04-25T17:49:50.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:50 smithi173 ceph-mon[32856]: pgmap v438: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 154 KiB/s wr, 22 op/s 2024-04-25T17:49:50.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:50 smithi203 ceph-mon[32834]: pgmap v438: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 154 KiB/s wr, 22 op/s 2024-04-25T17:49:51.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 800.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:49:52.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:52 smithi173 ceph-mon[32856]: pgmap v439: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 154 KiB/s wr, 22 op/s 2024-04-25T17:49:52.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:52 smithi098 ceph-mon[29192]: pgmap v439: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 154 KiB/s wr, 22 op/s 2024-04-25T17:49:52.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:52 smithi203 ceph-mon[32834]: pgmap v439: 97 pgs: 97 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 154 KiB/s wr, 22 op/s 2024-04-25T17:49:54.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:54 smithi173 ceph-mon[32856]: pgmap v440: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.8 MiB/s wr, 627 op/s 2024-04-25T17:49:54.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:54 smithi098 ceph-mon[29192]: pgmap v440: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.8 MiB/s wr, 627 op/s 2024-04-25T17:49:54.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:54 smithi203 ceph-mon[32834]: pgmap v440: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.8 MiB/s wr, 627 op/s 2024-04-25T17:49:56.514 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:56 smithi203 ceph-mon[32834]: pgmap v441: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 609 op/s 2024-04-25T17:49:56.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:56 smithi173 ceph-mon[32856]: pgmap v441: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 609 op/s 2024-04-25T17:49:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 805.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:49:56.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:56 smithi098 ceph-mon[29192]: pgmap v441: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 609 op/s 2024-04-25T17:49:58.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:49:58 smithi173 ceph-mon[32856]: pgmap v442: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.6 MiB/s wr, 613 op/s 2024-04-25T17:49:58.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:49:58 smithi098 ceph-mon[29192]: pgmap v442: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.6 MiB/s wr, 613 op/s 2024-04-25T17:49:58.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:49:58 smithi203 ceph-mon[32834]: pgmap v442: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.6 MiB/s wr, 613 op/s 2024-04-25T17:50:00.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:00 smithi173 ceph-mon[32856]: pgmap v443: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 607 op/s 2024-04-25T17:50:00.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:00 smithi173 ceph-mon[32856]: overall HEALTH_OK 2024-04-25T17:50:00.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:00 smithi098 ceph-mon[29192]: pgmap v443: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 607 op/s 2024-04-25T17:50:00.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:00 smithi098 ceph-mon[29192]: overall HEALTH_OK 2024-04-25T17:50:00.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:00 smithi203 ceph-mon[32834]: pgmap v443: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 607 op/s 2024-04-25T17:50:00.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:00 smithi203 ceph-mon[32834]: overall HEALTH_OK 2024-04-25T17:50:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 810.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:02.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:02 smithi173 ceph-mon[32856]: pgmap v444: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 607 op/s 2024-04-25T17:50:02.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:02 smithi098 ceph-mon[29192]: pgmap v444: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 607 op/s 2024-04-25T17:50:02.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:02 smithi203 ceph-mon[32834]: pgmap v444: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 MiB/s wr, 607 op/s 2024-04-25T17:50:04.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:04 smithi173 ceph-mon[32856]: pgmap v445: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.6 MiB/s wr, 611 op/s 2024-04-25T17:50:04.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:04 smithi098 ceph-mon[29192]: pgmap v445: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.6 MiB/s wr, 611 op/s 2024-04-25T17:50:04.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:04 smithi203 ceph-mon[32834]: pgmap v445: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.6 MiB/s wr, 611 op/s 2024-04-25T17:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 815.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:06.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:50:06.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:06 smithi098 ceph-mon[29192]: pgmap v446: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.2 KiB/s wr, 7 op/s 2024-04-25T17:50:06.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:06 smithi098 ceph-mon[29192]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T17:50:06.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:06 smithi203 ceph-mon[32834]: pgmap v446: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.2 KiB/s wr, 7 op/s 2024-04-25T17:50:06.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:06 smithi203 ceph-mon[32834]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T17:50:06.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:06 smithi173 ceph-mon[32856]: pgmap v446: 97 pgs: 97 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.2 KiB/s wr, 7 op/s 2024-04-25T17:50:06.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:06 smithi173 ceph-mon[32856]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T17:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:50:08.417 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:08 smithi098 ceph-mon[29192]: pgmap v448: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 30 op/s 2024-04-25T17:50:08.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:08 smithi203 ceph-mon[32834]: pgmap v448: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 30 op/s 2024-04-25T17:50:08.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:08 smithi173 ceph-mon[32856]: pgmap v448: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 30 op/s 2024-04-25T17:50:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.971+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/1620317796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b38101980 msgr2=0x7f1b38101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:50:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.971+0000 7f1b3ef6e700 1 --2- 172.21.15.98:0/1620317796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b38101980 0x7f1b38101de0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1b24009990 tx=0x7f1b24009ca0 comp rx=0 tx=0).stop 2024-04-25T17:50:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.971+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/1620317796 shutdown_connections 2024-04-25T17:50:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.971+0000 7f1b3ef6e700 1 --2- 172.21.15.98:0/1620317796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b38102320 0x7f1b3810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.971+0000 7f1b3ef6e700 1 --2- 172.21.15.98:0/1620317796 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b38107930 0x7f1b38107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:08.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.971+0000 7f1b3ef6e700 1 --2- 172.21.15.98:0/1620317796 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b38101980 0x7f1b38101de0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.972+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/1620317796 >> 172.21.15.98:0/1620317796 conn(0x7f1b38078870 msgr2=0x7f1b38078c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.972+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/1620317796 shutdown_connections 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.972+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/1620317796 wait complete. 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.972+0000 7f1b3ef6e700 1 Processor -- start 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.972+0000 7f1b3ef6e700 1 -- start start 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b3ef6e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b38101980 0x7f1b3819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b3ef6e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b38102320 0x7f1b3819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b3ef6e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b38107930 0x7f1b381a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b3ef6e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38111bb0 con 0x7f1b38102320 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b3ef6e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38111a30 con 0x7f1b38107930 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b3ef6e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38111d30 con 0x7f1b38101980 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b37fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b38102320 0x7f1b3819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:08.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b37fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b38102320 0x7f1b3819a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50584/0 (socket says 172.21.15.98:50584) 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b37fff700 1 -- 172.21.15.98:0/3707526603 learned_addr learned my addr 172.21.15.98:0/3707526603 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b3cd0a700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b38101980 0x7f1b3819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b3d50b700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b38107930 0x7f1b381a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b37fff700 1 -- 172.21.15.98:0/3707526603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b38101980 msgr2=0x7f1b3819a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b37fff700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b38101980 0x7f1b3819a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b37fff700 1 -- 172.21.15.98:0/3707526603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b38107930 msgr2=0x7f1b381a5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b37fff700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b38107930 0x7f1b381a5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b37fff700 1 -- 172.21.15.98:0/3707526603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b381a5670 con 0x7f1b38102320 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b37fff700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b38102320 0x7f1b3819a5f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f1b24000c00 tx=0x7f1b240047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b35ffb700 1 -- 172.21.15.98:0/3707526603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b24027070 con 0x7f1b38102320 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b35ffb700 1 -- 172.21.15.98:0/3707526603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b240040e0 con 0x7f1b38102320 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b35ffb700 1 -- 172.21.15.98:0/3707526603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2400f460 con 0x7f1b38102320 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.973+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/3707526603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b381a5840 con 0x7f1b38102320 2024-04-25T17:50:08.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.974+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/3707526603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b381a5c90 con 0x7f1b38102320 2024-04-25T17:50:08.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.975+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/3707526603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b38103070 con 0x7f1b38102320 2024-04-25T17:50:08.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.976+0000 7f1b35ffb700 1 -- 172.21.15.98:0/3707526603 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1b2402a070 con 0x7f1b38102320 2024-04-25T17:50:08.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.976+0000 7f1b35ffb700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1b280a3db0 0x7f1b280a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:08.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.976+0000 7f1b35ffb700 1 -- 172.21.15.98:0/3707526603 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8557+0+0 (secure 0 0 0) 0x7f1b240d17f0 con 0x7f1b38102320 2024-04-25T17:50:08.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.976+0000 7f1b3cd0a700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1b280a3db0 0x7f1b280a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:08.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.977+0000 7f1b3cd0a700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1b280a3db0 0x7f1b280a6270 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009a10 tx=0x7f1b2c009190 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:50:08.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:08.981+0000 7f1b35ffb700 1 -- 172.21.15.98:0/3707526603 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b2405eff0 con 0x7f1b38102320 2024-04-25T17:50:09.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.224+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/3707526603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b3802df80 con 0x7f1b38102320 2024-04-25T17:50:09.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.225+0000 7f1b35ffb700 1 -- 172.21.15.98:0/3707526603 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+11878 (secure 0 0 0) 0x7f1b24013070 con 0x7f1b38102320 2024-04-25T17:50:09.226 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:50:09.226 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":51,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:48:59.915105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26026":{"gid":26026,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/3925531495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":3925531495},{"type":"v1","addr":"172.21.15.203:6837","nonce":3925531495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:50:09.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/3707526603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1b280a3db0 msgr2=0x7f1b280a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:50:09.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1b280a3db0 0x7f1b280a6270 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009a10 tx=0x7f1b2c009190 comp rx=0 tx=0).stop 2024-04-25T17:50:09.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/3707526603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b38102320 msgr2=0x7f1b3819a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:50:09.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b38102320 0x7f1b3819a5f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f1b24000c00 tx=0x7f1b240047b0 comp rx=0 tx=0).stop 2024-04-25T17:50:09.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/3707526603 shutdown_connections 2024-04-25T17:50:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f1b280a3db0 0x7f1b280a6270 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b38101980 0x7f1b3819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b38107930 0x7f1b381a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 --2- 172.21.15.98:0/3707526603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1b38102320 0x7f1b3819a5f0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/3707526603 >> 172.21.15.98:0/3707526603 conn(0x7f1b38078870 msgr2=0x7f1b380ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:50:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/3707526603 shutdown_connections 2024-04-25T17:50:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:09.227+0000 7f1b3ef6e700 1 -- 172.21.15.98:0/3707526603 wait complete. 2024-04-25T17:50:09.229 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 51 2024-04-25T17:50:09.520 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:09 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3707526603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:50:09.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:09 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3707526603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:50:09.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:09 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3707526603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:50:09.905 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --set-id d4deafbb --release --await' 2024-04-25T17:50:09.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --set-id d4deafbb --release --await 2024-04-25T17:50:10.192 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:50:10.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:10 smithi098 ceph-mon[29192]: pgmap v449: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 30 op/s 2024-04-25T17:50:10.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:10 smithi098 ceph-mon[29192]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T17:50:10.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:10 smithi203 ceph-mon[32834]: pgmap v449: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 30 op/s 2024-04-25T17:50:10.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:10 smithi203 ceph-mon[32834]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T17:50:10.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:10 smithi173 ceph-mon[32856]: pgmap v449: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 74 KiB/s wr, 30 op/s 2024-04-25T17:50:10.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:10 smithi173 ceph-mon[32856]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T17:50:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 820.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.239+0000 7f0bf2c09700 1 -- 172.21.15.98:0/596596060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bec107990 msgr2=0x7f0bec107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:50:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.239+0000 7f0bf2c09700 1 --2- 172.21.15.98:0/596596060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bec107990 0x7f0bec107d70 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009990 tx=0x7f0be0009ca0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.239+0000 7f0bf2c09700 1 -- 172.21.15.98:0/596596060 shutdown_connections 2024-04-25T17:50:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.239+0000 7f0bf2c09700 1 --2- 172.21.15.98:0/596596060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bec102300 0x7f0bec10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.239+0000 7f0bf2c09700 1 --2- 172.21.15.98:0/596596060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bec107990 0x7f0bec107d70 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.239+0000 7f0bf2c09700 1 --2- 172.21.15.98:0/596596060 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bec101940 0x7f0bec101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.239+0000 7f0bf2c09700 1 -- 172.21.15.98:0/596596060 >> 172.21.15.98:0/596596060 conn(0x7f0bec0788f0 msgr2=0x7f0bec078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:50:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.239+0000 7f0bf2c09700 1 -- 172.21.15.98:0/596596060 shutdown_connections 2024-04-25T17:50:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.239+0000 7f0bf2c09700 1 -- 172.21.15.98:0/596596060 wait complete. 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf2c09700 1 Processor -- start 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf2c09700 1 -- start start 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf2c09700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bec101940 0x7f0bec199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf2c09700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bec102300 0x7f0bec1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf2c09700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bec107990 0x7f0bec19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf2c09700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec111ae0 con 0x7f0bec101940 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf2c09700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec111960 con 0x7f0bec107990 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf2c09700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec111c60 con 0x7f0bec102300 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf09a5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bec101940 0x7f0bec199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf09a5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bec101940 0x7f0bec199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50612/0 (socket says 172.21.15.98:50612) 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf09a5700 1 -- 172.21.15.98:0/1342986308 learned_addr learned my addr 172.21.15.98:0/1342986308 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bf11a6700 1 --2- 172.21.15.98:0/1342986308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bec107990 0x7f0bec19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.240+0000 7f0bebfff700 1 --2- 172.21.15.98:0/1342986308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bec102300 0x7f0bec1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0bf09a5700 1 -- 172.21.15.98:0/1342986308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bec102300 msgr2=0x7f0bec1995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0bf09a5700 1 --2- 172.21.15.98:0/1342986308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bec102300 0x7f0bec1995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0bf09a5700 1 -- 172.21.15.98:0/1342986308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bec107990 msgr2=0x7f0bec19d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0bf09a5700 1 --2- 172.21.15.98:0/1342986308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bec107990 0x7f0bec19d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0bf09a5700 1 -- 172.21.15.98:0/1342986308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bec19e050 con 0x7f0bec101940 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0bf09a5700 1 --2- 172.21.15.98:0/1342986308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bec101940 0x7f0bec199090 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0be00057c0 tx=0x7f0be0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0be9ffb700 1 -- 172.21.15.98:0/1342986308 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be0026070 con 0x7f0bec101940 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0bf2c09700 1 -- 172.21.15.98:0/1342986308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bec19e2e0 con 0x7f0bec101940 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0be9ffb700 1 -- 172.21.15.98:0/1342986308 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0be0021410 con 0x7f0bec101940 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0be9ffb700 1 -- 172.21.15.98:0/1342986308 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be0017650 con 0x7f0bec101940 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.241+0000 7f0bf2c09700 1 -- 172.21.15.98:0/1342986308 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bec1a5bc0 con 0x7f0bec101940 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.244+0000 7f0be9ffb700 1 -- 172.21.15.98:0/1342986308 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0be0029070 con 0x7f0bec101940 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.244+0000 7f0be9ffb700 1 --2- 172.21.15.98:0/1342986308 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0bdc0a3db0 0x7f0bdc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0bebfff700 1 --2- 172.21.15.98:0/1342986308 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0bdc0a3db0 0x7f0bdc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0be9ffb700 1 -- 172.21.15.98:0/1342986308 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8513+0+0 (secure 0 0 0) 0x7f0be00d1520 con 0x7f0bec101940 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0bebfff700 1 --2- 172.21.15.98:0/1342986308 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f0bdc0a3db0 0x7f0bdc0a6270 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0bec078e10 tx=0x7f0bd8009260 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0bfd492b80 1 Processor -- start 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0bfd492b80 1 -- start start 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c019f0 0x561386c01dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 0x561386c0cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c0f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561386b54e30 con 0x561386c0d1e0 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561386afba60 con 0x561386c019f0 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.245+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561386b50df0 con 0x561386c02310 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bf11a6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c0f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bf11a6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c0f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50614/0 (socket says 172.21.15.98:50614) 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bf11a6700 1 -- 172.21.15.98:0/1796804547 learned_addr learned my addr 172.21.15.98:0/1796804547 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bebfff700 1 --2- 172.21.15.98:0/1796804547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 0x561386c0cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bf09a5700 1 --2- 172.21.15.98:0/1796804547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c019f0 0x561386c01dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bf11a6700 1 -- 172.21.15.98:0/1796804547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 msgr2=0x561386c0cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bf11a6700 1 --2- 172.21.15.98:0/1796804547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 0x561386c0cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bf11a6700 1 -- 172.21.15.98:0/1796804547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c019f0 msgr2=0x561386c01dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bf11a6700 1 --2- 172.21.15.98:0/1796804547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c019f0 0x561386c01dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bf11a6700 1 -- 172.21.15.98:0/1796804547 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561386aa29d0 con 0x561386c0d1e0 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bf11a6700 1 --2- 172.21.15.98:0/1796804547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c0f5d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0be400f410 tx=0x7f0be400f720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf64572fabb40c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bd6ffd700 1 -- 172.21.15.98:0/1796804547 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be4018030 con 0x561386c0d1e0 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bd6ffd700 1 -- 172.21.15.98:0/1796804547 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0be4004bb0 con 0x561386c0d1e0 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.246+0000 7f0bd6ffd700 1 -- 172.21.15.98:0/1796804547 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be4005020 con 0x561386c0d1e0 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 -- 172.21.15.98:0/1796804547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 msgr2=0x561386c0f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/1796804547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c0f5d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0be400f410 tx=0x7f0be400f720 comp rx=0 tx=0).stop 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 -- 172.21.15.98:0/1796804547 shutdown_connections 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/1796804547 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c0f5d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/1796804547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c019f0 0x561386c01dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/1796804547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 0x561386c0cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 -- 172.21.15.98:0/1796804547 >> 172.21.15.98:0/1796804547 conn(0x561386b2f790 msgr2=0x561386bfded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 -- 172.21.15.98:0/1796804547 shutdown_connections 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 -- 172.21.15.98:0/1796804547 wait complete. 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 -- start start 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.247+0000 7f0bfd492b80 1 Processor -- start 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c019f0 0x561386c0c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c02310 0x561386c09bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c0d1e0 0x561386c0a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x561386c0fbd0 con 0x561386c02310 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561386af3cd0 con 0x561386c0d1e0 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561386c0fd50 con 0x561386c019f0 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bfd492b80 10 client.0 _ll_register_callbacks cb 0x561386aeda50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:50:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bebfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c02310 0x561386c09bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bebfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c02310 0x561386c09bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50622/0 (socket says 172.21.15.98:50622) 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bebfff700 1 -- 172.21.15.98:0/2890034496 learned_addr learned my addr 172.21.15.98:0/2890034496 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bf11a6700 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c0d1e0 0x561386c0a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bf09a5700 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c019f0 0x561386c0c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bebfff700 1 -- 172.21.15.98:0/2890034496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c019f0 msgr2=0x561386c0c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bebfff700 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c019f0 0x561386c0c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bebfff700 1 -- 172.21.15.98:0/2890034496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c0d1e0 msgr2=0x561386c0a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bebfff700 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c0d1e0 0x561386c0a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bebfff700 1 -- 172.21.15.98:0/2890034496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561386b0c370 con 0x561386c02310 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.248+0000 7f0bebfff700 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c02310 0x561386c09bd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8011fa0 tx=0x7f0bd80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261c42edfa6bcd30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bcbfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2890034496 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8003df0 con 0x561386c02310 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2890034496 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8513+0+0 (secure 0 0 0) 0x7f0bd801d410 con 0x561386c02310 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2890034496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0be4004bb0 con 0x561386c02310 2024-04-25T17:50:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2890034496 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0bd801d610 con 0x561386c02310 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bfd492b80 10 client.16863 fetch_fsmap learned FSMap version 51 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2890034496 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x561386c1a440 con 0x561386c02310 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2890034496 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0be4004bb0 con 0x561386c02310 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2890034496 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7f0bd8030810 con 0x561386c02310 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bfd492b80 10 client.16863 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bfd492b80 10 client.16863 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bfd492b80 20 client.16863 populate_metadata read hostname 'smithi098' 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561386c1c9d0 0x561386c1edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bfd492b80 4 client.16863 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.249+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2890034496 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561386b62620 con 0x561386c1c9d0 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.250+0000 7f0bf09a5700 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561386c1c9d0 0x561386c1edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.250+0000 7f0bf09a5700 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561386c1c9d0 0x561386c1edc0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43305c08ab143f79 server_cookie=215a3a6bcffa8b15 in_seq=0 out_seq=0 2024-04-25T17:50:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:50:12.250+0000 7f0bcbfff700 10 client.16863 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:50:12.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:12 smithi098 ceph-mon[29192]: pgmap v451: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 85 KiB/s wr, 33 op/s 2024-04-25T17:50:12.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:12 smithi203 ceph-mon[32834]: pgmap v451: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 85 KiB/s wr, 33 op/s 2024-04-25T17:50:12.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:12 smithi173 ceph-mon[32856]: pgmap v451: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 85 KiB/s wr, 33 op/s 2024-04-25T17:50:14.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:14 smithi098 ceph-mon[29192]: pgmap v452: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 90 KiB/s wr, 38 op/s 2024-04-25T17:50:14.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:14 smithi203 ceph-mon[32834]: pgmap v452: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 90 KiB/s wr, 38 op/s 2024-04-25T17:50:14.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:14 smithi173 ceph-mon[32856]: pgmap v452: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 90 KiB/s wr, 38 op/s 2024-04-25T17:50:16.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 825.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:16.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:16 smithi098 ceph-mon[29192]: pgmap v453: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 78 KiB/s wr, 33 op/s 2024-04-25T17:50:16.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:16 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:50:16.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:16 smithi203 ceph-mon[32834]: pgmap v453: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 78 KiB/s wr, 33 op/s 2024-04-25T17:50:16.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:16 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:50:16.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:16 smithi173 ceph-mon[32856]: pgmap v453: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 78 KiB/s wr, 33 op/s 2024-04-25T17:50:16.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:16 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:50:17.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:17 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:50:17.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:17 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:50:17.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:17 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:50:17.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:17 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:50:17.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:17 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:50:17.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:17 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:50:17.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:17 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:50:17.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:17 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:50:17.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:17 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:50:18.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:18 smithi098 ceph-mon[29192]: pgmap v454: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T17:50:18.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:18 smithi203 ceph-mon[32834]: pgmap v454: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T17:50:18.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:18 smithi173 ceph-mon[32856]: pgmap v454: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T17:50:20.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:20 smithi098 ceph-mon[29192]: pgmap v455: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T17:50:20.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:20 smithi203 ceph-mon[32834]: pgmap v455: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T17:50:20.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:20 smithi173 ceph-mon[32856]: pgmap v455: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2024-04-25T17:50:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 830.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:22.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:22 smithi098 ceph-mon[29192]: pgmap v456: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 KiB/s wr, 8 op/s 2024-04-25T17:50:22.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:22 smithi203 ceph-mon[32834]: pgmap v456: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 KiB/s wr, 8 op/s 2024-04-25T17:50:22.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:22 smithi173 ceph-mon[32856]: pgmap v456: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 KiB/s wr, 8 op/s 2024-04-25T17:50:24.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:24 smithi098 ceph-mon[29192]: pgmap v457: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-25T17:50:24.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:24 smithi203 ceph-mon[32834]: pgmap v457: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-25T17:50:24.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:24 smithi173 ceph-mon[32856]: pgmap v457: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-04-25T17:50:26.574 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:26 smithi173 ceph-mon[32856]: pgmap v458: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.1 KiB/s wr, 6 op/s 2024-04-25T17:50:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 835.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:26.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:26 smithi098 ceph-mon[29192]: pgmap v458: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.1 KiB/s wr, 6 op/s 2024-04-25T17:50:26.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:26 smithi203 ceph-mon[32834]: pgmap v458: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.1 KiB/s wr, 6 op/s 2024-04-25T17:50:28.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:28 smithi098 ceph-mon[29192]: pgmap v459: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-04-25T17:50:28.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:28 smithi203 ceph-mon[32834]: pgmap v459: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-04-25T17:50:28.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:28 smithi173 ceph-mon[32856]: pgmap v459: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 10 op/s 2024-04-25T17:50:30.601 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:30 smithi098 ceph-mon[29192]: pgmap v460: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 6 op/s 2024-04-25T17:50:30.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:30 smithi203 ceph-mon[32834]: pgmap v460: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 6 op/s 2024-04-25T17:50:30.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:30 smithi173 ceph-mon[32856]: pgmap v460: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 6 op/s 2024-04-25T17:50:31.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 840.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:32.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:32 smithi098 ceph-mon[29192]: pgmap v461: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 6 op/s 2024-04-25T17:50:32.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:32 smithi203 ceph-mon[32834]: pgmap v461: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 6 op/s 2024-04-25T17:50:32.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:32 smithi173 ceph-mon[32856]: pgmap v461: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 KiB/s wr, 6 op/s 2024-04-25T17:50:34.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:34 smithi098 ceph-mon[29192]: pgmap v462: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.3 KiB/s wr, 9 op/s 2024-04-25T17:50:34.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:34 smithi203 ceph-mon[32834]: pgmap v462: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.3 KiB/s wr, 9 op/s 2024-04-25T17:50:34.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:34 smithi173 ceph-mon[32856]: pgmap v462: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.3 KiB/s wr, 9 op/s 2024-04-25T17:50:36.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 845.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:36.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:36 smithi098 ceph-mon[29192]: pgmap v463: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.3 KiB/s wr, 6 op/s 2024-04-25T17:50:36.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:36 smithi203 ceph-mon[32834]: pgmap v463: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.3 KiB/s wr, 6 op/s 2024-04-25T17:50:36.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:36 smithi173 ceph-mon[32856]: pgmap v463: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.3 KiB/s wr, 6 op/s 2024-04-25T17:50:38.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:38 smithi098 ceph-mon[29192]: pgmap v464: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-25T17:50:38.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:38 smithi203 ceph-mon[32834]: pgmap v464: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-25T17:50:38.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:38 smithi173 ceph-mon[32856]: pgmap v464: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 10 op/s 2024-04-25T17:50:40.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:40 smithi098 ceph-mon[29192]: pgmap v465: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2024-04-25T17:50:40.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:40 smithi203 ceph-mon[32834]: pgmap v465: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2024-04-25T17:50:40.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:40 smithi173 ceph-mon[32856]: pgmap v465: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2024-04-25T17:50:41.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 850.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:42.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:42 smithi098 ceph-mon[29192]: pgmap v466: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2024-04-25T17:50:42.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:42 smithi203 ceph-mon[32834]: pgmap v466: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2024-04-25T17:50:42.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:42 smithi173 ceph-mon[32856]: pgmap v466: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2024-04-25T17:50:44.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:44 smithi098 ceph-mon[29192]: pgmap v467: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-04-25T17:50:44.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:44 smithi203 ceph-mon[32834]: pgmap v467: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-04-25T17:50:44.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:44 smithi173 ceph-mon[32856]: pgmap v467: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 11 op/s 2024-04-25T17:50:46.596 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 855.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:46.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:46 smithi098 ceph-mon[29192]: pgmap v468: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 20 KiB/s wr, 8 op/s 2024-04-25T17:50:46.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:46 smithi203 ceph-mon[32834]: pgmap v468: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 20 KiB/s wr, 8 op/s 2024-04-25T17:50:46.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:46 smithi173 ceph-mon[32856]: pgmap v468: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 20 KiB/s wr, 8 op/s 2024-04-25T17:50:48.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:48 smithi098 ceph-mon[29192]: pgmap v469: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 11 op/s 2024-04-25T17:50:48.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:48 smithi203 ceph-mon[32834]: pgmap v469: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 11 op/s 2024-04-25T17:50:48.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:48 smithi173 ceph-mon[32856]: pgmap v469: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 11 op/s 2024-04-25T17:50:50.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:50 smithi098 ceph-mon[29192]: pgmap v470: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-25T17:50:50.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:50 smithi203 ceph-mon[32834]: pgmap v470: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-25T17:50:50.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:50 smithi173 ceph-mon[32856]: pgmap v470: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-25T17:50:51.595 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 860.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:52.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:52 smithi098 ceph-mon[29192]: pgmap v471: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-25T17:50:52.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:52 smithi203 ceph-mon[32834]: pgmap v471: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-25T17:50:52.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:52 smithi173 ceph-mon[32856]: pgmap v471: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-25T17:50:54.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:54 smithi098 ceph-mon[29192]: pgmap v472: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-25T17:50:54.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:54 smithi203 ceph-mon[32834]: pgmap v472: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-25T17:50:54.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:54 smithi173 ceph-mon[32856]: pgmap v472: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 22 KiB/s wr, 10 op/s 2024-04-25T17:50:56.598 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 865.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:50:56.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:56 smithi098 ceph-mon[29192]: pgmap v473: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 KiB/s wr, 5 op/s 2024-04-25T17:50:56.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:56 smithi203 ceph-mon[32834]: pgmap v473: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 KiB/s wr, 5 op/s 2024-04-25T17:50:56.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:56 smithi173 ceph-mon[32856]: pgmap v473: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 KiB/s wr, 5 op/s 2024-04-25T17:50:58.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:50:58 smithi098 ceph-mon[29192]: pgmap v474: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-25T17:50:58.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:50:58 smithi203 ceph-mon[32834]: pgmap v474: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-25T17:50:58.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:50:58 smithi173 ceph-mon[32856]: pgmap v474: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-25T17:51:00.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:00 smithi098 ceph-mon[29192]: pgmap v475: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 5 op/s 2024-04-25T17:51:00.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:00 smithi203 ceph-mon[32834]: pgmap v475: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 5 op/s 2024-04-25T17:51:00.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:00 smithi173 ceph-mon[32856]: pgmap v475: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 5 op/s 2024-04-25T17:51:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 870.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-04-25T17:51:02.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:02 smithi098 ceph-mon[29192]: pgmap v476: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 5 op/s 2024-04-25T17:51:02.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:02 smithi203 ceph-mon[32834]: pgmap v476: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 5 op/s 2024-04-25T17:51:02.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:02 smithi173 ceph-mon[32856]: pgmap v476: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 5 op/s 2024-04-25T17:51:04.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:04 smithi098 ceph-mon[29192]: pgmap v477: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.7 KiB/s wr, 8 op/s 2024-04-25T17:51:04.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:04 smithi203 ceph-mon[32834]: pgmap v477: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.7 KiB/s wr, 8 op/s 2024-04-25T17:51:04.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:04 smithi173 ceph-mon[32856]: pgmap v477: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.7 KiB/s wr, 8 op/s 2024-04-25T17:51:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.893+0000 7f780effd640 1 -- 172.21.15.98:0/3940180686 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e56275e200 con 0x55e56280c670 2024-04-25T17:51:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.893+0000 7f780effd640 10 client.16845 handle_command_reply: tid=0 2024-04-25T17:51:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.893+0000 7f7838dba740 1 client.16845 shutdown 2024-04-25T17:51:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f7838dba740 1 -- 172.21.15.98:0/3940180686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 msgr2=0x55e5628047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:05.893 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f7838dba740 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5628047a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7818003a90 tx=0x7f781802a410 comp rx=0 tx=0).stop 2024-04-25T17:51:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f7838dba740 1 -- 172.21.15.98:0/3940180686 shutdown_connections 2024-04-25T17:51:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f7838dba740 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e56280c670 0x55e562874b10 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f7838dba740 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e5627fc800 0x55e5627f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f7838dba740 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e5627f1f40 0x55e562804ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f7838dba740 1 --2- 172.21.15.98:0/3940180686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e5627f1620 0x55e5628047a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.894 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f7838dba740 1 -- 172.21.15.98:0/3940180686 >> 172.21.15.98:0/3940180686 conn(0x55e5627edb30 msgr2=0x55e5627f2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:05.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.893+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2890034496 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561386b62620 con 0x561386c1c9d0 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.893+0000 7f0bcbfff700 10 client.16863 handle_command_reply: tid=0 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f0bfd492b80 1 client.16863 shutdown 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2890034496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c02310 msgr2=0x561386c09bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c02310 0x561386c09bd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8011fa0 tx=0x7f0bd80047c0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2890034496 shutdown_connections 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.894+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561386c1c9d0 0x561386c1edc0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.895+0000 7f0bcbfff700 0 client.16863 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.895+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c019f0 0x561386c0c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.895+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c0d1e0 0x561386c0a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.895+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2890034496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c02310 0x561386c09bd0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.895+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2890034496 >> 172.21.15.98:0/2890034496 conn(0x561386b2f790 msgr2=0x561386bfdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:51:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:51:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:51:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:51:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.895+0000 7f780effd640 0 client.16845 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.895+0000 7f7838dba740 1 -- 172.21.15.98:0/3940180686 shutdown_connections 2024-04-25T17:51:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.895+0000 7f7838dba740 1 -- 172.21.15.98:0/3940180686 wait complete. 2024-04-25T17:51:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.895+0000 7f7838dba740 20 client.16845 trim_cache size 0 max 16384 2024-04-25T17:51:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.896+0000 7f7835fc6640 1 -- 172.21.15.98:0/3171625815 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f781c0a3e40 msgr2=0x7f781c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.896+0000 7f7835fc6640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f781c0a3e40 0x7f781c0a6300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f782809c8e0 tx=0x7f782808e040 comp rx=0 tx=0).stop 2024-04-25T17:51:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.896+0000 7f7835fc6640 1 -- 172.21.15.98:0/3171625815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783019cf60 msgr2=0x7f783019d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:05.898 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.896+0000 7f7835fc6640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783019cf60 0x7f783019d410 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f782800eaa0 tx=0x7f782800ef60 comp rx=0 tx=0).stop 2024-04-25T17:51:05.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.896+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2890034496 shutdown_connections 2024-04-25T17:51:05.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.896+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2890034496 wait complete. 2024-04-25T17:51:05.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.896+0000 7f0bfd492b80 20 client.16863 trim_cache size 0 max 16384 2024-04-25T17:51:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.902+0000 7f7835fc6640 1 -- 172.21.15.98:0/3171625815 shutdown_connections 2024-04-25T17:51:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.902+0000 7f7835fc6640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f781c0a3e40 0x7f781c0a6300 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.902+0000 7f7835fc6640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f783019cf60 0x7f783019d410 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.902+0000 7f7835fc6640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7830106170 0x7f7830198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.902+0000 7f7835fc6640 1 --2- 172.21.15.98:0/3171625815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78301010f0 0x7f78301986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.902+0000 7f7835fc6640 1 -- 172.21.15.98:0/3171625815 >> 172.21.15.98:0/3171625815 conn(0x7f7830077fd0 msgr2=0x7f78300699b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.902+0000 7f7835fc6640 1 -- 172.21.15.98:0/3171625815 shutdown_connections 2024-04-25T17:51:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:05.902+0000 7f7835fc6640 1 -- 172.21.15.98:0/3171625815 wait complete. 2024-04-25T17:51:05.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.936+0000 7f0bfd492b80 1 Processor -- start 2024-04-25T17:51:05.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.937+0000 7f0bfd492b80 1 -- start start 2024-04-25T17:51:05.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.937+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c019f0 0x561386c24700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:05.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.937+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c02310 0x561386c24c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:05.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.937+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c14fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:05.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.937+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec1afb40 con 0x561386c0d1e0 2024-04-25T17:51:05.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec1ad3d0 con 0x561386c02310 2024-04-25T17:51:05.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bf11a6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c14fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:05.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bf11a6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c14fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55034/0 (socket says 172.21.15.98:55034) 2024-04-25T17:51:05.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bf11a6700 1 -- 172.21.15.98:0/2562338680 learned_addr learned my addr 172.21.15.98:0/2562338680 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:05.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bebfff700 1 --2- 172.21.15.98:0/2562338680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c02310 0x561386c24c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:05.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec1ab240 con 0x561386c019f0 2024-04-25T17:51:05.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bf11a6700 1 -- 172.21.15.98:0/2562338680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c019f0 msgr2=0x561386c24700 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T17:51:05.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bf11a6700 1 --2- 172.21.15.98:0/2562338680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c019f0 0x561386c24700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bf11a6700 1 -- 172.21.15.98:0/2562338680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c02310 msgr2=0x561386c24c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:05.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bf11a6700 1 --2- 172.21.15.98:0/2562338680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c02310 0x561386c24c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.938+0000 7f0bf11a6700 1 -- 172.21.15.98:0/2562338680 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561386c031b0 con 0x561386c0d1e0 2024-04-25T17:51:05.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.939+0000 7f0bf11a6700 1 --2- 172.21.15.98:0/2562338680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c14fc0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0be4018000 tx=0x7f0be4005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da664ed17c0d059 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:05.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.940+0000 7f0bd67fc700 1 -- 172.21.15.98:0/2562338680 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be4014470 con 0x561386c0d1e0 2024-04-25T17:51:05.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.940+0000 7f0bd67fc700 1 -- 172.21.15.98:0/2562338680 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0be40050b0 con 0x561386c0d1e0 2024-04-25T17:51:05.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.941+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2562338680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 msgr2=0x561386c14fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:05.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.941+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2562338680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c14fc0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0be4018000 tx=0x7f0be4005040 comp rx=0 tx=0).stop 2024-04-25T17:51:05.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.941+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2562338680 shutdown_connections 2024-04-25T17:51:05.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.941+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2562338680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c019f0 0x561386c24700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.941+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2562338680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c0d1e0 0x561386c14fc0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.941+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2562338680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c02310 0x561386c24c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.941+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2562338680 >> 172.21.15.98:0/2562338680 conn(0x561386b2f790 msgr2=0x561386c084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:05.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.942+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2562338680 shutdown_connections 2024-04-25T17:51:05.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.942+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2562338680 wait complete. 2024-04-25T17:51:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.943+0000 7f0bfd492b80 1 -- start start 2024-04-25T17:51:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.943+0000 7f0bfd492b80 1 Processor -- start 2024-04-25T17:51:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.943+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c019f0 0x561386c7ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.944+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 0x561386c7b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.944+0000 7f0bfd492b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c0d1e0 0x561386c7b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.944+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec1ab890 con 0x561386c019f0 2024-04-25T17:51:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.944+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec1acd00 con 0x561386c0d1e0 2024-04-25T17:51:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.944+0000 7f0bfd492b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0bec1abf60 con 0x561386c02310 2024-04-25T17:51:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.944+0000 7f0bfd492b80 10 client.0 _ll_register_callbacks cb 0x561386aeda50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:51:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.945+0000 7f0bebfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 0x561386c7b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.945+0000 7f0bebfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 0x561386c7b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:48358/0 (socket says 172.21.15.98:48358) 2024-04-25T17:51:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.945+0000 7f0bebfff700 1 -- 172.21.15.98:0/2782541028 learned_addr learned my addr 172.21.15.98:0/2782541028 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.945+0000 7f0bf09a5700 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c019f0 0x561386c7ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.945+0000 7f0bf11a6700 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c0d1e0 0x561386c7b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.946+0000 7f0bebfff700 1 -- 172.21.15.98:0/2782541028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c0d1e0 msgr2=0x561386c7b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.946+0000 7f0bebfff700 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c0d1e0 0x561386c7b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.946+0000 7f0bebfff700 1 -- 172.21.15.98:0/2782541028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c019f0 msgr2=0x561386c7ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.946+0000 7f0bebfff700 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c019f0 0x561386c7ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.946+0000 7f0bebfff700 1 -- 172.21.15.98:0/2782541028 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561386c15890 con 0x561386c02310 2024-04-25T17:51:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.946+0000 7f0bebfff700 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 0x561386c7b010 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8011850 tx=0x7f0bd8035000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d50609ccaac5e59c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.946+0000 7f0bcbfff700 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:51:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.949+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2782541028 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8034050 con 0x561386c02310 2024-04-25T17:51:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.949+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2782541028 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8557+0+0 (secure 0 0 0) 0x7f0bd801d410 con 0x561386c02310 2024-04-25T17:51:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.949+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2782541028 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0be40050b0 con 0x561386c02310 2024-04-25T17:51:05.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.949+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2782541028 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f0bd801d7b0 con 0x561386c02310 2024-04-25T17:51:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.955+0000 7f0bfd492b80 10 client.26299 fetch_fsmap learned FSMap version 51 2024-04-25T17:51:05.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.955+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2782541028 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x561386c7fee0 con 0x561386c02310 2024-04-25T17:51:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.955+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2782541028 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0be40050b0 con 0x561386c02310 2024-04-25T17:51:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.955+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2782541028 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7f0bd80387d0 con 0x561386c02310 2024-04-25T17:51:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.956+0000 7f0bfd492b80 10 client.26299 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:51:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.956+0000 7f0bfd492b80 10 client.26299 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:51:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.956+0000 7f0bfd492b80 20 client.26299 populate_metadata read hostname 'smithi098' 2024-04-25T17:51:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.956+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561386c1c9d0 0x561386c82620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.956+0000 7f0bfd492b80 4 client.26299 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "d4deafbb", "release": true, "await": true}] 2024-04-25T17:51:05.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.956+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2782541028 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "set_id": "d4deafbb", "release": true, "await": true}) -- 0x561386b5c6c0 con 0x561386c1c9d0 2024-04-25T17:51:05.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.956+0000 7f0bf09a5700 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561386c1c9d0 0x561386c82620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.961+0000 7f0bcbfff700 10 client.26299 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.961+0000 7f0bf09a5700 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561386c1c9d0 0x561386c82620 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6738e882756faf59 server_cookie=74b0aba322e7c60e in_seq=0 out_seq=0 2024-04-25T17:51:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.961+0000 7f0bcbfff700 1 -- 172.21.15.98:0/2782541028 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: -1 ) ==== 8+0+590 (crc 0 0 0) 0x561386b5c6c0 con 0x561386c1c9d0 2024-04-25T17:51:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.961+0000 7f0bcbfff700 10 client.26299 handle_command_reply: tid=0 2024-04-25T17:51:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.961+0000 7f0bfd492b80 1 client.26299 shutdown 2024-04-25T17:51:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.962+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2782541028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 msgr2=0x561386c7b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:05.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.962+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 0x561386c7b010 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8011850 tx=0x7f0bd8035000 comp rx=0 tx=0).stop 2024-04-25T17:51:05.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.964+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2782541028 shutdown_connections 2024-04-25T17:51:05.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.964+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x561386c1c9d0 0x561386c82620 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.964+0000 7f0bcbfff700 0 client.26299 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:05.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.964+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561386c0d1e0 0x561386c7b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.964+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561386c019f0 0x561386c7ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.964+0000 7f0bfd492b80 1 --2- 172.21.15.98:0/2782541028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561386c02310 0x561386c7b010 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:05.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.964+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2782541028 >> 172.21.15.98:0/2782541028 conn(0x561386b2f790 msgr2=0x561386bfd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:05.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.965+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2782541028 shutdown_connections 2024-04-25T17:51:05.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.965+0000 7f0bfd492b80 1 -- 172.21.15.98:0/2782541028 wait complete. 2024-04-25T17:51:05.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:05.965+0000 7f0bfd492b80 20 client.26299 trim_cache size 0 max 16384 2024-04-25T17:51:05.965 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:51:05.965 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 51, 2024-04-25T17:51:05.965 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 43, 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": { 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "d4deafbb": { 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "version": 43, 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "age_ref": 0.3, 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "EXPIRED", 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 0.0 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "timeout": 60.0, 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "expiration": 89.1, 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "members": { 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "file:/": { 2024-04-25T17:51:05.966 INFO:teuthology.orchestra.run.smithi098.stdout: "excluded": false, 2024-04-25T17:51:05.967 INFO:teuthology.orchestra.run.smithi098.stdout: "state": { 2024-04-25T17:51:05.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "QUIESCED", 2024-04-25T17:51:05.967 INFO:teuthology.orchestra.run.smithi098.stdout: "age": 89.1 2024-04-25T17:51:05.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:05.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:05.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:05.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:05.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:05.967 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:51:05.967 INFO:teuthology.orchestra.run.smithi098.stderr:Error EPERM: 2024-04-25T17:51:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 875.0 s, 31.6 tps, lat 46067.082 ms stddev 44598.178 2024-04-25T17:51:06.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:06 smithi098 ceph-mon[29192]: pgmap v478: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-25T17:51:06.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:06 smithi098 ceph-mon[29192]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T17:51:06.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:06 smithi203 ceph-mon[32834]: pgmap v478: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-25T17:51:06.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:06 smithi203 ceph-mon[32834]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T17:51:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:51:06.723 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't release set 'd4deafbb' with rc: 1 (EPERM), stdout: { "epoch": 51, "leader": 24461, "set_version": 43, "sets": { "d4deafbb": { "version": 43, "age_ref": 0.3, "state": { "name": "EXPIRED", "age": 0.0 }, "timeout": 60.0, "expiration": 89.1, "members": { "file:/": { "excluded": false, "state": { "name": "QUIESCED", "age": 89.1 } } } } } } 2024-04-25T17:51:06.723 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_3071830b3533a96301fd87d582ed5f17f0b618cd/qa/tasks/quiescer.py", line 346, in _run self.do_quiesce(d) File "/home/teuthworker/src/github.com_ceph_ceph_3071830b3533a96301fd87d582ed5f17f0b618cd/qa/tasks/quiescer.py", line 322, in do_quiesce raise RuntimeError(f"Error releasing set '{set_id}': {rcinfo}") RuntimeError: Error releasing set 'd4deafbb': 1 (EPERM) 2024-04-25T17:51:06.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:06 smithi173 ceph-mon[32856]: pgmap v478: 97 pgs: 97 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-25T17:51:06.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:06 smithi173 ceph-mon[32856]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T17:51:08.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:08 smithi098 ceph-mon[29192]: pgmap v480: 97 pgs: 5 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 75 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 425 KiB/s wr, 61 op/s 2024-04-25T17:51:08.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:08 smithi203 ceph-mon[32834]: pgmap v480: 97 pgs: 5 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 75 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 425 KiB/s wr, 61 op/s 2024-04-25T17:51:08.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:08 smithi173 ceph-mon[32856]: pgmap v480: 97 pgs: 5 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 75 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 425 KiB/s wr, 61 op/s 2024-04-25T17:51:10.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:10 smithi098 ceph-mon[29192]: pgmap v481: 97 pgs: 5 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 75 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 425 KiB/s wr, 61 op/s 2024-04-25T17:51:10.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:10 smithi203 ceph-mon[32834]: pgmap v481: 97 pgs: 5 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 75 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 425 KiB/s wr, 61 op/s 2024-04-25T17:51:10.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:10 smithi173 ceph-mon[32856]: pgmap v481: 97 pgs: 5 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 75 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 425 KiB/s wr, 61 op/s 2024-04-25T17:51:11.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 880.0 s, 1839.8 tps, lat 242.540 ms stddev 4119.966 2024-04-25T17:51:11.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:11 smithi098 ceph-mon[29192]: pgmap v482: 97 pgs: 5 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 75 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 425 KiB/s wr, 61 op/s 2024-04-25T17:51:11.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:11 smithi203 ceph-mon[32834]: pgmap v482: 97 pgs: 5 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 75 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 425 KiB/s wr, 61 op/s 2024-04-25T17:51:11.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:11 smithi173 ceph-mon[32856]: pgmap v482: 97 pgs: 5 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 75 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 425 KiB/s wr, 61 op/s 2024-04-25T17:51:13.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:13 smithi098 ceph-mon[29192]: pgmap v483: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-25T17:51:13.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:13 smithi203 ceph-mon[32834]: pgmap v483: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-25T17:51:14.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:13 smithi173 ceph-mon[32856]: pgmap v483: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-25T17:51:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:16 smithi098 ceph-mon[29192]: pgmap v484: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-25T17:51:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:16 smithi098 ceph-mon[29192]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T17:51:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:16 smithi203 ceph-mon[32834]: pgmap v484: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-25T17:51:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:16 smithi203 ceph-mon[32834]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T17:51:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:16 smithi173 ceph-mon[32856]: pgmap v484: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 559 op/s 2024-04-25T17:51:16.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:16 smithi173 ceph-mon[32856]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T17:51:16.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 885.0 s, 2214.0 tps, lat 45.538 ms stddev 52.191 2024-04-25T17:51:17.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:17 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:51:17.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:17 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:51:17.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:17 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:51:18.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:18 smithi203 ceph-mon[32834]: pgmap v486: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 20 MiB/s wr, 839 op/s 2024-04-25T17:51:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:18 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:51:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:18 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:51:18.462 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:18 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:51:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:18 smithi173 ceph-mon[32856]: pgmap v486: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 20 MiB/s wr, 839 op/s 2024-04-25T17:51:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:18 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:51:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:18 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:51:18.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:18 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:51:18.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:18 smithi098 ceph-mon[29192]: pgmap v486: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 20 MiB/s wr, 839 op/s 2024-04-25T17:51:18.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:18 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:51:18.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:18 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:51:18.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:18 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:51:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.038+0000 7fd790764640 1 -- 172.21.15.98:0/961247473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7881006c0 msgr2=0x7fd788100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.038+0000 7fd790764640 1 --2- 172.21.15.98:0/961247473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7881006c0 0x7fd788100aa0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd78400fb10 tx=0x7fd78402e570 comp rx=0 tx=0).stop 2024-04-25T17:51:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.038+0000 7fd790764640 1 -- 172.21.15.98:0/961247473 shutdown_connections 2024-04-25T17:51:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.038+0000 7fd790764640 1 --2- 172.21.15.98:0/961247473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd788107280 0x7fd78810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.038+0000 7fd790764640 1 --2- 172.21.15.98:0/961247473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd788101070 0x7fd788106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.038+0000 7fd790764640 1 --2- 172.21.15.98:0/961247473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7881006c0 0x7fd788100aa0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.038+0000 7fd790764640 1 -- 172.21.15.98:0/961247473 >> 172.21.15.98:0/961247473 conn(0x7fd788077390 msgr2=0x7fd7880fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.038+0000 7fd790764640 1 -- 172.21.15.98:0/961247473 shutdown_connections 2024-04-25T17:51:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.039+0000 7fd790764640 1 -- 172.21.15.98:0/961247473 wait complete. 2024-04-25T17:51:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.039+0000 7fd790764640 1 Processor -- start 2024-04-25T17:51:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd790764640 1 -- start start 2024-04-25T17:51:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd790764640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7881006c0 0x7fd7881964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd790764640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd788107280 0x7fd788196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd790764640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd78819ad80 0x7fd78819b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd790764640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd788110020 con 0x7fd788107280 2024-04-25T17:51:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd790764640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd78810fea0 con 0x7fd7881006c0 2024-04-25T17:51:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd790764640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7881101a0 con 0x7fd78819ad80 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd78dcff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd788107280 0x7fd788196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd78e500640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7881006c0 0x7fd7881964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd78dcff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd788107280 0x7fd788196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40960/0 (socket says 172.21.15.98:40960) 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd78dcff640 1 -- 172.21.15.98:0/4223782504 learned_addr learned my addr 172.21.15.98:0/4223782504 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd78ed01640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd78819ad80 0x7fd78819b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd78dcff640 1 -- 172.21.15.98:0/4223782504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd78819ad80 msgr2=0x7fd78819b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd78dcff640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd78819ad80 0x7fd78819b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd78dcff640 1 -- 172.21.15.98:0/4223782504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7881006c0 msgr2=0x7fd7881964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.040+0000 7fd78dcff640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7881006c0 0x7fd7881964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.041+0000 7fd78dcff640 1 -- 172.21.15.98:0/4223782504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78819b870 con 0x7fd788107280 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.041+0000 7fd78e500640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7881006c0 0x7fd7881964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:51:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.041+0000 7fd78dcff640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd788107280 0x7fd788196a00 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd77800c970 tx=0x7fd77800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.041+0000 7fd77f7fe640 1 -- 172.21.15.98:0/4223782504 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778014070 con 0x7fd788107280 2024-04-25T17:51:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.041+0000 7fd77f7fe640 1 -- 172.21.15.98:0/4223782504 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd778004440 con 0x7fd788107280 2024-04-25T17:51:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.041+0000 7fd77f7fe640 1 -- 172.21.15.98:0/4223782504 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778003e50 con 0x7fd788107280 2024-04-25T17:51:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.041+0000 7fd790764640 1 -- 172.21.15.98:0/4223782504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78819bb60 con 0x7fd788107280 2024-04-25T17:51:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.041+0000 7fd790764640 1 -- 172.21.15.98:0/4223782504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7881a32a0 con 0x7fd788107280 2024-04-25T17:51:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.044+0000 7fd77f7fe640 1 -- 172.21.15.98:0/4223782504 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd778017070 con 0x7fd788107280 2024-04-25T17:51:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.045+0000 7fd77f7fe640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd7740a4110 0x7fd7740a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.045+0000 7fd77f7fe640 1 -- 172.21.15.98:0/4223782504 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fd7780d04f0 con 0x7fd788107280 2024-04-25T17:51:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.045+0000 7fd79347f740 1 Processor -- start 2024-04-25T17:51:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.045+0000 7fd78e500640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd7740a4110 0x7fd7740a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd79347f740 1 -- start start 2024-04-25T17:51:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd79347f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc53931620 0x55cc53931a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd79347f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5393c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd79347f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 0x55cc5393ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd79347f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc53896970 con 0x55cc53931620 2024-04-25T17:51:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd79347f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cc538334a0 con 0x55cc5393c800 2024-04-25T17:51:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd79347f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cc53892930 con 0x55cc53931f40 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd78e500640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc53931620 0x55cc53931a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd78ed01640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 0x55cc5393ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd78dcff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5393c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd78dcff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5393c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:35148/0 (socket says 172.21.15.98:35148) 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd78e500640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc53931620 0x55cc53931a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40972/0 (socket says 172.21.15.98:40972) 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd78ed01640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 0x55cc5393ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:42692/0 (socket says 172.21.15.98:42692) 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.046+0000 7fd78dcff640 1 -- 172.21.15.98:0/484995784 learned_addr learned my addr 172.21.15.98:0/484995784 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.047+0000 7fd78dcff640 1 -- 172.21.15.98:0/484995784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 msgr2=0x55cc5393ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.047+0000 7fd78e500640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd7740a4110 0x7fd7740a65d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd78402eb60 tx=0x7fd7840041a0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.047+0000 7fd78dcff640 1 --2- 172.21.15.98:0/484995784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 0x55cc5393ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.047+0000 7fd78dcff640 1 -- 172.21.15.98:0/484995784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc53931620 msgr2=0x55cc53931a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.047+0000 7fd78dcff640 1 --2- 172.21.15.98:0/484995784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc53931620 0x55cc53931a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.047+0000 7fd78dcff640 1 -- 172.21.15.98:0/484995784 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc535a9210 con 0x55cc53931f40 2024-04-25T17:51:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.047+0000 7fd78dcff640 1 --2- 172.21.15.98:0/484995784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5393c2c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7780a4bd0 tx=0x7fd77809c0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb525fffccf393ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.047+0000 7fd77cff9640 1 -- 172.21.15.98:0/484995784 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7780b4020 con 0x55cc53931f40 2024-04-25T17:51:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.047+0000 7fd77cff9640 1 -- 172.21.15.98:0/484995784 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd77809c490 con 0x55cc53931f40 2024-04-25T17:51:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.047+0000 7fd77cff9640 1 -- 172.21.15.98:0/484995784 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7780981e0 con 0x55cc53931f40 2024-04-25T17:51:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.048+0000 7fd79347f740 1 -- 172.21.15.98:0/484995784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 msgr2=0x55cc5393c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.048+0000 7fd79347f740 1 --2- 172.21.15.98:0/484995784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5393c2c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7780a4bd0 tx=0x7fd77809c0d0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.048+0000 7fd79347f740 1 -- 172.21.15.98:0/484995784 shutdown_connections 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.048+0000 7fd79347f740 1 --2- 172.21.15.98:0/484995784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 0x55cc5393ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.048+0000 7fd79347f740 1 --2- 172.21.15.98:0/484995784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5393c2c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.048+0000 7fd79347f740 1 --2- 172.21.15.98:0/484995784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc53931620 0x55cc53931a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.048+0000 7fd79347f740 1 -- 172.21.15.98:0/484995784 >> 172.21.15.98:0/484995784 conn(0x55cc5392db30 msgr2=0x55cc5392df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.048+0000 7fd79347f740 1 -- 172.21.15.98:0/484995784 shutdown_connections 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.048+0000 7fd79347f740 1 -- 172.21.15.98:0/484995784 wait complete. 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.049+0000 7fd79347f740 1 -- start start 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.049+0000 7fd79347f740 1 Processor -- start 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.049+0000 7fd79347f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931620 0x55cc5393c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.049+0000 7fd79347f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc53931f40 0x55cc53939420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.049+0000 7fd79347f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 0x55cc53939960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.049+0000 7fd79347f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55cc5393f430 con 0x55cc53931f40 2024-04-25T17:51:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.049+0000 7fd78e500640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931620 0x55cc5393c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.049+0000 7fd78e500640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931620 0x55cc5393c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:35158/0 (socket says 172.21.15.98:35158) 2024-04-25T17:51:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.049+0000 7fd78e500640 1 -- 172.21.15.98:0/2788449338 learned_addr learned my addr 172.21.15.98:0/2788449338 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.050+0000 7fd78ed01640 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 0x55cc53939960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.050+0000 7fd79347f740 1 -- 172.21.15.98:0/2788449338 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55cc538310b0 con 0x55cc5393c800 2024-04-25T17:51:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.050+0000 7fd79347f740 1 -- 172.21.15.98:0/2788449338 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cc5393f5b0 con 0x55cc53931620 2024-04-25T17:51:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.050+0000 7fd79347f740 10 client.0 _ll_register_callbacks cb 0x55cc5381ed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.050+0000 7fd78e500640 1 -- 172.21.15.98:0/2788449338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 msgr2=0x55cc53939960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.050+0000 7fd78e500640 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 0x55cc53939960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.050+0000 7fd78e500640 1 -- 172.21.15.98:0/2788449338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc53931f40 msgr2=0x55cc53939420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.050+0000 7fd78e500640 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc53931f40 0x55cc53939420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.050+0000 7fd78e500640 1 -- 172.21.15.98:0/2788449338 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc53675620 con 0x55cc53931620 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.050+0000 7fd78e500640 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931620 0x55cc5393c130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd78400fb10 tx=0x7fd78404a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28010ee11aec1c5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.051+0000 7fd771ffb640 10 client.26308 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.051+0000 7fd79347f740 1 -- 172.21.15.98:0/2788449338 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd77809c490 con 0x55cc53931620 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.051+0000 7fd771ffb640 1 -- 172.21.15.98:0/2788449338 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784055060 con 0x55cc53931620 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.051+0000 7fd771ffb640 1 -- 172.21.15.98:0/2788449338 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fd78400f5a0 con 0x55cc53931620 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.051+0000 7fd771ffb640 1 -- 172.21.15.98:0/2788449338 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fd784003ac0 con 0x55cc53931620 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.051+0000 7fd79347f740 10 client.26308 fetch_fsmap learned FSMap version 51 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.051+0000 7fd79347f740 1 -- 172.21.15.98:0/2788449338 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=51}) -- 0x55cc53953a80 con 0x55cc53931620 2024-04-25T17:51:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.052+0000 7fd771ffb640 1 -- 172.21.15.98:0/2788449338 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7fd784003c60 con 0x55cc53931620 2024-04-25T17:51:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.052+0000 7fd79347f740 10 client.26308 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:51:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.052+0000 7fd79347f740 10 client.26308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:51:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.052+0000 7fd79347f740 20 client.26308 populate_metadata read hostname 'smithi098' 2024-04-25T17:51:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.052+0000 7fd79347f740 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc53954e30 0x55cc53957280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.052+0000 7fd79347f740 4 client.26308 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:51:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.052+0000 7fd79347f740 1 -- 172.21.15.98:0/2788449338 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd77809c490 con 0x55cc53954e30 2024-04-25T17:51:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.052+0000 7fd78dcff640 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc53954e30 0x55cc53957280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.054+0000 7fd78dcff640 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc53954e30 0x55cc53957280 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df6b921fa0e212ea server_cookie=388f242d98e47af7 in_seq=0 out_seq=0 2024-04-25T17:51:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.054+0000 7fd771ffb640 10 client.26308 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.054+0000 7fd771ffb640 1 -- 172.21.15.98:0/2788449338 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fd77809c490 con 0x55cc53954e30 2024-04-25T17:51:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.054+0000 7fd771ffb640 10 client.26308 handle_command_reply: tid=0 2024-04-25T17:51:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.054+0000 7fd79347f740 1 client.26308 shutdown 2024-04-25T17:51:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 1 -- 172.21.15.98:0/2788449338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931620 msgr2=0x55cc5393c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931620 0x55cc5393c130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd78400fb10 tx=0x7fd78404a000 comp rx=0 tx=0).stop 2024-04-25T17:51:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 1 -- 172.21.15.98:0/2788449338 shutdown_connections 2024-04-25T17:51:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc53954e30 0x55cc53957280 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc5393c800 0x55cc53939960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc53931f40 0x55cc53939420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 1 --2- 172.21.15.98:0/2788449338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931620 0x55cc5393c130 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd771ffb640 0 client.26308 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 1 -- 172.21.15.98:0/2788449338 >> 172.21.15.98:0/2788449338 conn(0x55cc5392db30 msgr2=0x55cc539338b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 1 -- 172.21.15.98:0/2788449338 shutdown_connections 2024-04-25T17:51:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 1 -- 172.21.15.98:0/2788449338 wait complete. 2024-04-25T17:51:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.055+0000 7fd79347f740 20 client.26308 trim_cache size 0 max 16384 2024-04-25T17:51:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd79347f740 1 Processor -- start 2024-04-25T17:51:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd79347f740 1 -- start start 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd79347f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 0x55cc5395cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd79347f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5395d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd79347f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5392cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd79347f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7881ae240 con 0x55cc5393c800 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd79347f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd7881abb10 con 0x55cc53931620 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd79347f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7881a9980 con 0x55cc53931f40 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd78ed01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5392cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd78ed01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5392cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40998/0 (socket says 172.21.15.98:40998) 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd78ed01640 1 -- 172.21.15.98:0/2046136304 learned_addr learned my addr 172.21.15.98:0/2046136304 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd78dcff640 1 --2- 172.21.15.98:0/2046136304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5395d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.075+0000 7fd78e500640 1 --2- 172.21.15.98:0/2046136304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 0x55cc5395cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.076+0000 7fd78ed01640 1 -- 172.21.15.98:0/2046136304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 msgr2=0x55cc5395d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.076+0000 7fd78ed01640 1 --2- 172.21.15.98:0/2046136304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5395d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.076+0000 7fd78ed01640 1 -- 172.21.15.98:0/2046136304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 msgr2=0x55cc5395cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.076+0000 7fd78ed01640 1 --2- 172.21.15.98:0/2046136304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 0x55cc5395cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.076+0000 7fd78ed01640 1 -- 172.21.15.98:0/2046136304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc53934fe0 con 0x55cc5393c800 2024-04-25T17:51:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.076+0000 7fd78ed01640 1 --2- 172.21.15.98:0/2046136304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5392cb30 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd780005fd0 tx=0x7fd780013880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba211cb6bf7b91a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.076+0000 7fd773fff640 1 -- 172.21.15.98:0/2046136304 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780004b20 con 0x55cc5393c800 2024-04-25T17:51:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.076+0000 7fd773fff640 1 -- 172.21.15.98:0/2046136304 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd780003e20 con 0x55cc5393c800 2024-04-25T17:51:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.076+0000 7fd773fff640 1 -- 172.21.15.98:0/2046136304 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780005020 con 0x55cc5393c800 2024-04-25T17:51:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.077+0000 7fd79347f740 1 -- 172.21.15.98:0/2046136304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 msgr2=0x55cc5392cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.077+0000 7fd79347f740 1 --2- 172.21.15.98:0/2046136304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5392cb30 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd780005fd0 tx=0x7fd780013880 comp rx=0 tx=0).stop 2024-04-25T17:51:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.077+0000 7fd79347f740 1 -- 172.21.15.98:0/2046136304 shutdown_connections 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.078+0000 7fd79347f740 1 --2- 172.21.15.98:0/2046136304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5392cb30 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.078+0000 7fd79347f740 1 --2- 172.21.15.98:0/2046136304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5395d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.078+0000 7fd79347f740 1 --2- 172.21.15.98:0/2046136304 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 0x55cc5395cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.078+0000 7fd79347f740 1 -- 172.21.15.98:0/2046136304 >> 172.21.15.98:0/2046136304 conn(0x55cc5392db30 msgr2=0x55cc5392c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.078+0000 7fd79347f740 1 -- 172.21.15.98:0/2046136304 shutdown_connections 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.078+0000 7fd79347f740 1 -- 172.21.15.98:0/2046136304 wait complete. 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.078+0000 7fd79347f740 1 -- start start 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.078+0000 7fd79347f740 1 Processor -- start 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd79347f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 0x55cc5394d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd79347f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5394a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd79347f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5394acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd78dcff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5394a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd78ed01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5394acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd78e500640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 0x55cc5394d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd78e500640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 0x55cc5394d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:42726/0 (socket says 172.21.15.98:42726) 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd78ed01640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5394acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41014/0 (socket says 172.21.15.98:41014) 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd78e500640 1 -- 172.21.15.98:0/3710129783 learned_addr learned my addr 172.21.15.98:0/3710129783 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd7881a9fd0 con 0x55cc5393c800 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd7881ab440 con 0x55cc53931620 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7881aa6a0 con 0x55cc53931f40 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.079+0000 7fd79347f740 10 client.0 _ll_register_callbacks cb 0x7fd78400ed40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd78dcff640 1 -- 172.21.15.98:0/3710129783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 msgr2=0x55cc5394d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd78dcff640 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 0x55cc5394d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd78dcff640 1 -- 172.21.15.98:0/3710129783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 msgr2=0x55cc5394acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd78dcff640 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5394acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd78dcff640 1 -- 172.21.15.98:0/3710129783 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc5394b4f0 con 0x55cc53931f40 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd78e500640 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 0x55cc5394d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd78ed01640 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5394acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:51:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd78dcff640 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5394a780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7780a52c0 tx=0x7fd778098130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=815cc0d7db9350 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd771ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd771ffb640 1 -- 172.21.15.98:0/3710129783 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7780abe60 con 0x55cc53931f40 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd780003e20 con 0x55cc53931f40 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd771ffb640 1 -- 172.21.15.98:0/3710129783 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fd77809a790 con 0x55cc53931f40 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd771ffb640 1 -- 172.21.15.98:0/3710129783 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fd7780b4040 con 0x55cc53931f40 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd79347f740 10 client.26314 fetch_fsmap learned FSMap version 51 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55cc539bb3c0 con 0x55cc53931f40 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd771ffb640 1 -- 172.21.15.98:0/3710129783 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd780003e20 con 0x55cc53931f40 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd771ffb640 1 -- 172.21.15.98:0/3710129783 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7fd778098b00 con 0x55cc53931f40 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd79347f740 10 client.26314 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd79347f740 10 client.26314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd79347f740 20 client.26314 populate_metadata read hostname 'smithi098' 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd79347f740 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc539bb5f0 0x55cc539bd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd79347f740 4 client.26314 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:51:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.080+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cc53939cf0 con 0x55cc539bb5f0 2024-04-25T17:51:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.081+0000 7fd78e500640 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc539bb5f0 0x55cc539bd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.081+0000 7fd771ffb640 10 client.26314 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.081+0000 7fd78e500640 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc539bb5f0 0x55cc539bd9e0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a3a7dc043429a2c server_cookie=959eed8f13833d5a in_seq=0 out_seq=0 2024-04-25T17:51:19.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.098+0000 7fd771ffb640 1 -- 172.21.15.98:0/3710129783 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cc53939cf0 con 0x55cc539bb5f0 2024-04-25T17:51:19.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.098+0000 7fd771ffb640 10 client.26314 handle_command_reply: tid=0 2024-04-25T17:51:19.097 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.098+0000 7fd79347f740 1 client.26314 shutdown 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 msgr2=0x55cc5394a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5394a780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7780a52c0 tx=0x7fd778098130 comp rx=0 tx=0).stop 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 shutdown_connections 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55cc539bb5f0 0x55cc539bd9e0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cc5393c800 0x55cc5394acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cc53931f40 0x55cc5394a780 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 1 --2- 172.21.15.98:0/3710129783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55cc53931620 0x55cc5394d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 >> 172.21.15.98:0/3710129783 conn(0x55cc5392db30 msgr2=0x55cc5392ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd771ffb640 0 client.26314 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 shutdown_connections 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 1 -- 172.21.15.98:0/3710129783 wait complete. 2024-04-25T17:51:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.099+0000 7fd79347f740 20 client.26314 trim_cache size 0 max 16384 2024-04-25T17:51:19.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:51:19.100 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:51:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:51:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:51:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 -- 172.21.15.98:0/4223782504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd7740a4110 msgr2=0x7fd7740a65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd7740a4110 0x7fd7740a65d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd78402eb60 tx=0x7fd7840041a0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 -- 172.21.15.98:0/4223782504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd788107280 msgr2=0x7fd788196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd788107280 0x7fd788196a00 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd77800c970 tx=0x7fd77800ce30 comp rx=0 tx=0).stop 2024-04-25T17:51:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 -- 172.21.15.98:0/4223782504 shutdown_connections 2024-04-25T17:51:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fd7740a4110 0x7fd7740a65d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd78819ad80 0x7fd78819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd788107280 0x7fd788196a00 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 --2- 172.21.15.98:0/4223782504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7881006c0 0x7fd7881964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 -- 172.21.15.98:0/4223782504 >> 172.21.15.98:0/4223782504 conn(0x7fd788077390 msgr2=0x7fd7880fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 -- 172.21.15.98:0/4223782504 shutdown_connections 2024-04-25T17:51:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:19.100+0000 7fd790764640 1 -- 172.21.15.98:0/4223782504 wait complete. 2024-04-25T17:51:20.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:20 smithi203 ceph-mon[32834]: pgmap v487: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 20 MiB/s wr, 839 op/s 2024-04-25T17:51:20.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:20 smithi098 ceph-mon[29192]: pgmap v487: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 20 MiB/s wr, 839 op/s 2024-04-25T17:51:20.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:20 smithi173 ceph-mon[32856]: pgmap v487: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 20 MiB/s wr, 839 op/s 2024-04-25T17:51:21.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 890.0 s, 2163.2 tps, lat 46.056 ms stddev 50.343 2024-04-25T17:51:22.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:22 smithi173 ceph-mon[32856]: pgmap v488: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 20 MiB/s wr, 839 op/s 2024-04-25T17:51:22.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:22 smithi098 ceph-mon[29192]: pgmap v488: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 20 MiB/s wr, 839 op/s 2024-04-25T17:51:22.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:22 smithi203 ceph-mon[32834]: pgmap v488: 97 pgs: 97 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 20 MiB/s wr, 839 op/s 2024-04-25T17:51:24.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:24 smithi173 ceph-mon[32856]: pgmap v489: 97 pgs: 97 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 19 MiB/s wr, 725 op/s 2024-04-25T17:51:24.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:24 smithi098 ceph-mon[29192]: pgmap v489: 97 pgs: 97 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 19 MiB/s wr, 725 op/s 2024-04-25T17:51:24.760 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:24 smithi203 ceph-mon[32834]: pgmap v489: 97 pgs: 97 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 19 MiB/s wr, 725 op/s 2024-04-25T17:51:26.513 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:26 smithi203 ceph-mon[32834]: pgmap v490: 97 pgs: 97 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 19 MiB/s wr, 725 op/s 2024-04-25T17:51:26.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:26 smithi173 ceph-mon[32856]: pgmap v490: 97 pgs: 97 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 19 MiB/s wr, 725 op/s 2024-04-25T17:51:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 895.0 s, 2339.6 tps, lat 42.947 ms stddev 50.849 2024-04-25T17:51:26.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:26 smithi098 ceph-mon[29192]: pgmap v490: 97 pgs: 97 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 19 MiB/s wr, 725 op/s 2024-04-25T17:51:28.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:28 smithi173 ceph-mon[32856]: pgmap v491: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 23 MiB/s wr, 919 op/s 2024-04-25T17:51:28.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:28 smithi098 ceph-mon[29192]: pgmap v491: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 23 MiB/s wr, 919 op/s 2024-04-25T17:51:28.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:28 smithi203 ceph-mon[32834]: pgmap v491: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 23 MiB/s wr, 919 op/s 2024-04-25T17:51:30.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:30 smithi173 ceph-mon[32856]: pgmap v492: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 14 MiB/s wr, 591 op/s 2024-04-25T17:51:30.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:30 smithi098 ceph-mon[29192]: pgmap v492: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 14 MiB/s wr, 591 op/s 2024-04-25T17:51:30.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:30 smithi203 ceph-mon[32834]: pgmap v492: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 14 MiB/s wr, 591 op/s 2024-04-25T17:51:31.593 INFO:teuthology.orchestra.run.smithi098.stderr:progress: 900.0 s, 2473.0 tps, lat 40.036 ms stddev 49.330 2024-04-25T17:51:31.680 INFO:teuthology.orchestra.run.smithi098.stdout:transaction type: 2024-04-25T17:51:31.680 INFO:teuthology.orchestra.run.smithi098.stdout:scaling factor: 32 2024-04-25T17:51:31.680 INFO:teuthology.orchestra.run.smithi098.stdout:query mode: simple 2024-04-25T17:51:31.680 INFO:teuthology.orchestra.run.smithi098.stdout:number of clients: 100 2024-04-25T17:51:31.680 INFO:teuthology.orchestra.run.smithi098.stdout:number of threads: 4 2024-04-25T17:51:31.680 INFO:teuthology.orchestra.run.smithi098.stdout:duration: 900 s 2024-04-25T17:51:31.681 INFO:teuthology.orchestra.run.smithi098.stdout:number of transactions actually processed: 1413991 2024-04-25T17:51:31.681 INFO:teuthology.orchestra.run.smithi098.stdout:latency average = 63.645 ms 2024-04-25T17:51:31.681 INFO:teuthology.orchestra.run.smithi098.stdout:latency stddev = 1127.461 ms 2024-04-25T17:51:31.681 INFO:teuthology.orchestra.run.smithi098.stdout:tps = 1570.948871 (including connections establishing) 2024-04-25T17:51:31.681 INFO:teuthology.orchestra.run.smithi098.stdout:tps = 1570.952862 (excluding connections establishing) 2024-04-25T17:51:31.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-04-25T17:51:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.263+0000 7fdffb920640 1 -- 172.21.15.98:0/1505888274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdff4110740 msgr2=0x7fdff4108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.263+0000 7fdffb920640 1 --2- 172.21.15.98:0/1505888274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdff4110740 0x7fdff4108f50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009900 tx=0x7fdfe802a4d0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.265 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.266+0000 7fdffb920640 1 -- 172.21.15.98:0/1505888274 shutdown_connections 2024-04-25T17:51:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.266+0000 7fdffb920640 1 --2- 172.21.15.98:0/1505888274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff411b730 0x7fdff411db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.266+0000 7fdffb920640 1 --2- 172.21.15.98:0/1505888274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff4110b20 0x7fdff4109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.266+0000 7fdffb920640 1 --2- 172.21.15.98:0/1505888274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdff4110740 0x7fdff4108f50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.266+0000 7fdffb920640 1 -- 172.21.15.98:0/1505888274 >> 172.21.15.98:0/1505888274 conn(0x7fdff406d020 msgr2=0x7fdff406d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:32.271 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.272+0000 7fdffb920640 1 -- 172.21.15.98:0/1505888274 shutdown_connections 2024-04-25T17:51:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.273+0000 7fdffb920640 1 -- 172.21.15.98:0/1505888274 wait complete. 2024-04-25T17:51:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.274+0000 7fdffb920640 1 Processor -- start 2024-04-25T17:51:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.278+0000 7fdffb920640 1 -- start start 2024-04-25T17:51:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.278+0000 7fdffb920640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff4110740 0x7fdff411ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.278+0000 7fdffb920640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdff411b730 0x7fdff4114de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.278+0000 7fdffb920640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff4115320 0x7fdff41157d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.278+0000 7fdffb920640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdff4111810 con 0x7fdff4115320 2024-04-25T17:51:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.278+0000 7fdffb920640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdff4111690 con 0x7fdff4110740 2024-04-25T17:51:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.278+0000 7fdffb920640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdff4111990 con 0x7fdff411b730 2024-04-25T17:51:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.282+0000 7fdff9ebd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff4115320 0x7fdff41157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.282+0000 7fdff9ebd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff4115320 0x7fdff41157d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35404/0 (socket says 172.21.15.98:35404) 2024-04-25T17:51:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.282+0000 7fdff9ebd640 1 -- 172.21.15.98:0/1476321717 learned_addr learned my addr 172.21.15.98:0/1476321717 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.282+0000 7fdff96bc640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff4110740 0x7fdff411ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.284+0000 7fdff9ebd640 1 -- 172.21.15.98:0/1476321717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdff411b730 msgr2=0x7fdff4114de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.285+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdff411b730 0x7fdff4114de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.285+0000 7fdff9ebd640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdff411b730 0x7fdff4114de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.286+0000 7fdff9ebd640 1 -- 172.21.15.98:0/1476321717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff4110740 msgr2=0x7fdff411ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.286+0000 7fdff9ebd640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff4110740 0x7fdff411ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.286+0000 7fdff9ebd640 1 -- 172.21.15.98:0/1476321717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff41bc5a0 con 0x7fdff4115320 2024-04-25T17:51:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.286+0000 7fdff9ebd640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff4115320 0x7fdff41157d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fdff000b3f0 tx=0x7fdff000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:32.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.288+0000 7fdfe67fc640 1 -- 172.21.15.98:0/1476321717 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0015020 con 0x7fdff4115320 2024-04-25T17:51:32.288 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.288+0000 7fdfe67fc640 1 -- 172.21.15.98:0/1476321717 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdff0011ce0 con 0x7fdff4115320 2024-04-25T17:51:32.289 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.288+0000 7fdfe67fc640 1 -- 172.21.15.98:0/1476321717 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0010dd0 con 0x7fdff4115320 2024-04-25T17:51:32.289 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.289+0000 7fdffb920640 1 -- 172.21.15.98:0/1476321717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff41bc830 con 0x7fdff4115320 2024-04-25T17:51:32.289 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.289+0000 7fdffb920640 1 -- 172.21.15.98:0/1476321717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff41bcd00 con 0x7fdff4115320 2024-04-25T17:51:32.290 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.290+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdff411b730 0x7fdff4114de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:51:32.290 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.290+0000 7fdff96bc640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff4110740 0x7fdff411ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T17:51:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.293+0000 7fdffe63a740 1 Processor -- start 2024-04-25T17:51:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.296+0000 7fdffe63a740 1 -- start start 2024-04-25T17:51:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.296+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c415ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.296+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 0x5607c41672c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.296+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c4169bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.296+0000 7fdffe63a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5607c40c1970 con 0x5607c415cf40 2024-04-25T17:51:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.296+0000 7fdffe63a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5607c405e4a0 con 0x5607c4167800 2024-04-25T17:51:32.295 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.296+0000 7fdffe63a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5607c40bd930 con 0x5607c415c620 2024-04-25T17:51:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.297+0000 7fdfe67fc640 1 -- 172.21.15.98:0/1476321717 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdff0018020 con 0x7fdff4115320 2024-04-25T17:51:32.296 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.297+0000 7fdfe67fc640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdfe00a3ff0 0x7fdfe00a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.297 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.298+0000 7fdfe67fc640 1 -- 172.21.15.98:0/1476321717 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fdff00cfe10 con 0x7fdff4115320 2024-04-25T17:51:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.300+0000 7fdff9ebd640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdfe00a3ff0 0x7fdfe00a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.300+0000 7fdff96bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 0x5607c41672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.300+0000 7fdff96bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 0x5607c41672c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35406/0 (socket says 172.21.15.98:35406) 2024-04-25T17:51:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.300+0000 7fdff96bc640 1 -- 172.21.15.98:0/3101877442 learned_addr learned my addr 172.21.15.98:0/3101877442 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.301+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/3101877442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c415ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.301+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/3101877442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c4169bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.303+0000 7fdff96bc640 1 -- 172.21.15.98:0/3101877442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 msgr2=0x5607c415ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.303+0000 7fdff96bc640 1 --2- 172.21.15.98:0/3101877442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c415ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.302 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.303+0000 7fdff96bc640 1 -- 172.21.15.98:0/3101877442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 msgr2=0x5607c4169bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:32.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.303+0000 7fdff96bc640 1 --2- 172.21.15.98:0/3101877442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c4169bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.303 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.303+0000 7fdff96bc640 1 -- 172.21.15.98:0/3101877442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607c3dd4210 con 0x5607c415cf40 2024-04-25T17:51:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.306+0000 7fdff9ebd640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdfe00a3ff0 0x7fdfe00a64b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdff000b3c0 tx=0x7fdff0009200 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:32.306 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.307+0000 7fdff96bc640 1 --2- 172.21.15.98:0/3101877442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 0x5607c41672c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009900 tx=0x7fdfe80036a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa933e48a7ac0cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.309+0000 7fdfdf7fe640 1 -- 172.21.15.98:0/3101877442 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8035030 con 0x5607c415cf40 2024-04-25T17:51:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.309+0000 7fdfdf7fe640 1 -- 172.21.15.98:0/3101877442 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfe8040070 con 0x5607c415cf40 2024-04-25T17:51:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.310+0000 7fdffe63a740 1 -- 172.21.15.98:0/3101877442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 msgr2=0x5607c41672c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:32.309 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.310+0000 7fdffe63a740 1 --2- 172.21.15.98:0/3101877442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 0x5607c41672c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009900 tx=0x7fdfe80036a0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.314+0000 7fdffe63a740 1 -- 172.21.15.98:0/3101877442 shutdown_connections 2024-04-25T17:51:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.314+0000 7fdffe63a740 1 --2- 172.21.15.98:0/3101877442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c4169bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.314+0000 7fdffe63a740 1 --2- 172.21.15.98:0/3101877442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 0x5607c41672c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.314+0000 7fdffe63a740 1 --2- 172.21.15.98:0/3101877442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c415ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.313 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.314+0000 7fdffe63a740 1 -- 172.21.15.98:0/3101877442 >> 172.21.15.98:0/3101877442 conn(0x5607c4158b30 msgr2=0x5607c4158f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:32.314 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.315+0000 7fdffe63a740 1 -- 172.21.15.98:0/3101877442 shutdown_connections 2024-04-25T17:51:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.315+0000 7fdffe63a740 1 -- 172.21.15.98:0/3101877442 wait complete. 2024-04-25T17:51:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.315+0000 7fdffe63a740 1 -- start start 2024-04-25T17:51:32.315 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.315+0000 7fdffe63a740 1 Processor -- start 2024-04-25T17:51:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.316+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415c620 0x5607c41670b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.316+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415cf40 0x5607c41643a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.317 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.316+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c41648e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.316+0000 7fdffe63a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5607c416a3d0 con 0x5607c415c620 2024-04-25T17:51:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.316+0000 7fdffe63a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5607c405c0b0 con 0x5607c4167800 2024-04-25T17:51:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.316+0000 7fdffe63a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5607c416a550 con 0x5607c415cf40 2024-04-25T17:51:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.316+0000 7fdffe63a740 10 client.0 _ll_register_callbacks cb 0x5607c4049d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:51:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.318+0000 7fdff8ebb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c41648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.318+0000 7fdff8ebb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415c620 0x5607c41670b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.318+0000 7fdff8ebb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415c620 0x5607c41670b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35422/0 (socket says 172.21.15.98:35422) 2024-04-25T17:51:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.318+0000 7fdff8ebb640 1 -- 172.21.15.98:0/1466040355 learned_addr learned my addr 172.21.15.98:0/1466040355 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.318+0000 7fdff96bc640 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415cf40 0x5607c41643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.321+0000 7fdff8ebb640 1 -- 172.21.15.98:0/1466040355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415cf40 msgr2=0x5607c41643a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:32.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.321+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415cf40 0x5607c41643a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.321+0000 7fdff8ebb640 1 -- 172.21.15.98:0/1466040355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 msgr2=0x5607c41648e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:32.322 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.321+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c41648e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.321+0000 7fdff8ebb640 1 -- 172.21.15.98:0/1466040355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607c3ea0620 con 0x5607c415c620 2024-04-25T17:51:32.323 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.322+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415c620 0x5607c41670b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fdfec014f80 tx=0x7fdfec004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1c0d107acb96c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:32.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.324+0000 7fdfdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:51:32.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.325+0000 7fdfdcff9640 1 -- 172.21.15.98:0/1466040355 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec0125e0 con 0x5607c415c620 2024-04-25T17:51:32.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.325+0000 7fdffe63a740 1 -- 172.21.15.98:0/1466040355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfe8040070 con 0x5607c415c620 2024-04-25T17:51:32.324 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.325+0000 7fdfdcff9640 1 -- 172.21.15.98:0/1466040355 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fdfec012780 con 0x5607c415c620 2024-04-25T17:51:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.328+0000 7fdfdcff9640 1 -- 172.21.15.98:0/1466040355 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fdfec024050 con 0x5607c415c620 2024-04-25T17:51:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.329+0000 7fdffe63a740 10 client.16893 fetch_fsmap learned FSMap version 51 2024-04-25T17:51:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.329+0000 7fdffe63a740 1 -- 172.21.15.98:0/1466040355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x5607c4176380 con 0x5607c415c620 2024-04-25T17:51:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.330+0000 7fdfdcff9640 1 -- 172.21.15.98:0/1466040355 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfe8040070 con 0x5607c415c620 2024-04-25T17:51:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.330+0000 7fdfdcff9640 1 -- 172.21.15.98:0/1466040355 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7fdfec01f890 con 0x5607c415c620 2024-04-25T17:51:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.330+0000 7fdffe63a740 10 client.16893 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:51:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.330+0000 7fdffe63a740 10 client.16893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:51:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.330+0000 7fdffe63a740 20 client.16893 populate_metadata read hostname 'smithi098' 2024-04-25T17:51:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.330+0000 7fdffe63a740 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5607c4177c20 0x5607c417a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.331+0000 7fdffe63a740 4 client.16893 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:51:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.331+0000 7fdffe63a740 1 -- 172.21.15.98:0/1466040355 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607c40cf390 con 0x5607c4177c20 2024-04-25T17:51:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.332+0000 7fdff96bc640 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5607c4177c20 0x5607c417a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.344+0000 7fdff96bc640 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5607c4177c20 0x5607c417a0c0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4ec6a2ea6b039e5 server_cookie=d01d34999a5f26d5 in_seq=0 out_seq=0 2024-04-25T17:51:32.345 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.344+0000 7fdfdcff9640 10 client.16893 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.346+0000 7fdfdcff9640 1 -- 172.21.15.98:0/1466040355 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5607c40cf390 con 0x5607c4177c20 2024-04-25T17:51:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.346+0000 7fdfdcff9640 10 client.16893 handle_command_reply: tid=0 2024-04-25T17:51:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.346+0000 7fdffe63a740 1 client.16893 shutdown 2024-04-25T17:51:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.351+0000 7fdffe63a740 1 -- 172.21.15.98:0/1466040355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415c620 msgr2=0x5607c41670b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:32.351 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.351+0000 7fdffe63a740 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415c620 0x5607c41670b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fdfec014f80 tx=0x7fdfec004a60 comp rx=0 tx=0).stop 2024-04-25T17:51:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.354+0000 7fdffe63a740 1 -- 172.21.15.98:0/1466040355 shutdown_connections 2024-04-25T17:51:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.354+0000 7fdffe63a740 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5607c4177c20 0x5607c417a0c0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.354+0000 7fdffe63a740 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c41648e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.354+0000 7fdffe63a740 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415cf40 0x5607c41643a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.354+0000 7fdffe63a740 1 --2- 172.21.15.98:0/1466040355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415c620 0x5607c41670b0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.354+0000 7fdffe63a740 1 -- 172.21.15.98:0/1466040355 >> 172.21.15.98:0/1466040355 conn(0x5607c4158b30 msgr2=0x5607c415e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.354+0000 7fdfdcff9640 0 client.16893 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:32.358 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.357+0000 7fdffe63a740 1 -- 172.21.15.98:0/1466040355 shutdown_connections 2024-04-25T17:51:32.360 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.359+0000 7fdffe63a740 1 -- 172.21.15.98:0/1466040355 wait complete. 2024-04-25T17:51:32.360 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.361+0000 7fdffe63a740 20 client.16893 trim_cache size 0 max 16384 2024-04-25T17:51:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.394+0000 7fdffe63a740 1 Processor -- start 2024-04-25T17:51:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.394+0000 7fdffe63a740 1 -- start start 2024-04-25T17:51:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.395+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c41732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.395+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c415cf40 0x5607c41662e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.395+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c4167800 0x5607c417f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.395+0000 7fdffe63a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdff41c7c80 con 0x5607c4167800 2024-04-25T17:51:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.395+0000 7fdffe63a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdff41c5550 con 0x5607c415cf40 2024-04-25T17:51:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.395+0000 7fdffe63a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdff41c33c0 con 0x5607c415c620 2024-04-25T17:51:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.398+0000 7fdff96bc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c415cf40 0x5607c41662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.398+0000 7fdff96bc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c415cf40 0x5607c41662e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:57370/0 (socket says 172.21.15.98:57370) 2024-04-25T17:51:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.398+0000 7fdff96bc640 1 -- 172.21.15.98:0/552040657 learned_addr learned my addr 172.21.15.98:0/552040657 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.398+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/552040657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c41732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.398+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/552040657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c4167800 0x5607c417f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.399 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.399+0000 7fdff8ebb640 1 -- 172.21.15.98:0/552040657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c415cf40 msgr2=0x5607c41662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:32.400 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.399+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/552040657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c415cf40 0x5607c41662e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.400 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.399+0000 7fdff8ebb640 1 -- 172.21.15.98:0/552040657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c4167800 msgr2=0x5607c417f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:32.400 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.399+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/552040657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c4167800 0x5607c417f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.400 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.399+0000 7fdff8ebb640 1 -- 172.21.15.98:0/552040657 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607c41542e0 con 0x5607c415c620 2024-04-25T17:51:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.400+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/552040657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c41732e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdfec013040 tx=0x7fdfec004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71f8cad8c1b96ae1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.402+0000 7fdfdeffd640 1 -- 172.21.15.98:0/552040657 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec0084f0 con 0x5607c415c620 2024-04-25T17:51:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.402+0000 7fdfdeffd640 1 -- 172.21.15.98:0/552040657 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfec014590 con 0x5607c415c620 2024-04-25T17:51:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.402+0000 7fdfdeffd640 1 -- 172.21.15.98:0/552040657 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec004280 con 0x5607c415c620 2024-04-25T17:51:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.405+0000 7fdffe63a740 1 -- 172.21.15.98:0/552040657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 msgr2=0x5607c41732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.405+0000 7fdffe63a740 1 --2- 172.21.15.98:0/552040657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c41732e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdfec013040 tx=0x7fdfec004ad0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.405+0000 7fdffe63a740 1 -- 172.21.15.98:0/552040657 shutdown_connections 2024-04-25T17:51:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.405+0000 7fdffe63a740 1 --2- 172.21.15.98:0/552040657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c4167800 0x5607c417f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.405+0000 7fdffe63a740 1 --2- 172.21.15.98:0/552040657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c415cf40 0x5607c41662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.405+0000 7fdffe63a740 1 --2- 172.21.15.98:0/552040657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c41732e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.405+0000 7fdffe63a740 1 -- 172.21.15.98:0/552040657 >> 172.21.15.98:0/552040657 conn(0x5607c4158b30 msgr2=0x5607c4165bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.409+0000 7fdffe63a740 1 -- 172.21.15.98:0/552040657 shutdown_connections 2024-04-25T17:51:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.409+0000 7fdffe63a740 1 -- 172.21.15.98:0/552040657 wait complete. 2024-04-25T17:51:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.410+0000 7fdffe63a740 1 -- start start 2024-04-25T17:51:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.410+0000 7fdffe63a740 1 Processor -- start 2024-04-25T17:51:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.416+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c416f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.416+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 0x5607c416fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.416+0000 7fdffe63a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c4160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.416+0000 7fdffe63a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdff41c3a10 con 0x5607c415cf40 2024-04-25T17:51:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.416+0000 7fdffe63a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdff41c4e80 con 0x5607c4167800 2024-04-25T17:51:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.416+0000 7fdffe63a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdff41c40e0 con 0x5607c415c620 2024-04-25T17:51:32.417 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.416+0000 7fdffe63a740 10 client.0 _ll_register_callbacks cb 0x7fdfec014bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:51:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.418+0000 7fdff8ebb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c416f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.419+0000 7fdff96bc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 0x5607c416fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.419+0000 7fdff8ebb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c416f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40980/0 (socket says 172.21.15.98:40980) 2024-04-25T17:51:32.418 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.419+0000 7fdff8ebb640 1 -- 172.21.15.98:0/4168672935 learned_addr learned my addr 172.21.15.98:0/4168672935 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.419+0000 7fdff8ebb640 1 -- 172.21.15.98:0/4168672935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 msgr2=0x5607c4160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.419+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c4160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.419+0000 7fdff8ebb640 1 -- 172.21.15.98:0/4168672935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 msgr2=0x5607c416fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.419+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 0x5607c416fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.419 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.419+0000 7fdff8ebb640 1 -- 172.21.15.98:0/4168672935 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607c4160b50 con 0x5607c415c620 2024-04-25T17:51:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.424+0000 7fdff8ebb640 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c416f7f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdfec008330 tx=0x7fdfec014c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4336ed51095257f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.424+0000 7fdfdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T17:51:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.428+0000 7fdfdcff9640 1 -- 172.21.15.98:0/4168672935 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec03b020 con 0x5607c415c620 2024-04-25T17:51:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.428+0000 7fdffe63a740 1 -- 172.21.15.98:0/4168672935 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfec014590 con 0x5607c415c620 2024-04-25T17:51:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.428+0000 7fdfdcff9640 1 -- 172.21.15.98:0/4168672935 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fdfec008d40 con 0x5607c415c620 2024-04-25T17:51:32.430 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.430+0000 7fdfdcff9640 1 -- 172.21.15.98:0/4168672935 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fdfec01fa60 con 0x5607c415c620 2024-04-25T17:51:32.432 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.433+0000 7fdffe63a740 10 client.26326 fetch_fsmap learned FSMap version 51 2024-04-25T17:51:32.432 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.433+0000 7fdffe63a740 1 -- 172.21.15.98:0/4168672935 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x5607c41dda70 con 0x5607c415c620 2024-04-25T17:51:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.436+0000 7fdfdcff9640 1 -- 172.21.15.98:0/4168672935 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfec014590 con 0x5607c415c620 2024-04-25T17:51:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.436+0000 7fdfdcff9640 1 -- 172.21.15.98:0/4168672935 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 51) ==== 8342+0+0 (secure 0 0 0) 0x7fdfec036070 con 0x5607c415c620 2024-04-25T17:51:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.437+0000 7fdffe63a740 10 client.26326 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T17:51:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.437+0000 7fdffe63a740 10 client.26326 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:51:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.437+0000 7fdffe63a740 20 client.26326 populate_metadata read hostname 'smithi098' 2024-04-25T17:51:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.437+0000 7fdffe63a740 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5607c41de0b0 0x5607c41e04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.437+0000 7fdffe63a740 4 client.26326 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:51:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.437+0000 7fdffe63a740 1 -- 172.21.15.98:0/4168672935 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5607c40c9200 con 0x5607c41de0b0 2024-04-25T17:51:32.438 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.439+0000 7fdff96bc640 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5607c41de0b0 0x5607c41e04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:32.444 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.445+0000 7fdff96bc640 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5607c41de0b0 0x5607c41e04d0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8639abc4cf938986 server_cookie=32aa229ab4e3c76d in_seq=0 out_seq=0 2024-04-25T17:51:32.445 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.445+0000 7fdfdcff9640 10 client.26326 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:32.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:32 smithi173 ceph-mon[32856]: pgmap v493: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 14 MiB/s wr, 591 op/s 2024-04-25T17:51:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdfdcff9640 1 -- 172.21.15.98:0/4168672935 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5607c40c9200 con 0x5607c41de0b0 2024-04-25T17:51:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdfdcff9640 10 client.26326 handle_command_reply: tid=0 2024-04-25T17:51:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 client.26326 shutdown 2024-04-25T17:51:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 -- 172.21.15.98:0/4168672935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 msgr2=0x5607c416f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c416f7f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdfec008330 tx=0x7fdfec014c70 comp rx=0 tx=0).stop 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 -- 172.21.15.98:0/4168672935 shutdown_connections 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x5607c41de0b0 0x5607c41e04d0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5607c4167800 0x5607c4160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5607c415cf40 0x5607c416fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 --2- 172.21.15.98:0/4168672935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5607c415c620 0x5607c416f7f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 -- 172.21.15.98:0/4168672935 >> 172.21.15.98:0/4168672935 conn(0x5607c4158b30 msgr2=0x5607c4157420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdfdcff9640 0 client.26326 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 -- 172.21.15.98:0/4168672935 shutdown_connections 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.650+0000 7fdffe63a740 1 -- 172.21.15.98:0/4168672935 wait complete. 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffe63a740 20 client.26326 trim_cache size 0 max 16384 2024-04-25T17:51:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:51:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:51:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:51:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:51:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffb920640 1 -- 172.21.15.98:0/1476321717 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdfe00a3ff0 msgr2=0x7fdfe00a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffb920640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdfe00a3ff0 0x7fdfe00a64b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdff000b3c0 tx=0x7fdff0009200 comp rx=0 tx=0).stop 2024-04-25T17:51:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffb920640 1 -- 172.21.15.98:0/1476321717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff4115320 msgr2=0x7fdff41157d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffb920640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff4115320 0x7fdff41157d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fdff000b3f0 tx=0x7fdff000b8b0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffb920640 1 -- 172.21.15.98:0/1476321717 shutdown_connections 2024-04-25T17:51:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffb920640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdfe00a3ff0 0x7fdfe00a64b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffb920640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdff4115320 0x7fdff41157d0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffb920640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdff411b730 0x7fdff4114de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffb920640 1 --2- 172.21.15.98:0/1476321717 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdff4110740 0x7fdff411ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.651+0000 7fdffb920640 1 -- 172.21.15.98:0/1476321717 >> 172.21.15.98:0/1476321717 conn(0x7fdff406d020 msgr2=0x7fdff411bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.653+0000 7fdffb920640 1 -- 172.21.15.98:0/1476321717 shutdown_connections 2024-04-25T17:51:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:32.654+0000 7fdffb920640 1 -- 172.21.15.98:0/1476321717 wait complete. 2024-04-25T17:51:32.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:32 smithi098 ceph-mon[29192]: pgmap v493: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 14 MiB/s wr, 591 op/s 2024-04-25T17:51:32.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:32 smithi203 ceph-mon[32834]: pgmap v493: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 14 MiB/s wr, 591 op/s 2024-04-25T17:51:34.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:34 smithi173 ceph-mon[32856]: pgmap v494: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 20 MiB/s wr, 878 op/s 2024-04-25T17:51:34.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:34 smithi098 ceph-mon[29192]: pgmap v494: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 20 MiB/s wr, 878 op/s 2024-04-25T17:51:34.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:34 smithi203 ceph-mon[32834]: pgmap v494: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 20 MiB/s wr, 878 op/s 2024-04-25T17:51:36.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:36 smithi173 ceph-mon[32856]: pgmap v495: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 12 MiB/s wr, 554 op/s 2024-04-25T17:51:36.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:36 smithi098 ceph-mon[29192]: pgmap v495: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 12 MiB/s wr, 554 op/s 2024-04-25T17:51:36.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:36 smithi203 ceph-mon[32834]: pgmap v495: 97 pgs: 97 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 12 MiB/s wr, 554 op/s 2024-04-25T17:51:38.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:38 smithi173 ceph-mon[32856]: pgmap v496: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 30 MiB/s wr, 1.94k op/s 2024-04-25T17:51:38.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:38 smithi098 ceph-mon[29192]: pgmap v496: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 30 MiB/s wr, 1.94k op/s 2024-04-25T17:51:38.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:38 smithi203 ceph-mon[32834]: pgmap v496: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 30 MiB/s wr, 1.94k op/s 2024-04-25T17:51:39.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/: 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 . 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 .. 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 data 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data: 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:total 36 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 . 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 .. 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 17:36 base 2024-04-25T17:51:39.669 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Apr 25 17:36 current_logfiles 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 25 17:36 global 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 log 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_commit_ts 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_dynshmem 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 25 17:36 pg_hba.conf 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 25 17:36 pg_ident.conf 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:51 pg_logical 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 pg_multixact 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_notify 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_replslot 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_serial 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_snapshots 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:51 pg_stat 2024-04-25T17:51:39.670 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:51 pg_stat_tmp 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:51 pg_subtrans 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_tblspc 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pg_twophase 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 PG_VERSION 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 25 17:51 pg_wal 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:51 pg_xact 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 25 17:36 postgresql.auto.conf 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 25 17:36 postgresql.conf 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Apr 25 17:36 postmaster.opts 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/base: 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 17:36 . 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.671 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 17:36 1 2024-04-25T17:51:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 17:36 13493 2024-04-25T17:51:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 25 17:41 13494 2024-04-25T17:51:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 pgsql_tmp 2024-04-25T17:51:39.672 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-25T17:51:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:total 7777 2024-04-25T17:51:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 17:36 . 2024-04-25T17:51:39.672 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 17:36 .. 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 112 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 113 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 1247 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1247_fsm 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1247_vm 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 17:36 1249 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1249_fsm 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1249_vm 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 17:36 1255 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1255_fsm 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1255_vm 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 17:36 1259 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1259_fsm 2024-04-25T17:51:39.673 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1259_vm 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 17:36 13344 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13344_fsm 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13344_vm 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13346 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13348 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13349_fsm 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349_vm 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13351 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13353 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354 2024-04-25T17:51:39.674 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13354_fsm 2024-04-25T17:51:39.675 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354_vm 2024-04-25T17:51:39.675 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13356 2024-04-25T17:51:39.675 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13358 2024-04-25T17:51:39.675 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359 2024-04-25T17:51:39.675 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13359_fsm 2024-04-25T17:51:39.675 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359_vm 2024-04-25T17:51:39.675 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13361 2024-04-25T17:51:39.675 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13363 2024-04-25T17:51:39.675 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1417 2024-04-25T17:51:39.675 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1418 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 174 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 175 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2187 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2224 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2328 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2336 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2337 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2579 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2600 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2600_fsm 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2600_vm 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2601_fsm 2024-04-25T17:51:39.676 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601_vm 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2602 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2602_fsm 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2602_vm 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2603 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2603_fsm 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2603_vm 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2604 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2605 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2605_fsm 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2605_vm 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2606_fsm 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606_vm 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2607 2024-04-25T17:51:39.677 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2607_fsm 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2607_vm 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 17:36 2608 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2608_fsm 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2608_vm 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 17:36 2609 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2609_fsm 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2609_vm 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2610 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2610_fsm 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2610_vm 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2611 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2612_fsm 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612_vm 2024-04-25T17:51:39.678 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2613 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2615_fsm 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615_vm 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2616 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2616_fsm 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2616_vm 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 17:36 2617 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2617_fsm 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2617_vm 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 17:36 2618 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2618_fsm 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2618_vm 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 17:36 2619 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2619_fsm 2024-04-25T17:51:39.679 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2619_vm 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2620 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2650 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2651 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2652 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2653 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2654 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2655 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2656 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2657 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 17:36 2658 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 2659 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2660 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2661 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2662 2024-04-25T17:51:39.680 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2663 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2664 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2665 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2666 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2667 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2668 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2669 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2670 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 17:36 2673 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 17:36 2674 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 17:36 2675 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2678 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2679 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2680 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2681 2024-04-25T17:51:39.681 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2682 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2683 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2684 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2685 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2686 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2687 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2688 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2689 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 17:36 2690 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 17:36 2691 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2692 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2693 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2696 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2699 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2701 2024-04-25T17:51:39.682 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2702 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2703 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2704 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2753 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2753_fsm 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2753_vm 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2754 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2755 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2756 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2757 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2830 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2831 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2832 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2833 2024-04-25T17:51:39.683 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2834 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2835 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2836 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2837 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 17:36 2838 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2838_fsm 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2838_vm 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2839 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840_fsm 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2840_vm 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2841 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2995 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2996 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079 2024-04-25T17:51:39.684 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3079_fsm 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079_vm 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3080 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3081 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3085 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3118 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3119 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 17:36 3164 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3256 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3257 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3258 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3350 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3351 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3379 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3380 2024-04-25T17:51:39.685 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3381 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3394 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3394_fsm 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3394_vm 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3395 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3429 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3430 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3431 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3433 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3439 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3440 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3455 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 17:36 3456 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3456_fsm 2024-04-25T17:51:39.686 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3456_vm 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3466 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3467 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3468 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3501 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3502 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3503 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3534 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3541_fsm 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541_vm 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3542 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3574 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3575 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3576 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3596 2024-04-25T17:51:39.687 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3597 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3598 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3599 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3600_fsm 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600_vm 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3601_fsm 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601_vm 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3602_fsm 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602_vm 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603_fsm 2024-04-25T17:51:39.688 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3603_vm 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3604 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3605 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3606 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3607 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3608 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 3609 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3712 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3764_fsm 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764_vm 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3766 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3767 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3997 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4143 2024-04-25T17:51:39.689 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4144 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4145 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4146 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4147 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4148 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4149 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4150 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4151 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4152 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4153 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4154 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4155 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4156 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4157 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4158 2024-04-25T17:51:39.690 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4159 2024-04-25T17:51:39.691 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4160 2024-04-25T17:51:39.691 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4161 2024-04-25T17:51:39.691 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4162 2024-04-25T17:51:39.694 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4163 2024-04-25T17:51:39.694 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4164 2024-04-25T17:51:39.694 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4165 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4166 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4167 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4168 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4169 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4170 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4171 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4172 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4173 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4174 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 5002 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 548 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 549 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6102 2024-04-25T17:51:39.695 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6104 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6106 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6110 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6111 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6112 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6113 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6117 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 826 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 827 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 828 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 17:36 pg_filenode.map 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 PG_VERSION 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:total 7777 2024-04-25T17:51:39.696 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 17:36 . 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 17:36 .. 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 112 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 113 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 1247 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1247_fsm 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1247_vm 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 17:36 1249 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1249_fsm 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1249_vm 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 17:36 1255 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1255_fsm 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1255_vm 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 17:36 1259 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1259_fsm 2024-04-25T17:51:39.697 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1259_vm 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 17:36 13344 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13344_fsm 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13344_vm 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13346 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13348 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13349_fsm 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349_vm 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13351 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13353 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13354_fsm 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354_vm 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13356 2024-04-25T17:51:39.698 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13358 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13359_fsm 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359_vm 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13361 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13363 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1417 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1418 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 174 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 175 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2187 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2224 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2328 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2336 2024-04-25T17:51:39.699 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2337 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2579 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2600 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2600_fsm 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2600_vm 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2601_fsm 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601_vm 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2602 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2602_fsm 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2602_vm 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2603 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2603_fsm 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2603_vm 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2604 2024-04-25T17:51:39.700 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2605 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2605_fsm 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2605_vm 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2606_fsm 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606_vm 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2607 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2607_fsm 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2607_vm 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 17:36 2608 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2608_fsm 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2608_vm 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 17:36 2609 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2609_fsm 2024-04-25T17:51:39.701 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2609_vm 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2610 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2610_fsm 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2610_vm 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2611 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2612_fsm 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612_vm 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2613 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2615_fsm 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615_vm 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2616 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2616_fsm 2024-04-25T17:51:39.702 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2616_vm 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 17:36 2617 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2617_fsm 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2617_vm 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 17:36 2618 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2618_fsm 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2618_vm 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 17:36 2619 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2619_fsm 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2619_vm 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2620 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2650 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2651 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2652 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2653 2024-04-25T17:51:39.703 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2654 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2655 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2656 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2657 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 17:36 2658 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 2659 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2660 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2661 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2662 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2663 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2664 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2665 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2666 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2667 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2668 2024-04-25T17:51:39.704 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2669 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2670 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 17:36 2673 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 17:36 2674 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 17:36 2675 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2678 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2679 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2680 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2681 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2682 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2683 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2684 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2685 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2686 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2687 2024-04-25T17:51:39.705 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2688 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2689 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 17:36 2690 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 17:36 2691 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2692 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2693 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2696 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2699 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2701 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2702 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2703 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2704 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2753 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2753_fsm 2024-04-25T17:51:39.706 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2753_vm 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2754 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2755 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2756 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2757 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2830 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2831 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2832 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2833 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2834 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2835 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2836 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2837 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 17:36 2838 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2838_fsm 2024-04-25T17:51:39.707 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2838_vm 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2839 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840_fsm 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2840_vm 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2841 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2995 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2996 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3079_fsm 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079_vm 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3080 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3081 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3085 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3118 2024-04-25T17:51:39.708 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3119 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 17:36 3164 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3256 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3257 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3258 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3350 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3351 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3379 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3380 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3381 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3394 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3394_fsm 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3394_vm 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3395 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3429 2024-04-25T17:51:39.709 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3430 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3431 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3433 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3439 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3440 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3455 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 17:36 3456 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3456_fsm 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3456_vm 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3466 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3467 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3468 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3501 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3502 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3503 2024-04-25T17:51:39.710 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3534 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3541_fsm 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541_vm 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3542 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3574 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3575 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3576 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3596 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3597 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3598 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3599 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3600_fsm 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600_vm 2024-04-25T17:51:39.711 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3601_fsm 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601_vm 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3602_fsm 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602_vm 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603_fsm 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3603_vm 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3604 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3605 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3606 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3607 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3608 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 3609 2024-04-25T17:51:39.712 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3712 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3764_fsm 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764_vm 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3766 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3767 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3997 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4143 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4144 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4145 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4146 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4147 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4148 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4149 2024-04-25T17:51:39.713 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4150 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4151 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4152 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4153 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4154 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4155 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4156 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4157 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4158 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4159 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4160 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4161 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4162 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4163 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4164 2024-04-25T17:51:39.714 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4165 2024-04-25T17:51:39.715 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4166 2024-04-25T17:51:39.715 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4167 2024-04-25T17:51:39.715 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4168 2024-04-25T17:51:39.715 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4169 2024-04-25T17:51:39.715 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4170 2024-04-25T17:51:39.715 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4171 2024-04-25T17:51:39.715 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4172 2024-04-25T17:51:39.771 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4173 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4174 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 5002 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 548 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 549 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6102 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6104 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6106 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6110 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6111 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6112 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6113 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6117 2024-04-25T17:51:39.772 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 826 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 827 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 828 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 17:36 pg_filenode.map 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 PG_VERSION 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:total 585006 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 25 17:41 . 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 17:36 .. 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 112 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 113 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 1247 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1247_fsm 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1247_vm 2024-04-25T17:51:39.773 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 17:36 1249 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1249_fsm 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1249_vm 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 17:36 1255 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1255_fsm 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1255_vm 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 17:51 1259 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1259_fsm 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1259_vm 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 17:36 13344 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13344_fsm 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13344_vm 2024-04-25T17:51:39.774 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13346 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13348 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13349_fsm 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13349_vm 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13351 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13353 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13354_fsm 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13354_vm 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13356 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13358 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 13359_fsm 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13359_vm 2024-04-25T17:51:39.775 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 13361 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 13363 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1417 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 1418 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 442286080 Apr 25 17:51 16396 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Apr 25 17:51 16396_fsm 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:41 16396_vm 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 499712 Apr 25 17:51 16397 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:51 16397_fsm 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:51 16397_vm 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1695744 Apr 25 17:51 16399 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:51 16399_fsm 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:51 16399_vm 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:51 16400 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 17:51 16402 2024-04-25T17:51:39.776 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Apr 25 17:51 16404 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 74170368 Apr 25 17:51 16406 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:51 16406_fsm 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:48 16406_vm 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 174 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 175 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2187 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2224 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2328 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2336 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2337 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2579 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2600 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2600_fsm 2024-04-25T17:51:39.777 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2600_vm 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2601_fsm 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2601_vm 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2602 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2602_fsm 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2602_vm 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2603 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2603_fsm 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2603_vm 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2604 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2605 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2605_fsm 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2605_vm 2024-04-25T17:51:39.778 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2606_fsm 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2606_vm 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2607 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2607_fsm 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2607_vm 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 17:36 2608 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2608_fsm 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2608_vm 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 17:36 2609 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2609_fsm 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2609_vm 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2610 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2610_fsm 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2610_vm 2024-04-25T17:51:39.779 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2611 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2612_fsm 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2612_vm 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2613 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2615_fsm 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2615_vm 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2616 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2616_fsm 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2616_vm 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 17:36 2617 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2617_fsm 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2617_vm 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 17:36 2618 2024-04-25T17:51:39.780 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2618_fsm 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2618_vm 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 172032 Apr 25 17:51 2619 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:39 2619_fsm 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2619_vm 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2620 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2650 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2651 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2652 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2653 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2654 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2655 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2656 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2657 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 17:36 2658 2024-04-25T17:51:39.781 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 17:36 2659 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2660 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2661 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2662 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2663 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2664 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2665 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2666 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2667 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2668 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2669 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2670 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 17:36 2673 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 17:36 2674 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 17:36 2675 2024-04-25T17:51:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2678 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2679 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2680 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2681 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2682 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2683 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2684 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2685 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2686 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2687 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2688 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 17:36 2689 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 17:36 2690 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 17:36 2691 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2692 2024-04-25T17:51:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2693 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:41 2696 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2699 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2701 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2702 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2703 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 2704 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2753 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2753_fsm 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2753_vm 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2754 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2755 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2756 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 2757 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2830 2024-04-25T17:51:39.784 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2831 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2832 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2833 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2834 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2835 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2836 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2837 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 17:36 2838 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2838_fsm 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2838_vm 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2839 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2840_fsm 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2840_vm 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2841 2024-04-25T17:51:39.785 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2995 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2996 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3079_fsm 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3079_vm 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3080 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3081 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3085 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3118 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3119 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 17:36 3164 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3256 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3257 2024-04-25T17:51:39.786 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3258 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3350 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3351 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3379 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3380 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3381 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3394 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3394_fsm 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3394_vm 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3395 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3429 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3430 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3431 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3433 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3439 2024-04-25T17:51:39.787 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3440 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 17:36 3455 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 17:36 3456 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3456_fsm 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3456_vm 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3466 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3467 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3468 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3501 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3502 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3503 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3534 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3541_fsm 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3541_vm 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3542 2024-04-25T17:51:39.788 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3574 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3575 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3576 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3596 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3597 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3598 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3599 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3600_fsm 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3600_vm 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3601_fsm 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3601_vm 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3602_fsm 2024-04-25T17:51:39.789 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3602_vm 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3603_fsm 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3603_vm 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3604 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3605 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3606 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3607 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3608 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 17:36 3609 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3712 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 3764_fsm 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3764_vm 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3766 2024-04-25T17:51:39.790 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 3767 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3997 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4143 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4144 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4145 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4146 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4147 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4148 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4149 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4150 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4151 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4152 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4153 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4154 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4155 2024-04-25T17:51:39.791 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4156 2024-04-25T17:51:39.792 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4157 2024-04-25T17:51:39.792 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4158 2024-04-25T17:51:39.792 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4159 2024-04-25T17:51:39.793 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4160 2024-04-25T17:51:39.793 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4161 2024-04-25T17:51:39.793 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4162 2024-04-25T17:51:39.793 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4163 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4164 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4165 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4166 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4167 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4168 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4169 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4170 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4171 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4172 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4173 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4174 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 5002 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 548 2024-04-25T17:51:39.794 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 549 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6102 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6104 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6106 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6110 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6111 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6112 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6113 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6117 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 826 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 827 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 828 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 17:36 pg_filenode.map 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Apr 25 17:36 pg_internal.init 2024-04-25T17:51:39.795 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:36 PG_VERSION 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 17:36 .. 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/global: 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:total 560 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 25 17:36 . 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1213 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1213_fsm 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1213_vm 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1214 2024-04-25T17:51:39.796 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1214_fsm 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1214_vm 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 1232 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 1233 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1260 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1260_fsm 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1260_vm 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1261 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1261_fsm 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1261_vm 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1262 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 1262_fsm 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 1262_vm 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2396 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 17:36 2396_fsm 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2396_vm 2024-04-25T17:51:39.797 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2397 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2671 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2672 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2676 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2677 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2694 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2695 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2697 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 17:36 2698 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2846 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2847 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2964 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2965 2024-04-25T17:51:39.798 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 2966 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 2967 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 3592 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 3593 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4060 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4061 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4175 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4176 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4177 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4178 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4181 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4182 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4183 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4184 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 4185 2024-04-25T17:51:39.799 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 4186 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6000 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6001 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6002 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 6100 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6114 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 6115 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:51 pg_control 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 17:36 pg_filenode.map 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Apr 25 17:36 pg_internal.init 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/log: 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:total 2 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 . 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.800 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1354 Apr 25 17:51 postgresql-Thu.log 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:total 1 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:51 . 2024-04-25T17:51:39.801 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 mappings 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 25 17:51 replorigin_checkpoint 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 snapshots 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:51 .. 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:51 .. 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.802 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 . 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 members 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 offsets 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:total 8 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 . 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 .. 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:36 0000 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-25T17:51:39.803 INFO:teuthology.orchestra.run.smithi098.stdout:total 8 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:36 . 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:36 .. 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 17:41 0000 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:total 15 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 17:51 . 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Apr 25 17:51 db_0.stat 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Apr 25 17:51 db_13494.stat 2024-04-25T17:51:39.805 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Apr 25 17:51 global.stat 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:51 . 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:total 152 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 17:51 . 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 155648 Apr 25 17:51 0015 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.806 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:total 1048576 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 25 17:51 . 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 00000001000000000000008A 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:41 00000001000000000000008B 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:42 00000001000000000000008C 2024-04-25T17:51:39.807 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:42 00000001000000000000008D 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:41 00000001000000000000008E 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:42 00000001000000000000008F 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:41 000000010000000000000090 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:41 000000010000000000000091 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:41 000000010000000000000092 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 000000010000000000000093 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 000000010000000000000094 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 000000010000000000000095 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:47 000000010000000000000096 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:48 000000010000000000000097 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:47 000000010000000000000098 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 000000010000000000000099 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 00000001000000000000009A 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 00000001000000000000009B 2024-04-25T17:51:39.808 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:49 00000001000000000000009C 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:46 00000001000000000000009D 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 00000001000000000000009E 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 00000001000000000000009F 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:46 0000000100000000000000A0 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:46 0000000100000000000000A1 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000A2 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:46 0000000100000000000000A3 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000A4 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000A5 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:46 0000000100000000000000A6 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:47 0000000100000000000000A7 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:46 0000000100000000000000A8 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000A9 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:46 0000000100000000000000AA 2024-04-25T17:51:39.809 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000AB 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000AC 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:46 0000000100000000000000AD 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000AE 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:46 0000000100000000000000AF 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000B0 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000B1 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:46 0000000100000000000000B2 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000B3 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000B4 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:47 0000000100000000000000B5 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:48 0000000100000000000000B6 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:45 0000000100000000000000B7 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000B8 2024-04-25T17:51:39.810 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000B9 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000BA 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000BB 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000BC 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000BD 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000BE 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000BF 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000C0 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000C1 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000C2 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000C3 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000C4 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000C5 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000C6 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000C7 2024-04-25T17:51:39.811 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000C8 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 17:51 0000000100000000000000C9 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 archive_status 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:total 0 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 17:36 . 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 25 17:51 .. 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:total 352 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 17:51 . 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 17:51 .. 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Apr 25 17:51 0000 2024-04-25T17:51:39.812 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 17:51 0001 2024-04-25T17:51:39.813 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-25T17:51:39.824 INFO:tasks.quiescer:joining Quiescers 2024-04-25T17:51:39.825 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-25T17:51:39.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:51:40.063 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:51:40.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:40 smithi098 ceph-mon[29192]: pgmap v497: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 28 MiB/s wr, 1.93k op/s 2024-04-25T17:51:40.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:40 smithi173 ceph-mon[32856]: pgmap v497: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 28 MiB/s wr, 1.93k op/s 2024-04-25T17:51:40.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:40 smithi203 ceph-mon[32834]: pgmap v497: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 28 MiB/s wr, 1.93k op/s 2024-04-25T17:51:41.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:41 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:51:41.559 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:41 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:51:41.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:41 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:51:41.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.801+0000 7f48574f8700 1 -- 172.21.15.98:0/1701219893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4850108a50 msgr2=0x7f4850108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.801+0000 7f48574f8700 1 --2- 172.21.15.98:0/1701219893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4850108a50 0x7f4850108e30 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4840009990 tx=0x7f4840009ca0 comp rx=0 tx=0).stop 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.801+0000 7f48574f8700 1 -- 172.21.15.98:0/1701219893 shutdown_connections 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.801+0000 7f48574f8700 1 --2- 172.21.15.98:0/1701219893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4850103450 0x7f485010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.801+0000 7f48574f8700 1 --2- 172.21.15.98:0/1701219893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4850108a50 0x7f4850108e30 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.801+0000 7f48574f8700 1 --2- 172.21.15.98:0/1701219893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4850102a90 0x7f4850102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.801+0000 7f48574f8700 1 -- 172.21.15.98:0/1701219893 >> 172.21.15.98:0/1701219893 conn(0x7f4850079ba0 msgr2=0x7f4850079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.802+0000 7f48574f8700 1 -- 172.21.15.98:0/1701219893 shutdown_connections 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.802+0000 7f48574f8700 1 -- 172.21.15.98:0/1701219893 wait complete. 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.802+0000 7f48574f8700 1 Processor -- start 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.802+0000 7f48574f8700 1 -- start start 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.802+0000 7f48574f8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4850102a90 0x7f48501991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:41.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.802+0000 7f48574f8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4850103450 0x7f4850199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.802+0000 7f48574f8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4850108a50 0x7f485019daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.802+0000 7f48574f8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4850112c60 con 0x7f4850108a50 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.802+0000 7f48574f8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4850112ae0 con 0x7f4850102a90 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.802+0000 7f48574f8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4850112de0 con 0x7f4850103450 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4855a95700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4850108a50 0x7f485019daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4855a95700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4850108a50 0x7f485019daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39904/0 (socket says 172.21.15.98:39904) 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4855a95700 1 -- 172.21.15.98:0/87585386 learned_addr learned my addr 172.21.15.98:0/87585386 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4855294700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4850102a90 0x7f48501991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4854a93700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4850103450 0x7f4850199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4855a95700 1 -- 172.21.15.98:0/87585386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4850103450 msgr2=0x7f4850199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4855a95700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4850103450 0x7f4850199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4855a95700 1 -- 172.21.15.98:0/87585386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4850102a90 msgr2=0x7f48501991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:41.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4855a95700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4850102a90 0x7f48501991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:41.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4855a95700 1 -- 172.21.15.98:0/87585386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485019e1e0 con 0x7f4850108a50 2024-04-25T17:51:41.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f4855a95700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4850108a50 0x7f485019daa0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f484800b970 tx=0x7f484800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:41.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f48467fc700 1 -- 172.21.15.98:0/87585386 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484800d860 con 0x7f4850108a50 2024-04-25T17:51:41.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f48574f8700 1 -- 172.21.15.98:0/87585386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f485019e4d0 con 0x7f4850108a50 2024-04-25T17:51:41.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f48467fc700 1 -- 172.21.15.98:0/87585386 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f484800e430 con 0x7f4850108a50 2024-04-25T17:51:41.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f48467fc700 1 -- 172.21.15.98:0/87585386 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848017660 con 0x7f4850108a50 2024-04-25T17:51:41.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.803+0000 7f48574f8700 1 -- 172.21.15.98:0/87585386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48501a5d50 con 0x7f4850108a50 2024-04-25T17:51:41.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.804+0000 7f48574f8700 1 -- 172.21.15.98:0/87585386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4850069730 con 0x7f4850108a50 2024-04-25T17:51:41.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.805+0000 7f48467fc700 1 -- 172.21.15.98:0/87585386 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f484800da00 con 0x7f4850108a50 2024-04-25T17:51:41.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.806+0000 7f48467fc700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f483c0a3db0 0x7f483c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:41.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.806+0000 7f4855294700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f483c0a3db0 0x7f483c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:41.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.806+0000 7f48467fc700 1 -- 172.21.15.98:0/87585386 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7f48480d03f0 con 0x7f4850108a50 2024-04-25T17:51:41.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.806+0000 7f4855294700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f483c0a3db0 0x7f483c0a6270 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f484000b5c0 tx=0x7f4840005960 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:41.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:41.810+0000 7f48467fc700 1 -- 172.21.15.98:0/87585386 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4848013030 con 0x7f4850108a50 2024-04-25T17:51:42.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.057+0000 7f48574f8700 1 -- 172.21.15.98:0/87585386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f485019a530 con 0x7f4850108a50 2024-04-25T17:51:42.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.058+0000 7f48467fc700 1 -- 172.21.15.98:0/87585386 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+11879 (secure 0 0 0) 0x7f484809ca90 con 0x7f4850108a50 2024-04-25T17:51:42.059 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:42.059 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":52,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:51:39.371282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26026":{"gid":26026,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/3925531495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":3925531495},{"type":"v1","addr":"172.21.15.203:6837","nonce":3925531495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24461,24440]},"id":1}]} 2024-04-25T17:51:42.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.060+0000 7f48574f8700 1 -- 172.21.15.98:0/87585386 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f483c0a3db0 msgr2=0x7f483c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:42.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.060+0000 7f48574f8700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f483c0a3db0 0x7f483c0a6270 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f484000b5c0 tx=0x7f4840005960 comp rx=0 tx=0).stop 2024-04-25T17:51:42.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.060+0000 7f48574f8700 1 -- 172.21.15.98:0/87585386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4850108a50 msgr2=0x7f485019daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:42.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.060+0000 7f48574f8700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4850108a50 0x7f485019daa0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f484800b970 tx=0x7f484800bc80 comp rx=0 tx=0).stop 2024-04-25T17:51:42.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.060+0000 7f48574f8700 1 -- 172.21.15.98:0/87585386 shutdown_connections 2024-04-25T17:51:42.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.060+0000 7f48574f8700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f483c0a3db0 0x7f483c0a6270 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:42.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.060+0000 7f48574f8700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4850108a50 0x7f485019daa0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:42.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.060+0000 7f48574f8700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4850102a90 0x7f48501991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:42.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.060+0000 7f48574f8700 1 --2- 172.21.15.98:0/87585386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4850103450 0x7f4850199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:42.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.060+0000 7f48574f8700 1 -- 172.21.15.98:0/87585386 >> 172.21.15.98:0/87585386 conn(0x7f4850079ba0 msgr2=0x7f4850101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:42.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.061+0000 7f48574f8700 1 -- 172.21.15.98:0/87585386 shutdown_connections 2024-04-25T17:51:42.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:42.061+0000 7f48574f8700 1 -- 172.21.15.98:0/87585386 wait complete. 2024-04-25T17:51:42.062 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 52 2024-04-25T17:51:42.359 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:42 smithi098 ceph-mon[29192]: pgmap v498: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 28 MiB/s wr, 1.93k op/s 2024-04-25T17:51:42.360 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:42 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/87585386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:51:42.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:42 smithi173 ceph-mon[32856]: pgmap v498: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 28 MiB/s wr, 1.93k op/s 2024-04-25T17:51:42.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:42 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/87585386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:51:42.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:42 smithi203 ceph-mon[32834]: pgmap v498: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 28 MiB/s wr, 1.93k op/s 2024-04-25T17:51:42.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:42 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/87585386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:51:42.886 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24461 quiesce db --cancel --all' 2024-04-25T17:51:42.887 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24461 quiesce db --cancel --all 2024-04-25T17:51:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:51:44.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:44 smithi173 ceph-mon[32856]: pgmap v499: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 31 MiB/s wr, 2.25k op/s 2024-04-25T17:51:44.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:44 smithi098 ceph-mon[29192]: pgmap v499: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 31 MiB/s wr, 2.25k op/s 2024-04-25T17:51:44.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:44 smithi203 ceph-mon[32834]: pgmap v499: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 31 MiB/s wr, 2.25k op/s 2024-04-25T17:51:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.026+0000 7fef7cc18700 1 -- 172.21.15.98:0/4133425076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78102330 msgr2=0x7fef7810e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.026+0000 7fef7cc18700 1 --2- 172.21.15.98:0/4133425076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78102330 0x7fef7810e7f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fef7000b0a0 tx=0x7fef7000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.026+0000 7fef7cc18700 1 -- 172.21.15.98:0/4133425076 shutdown_connections 2024-04-25T17:51:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.026+0000 7fef7cc18700 1 --2- 172.21.15.98:0/4133425076 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78107920 0x7fef78107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.026+0000 7fef7cc18700 1 --2- 172.21.15.98:0/4133425076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78102330 0x7fef7810e7f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.026+0000 7fef7cc18700 1 --2- 172.21.15.98:0/4133425076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef78101970 0x7fef78101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.026+0000 7fef7cc18700 1 -- 172.21.15.98:0/4133425076 >> 172.21.15.98:0/4133425076 conn(0x7fef78078910 msgr2=0x7fef78078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.027+0000 7fef7cc18700 1 -- 172.21.15.98:0/4133425076 shutdown_connections 2024-04-25T17:51:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.027+0000 7fef7cc18700 1 -- 172.21.15.98:0/4133425076 wait complete. 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.027+0000 7fef7cc18700 1 Processor -- start 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.027+0000 7fef7cc18700 1 -- start start 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.027+0000 7fef7cc18700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78101970 0x7fef781034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.027+0000 7fef7cc18700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef78102330 0x7fef78103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.027+0000 7fef7cc18700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78107920 0x7fef78103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7cc18700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef78111b00 con 0x7fef78101970 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7cc18700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef78111980 con 0x7fef78102330 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7cc18700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fef78111c80 con 0x7fef78107920 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78101970 0x7fef781034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78101970 0x7fef781034f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56094/0 (socket says 172.21.15.98:56094) 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7659c700 1 -- 172.21.15.98:0/2738400558 learned_addr learned my addr 172.21.15.98:0/2738400558 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef76d9d700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78107920 0x7fef78103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef78102330 0x7fef78103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7659c700 1 -- 172.21.15.98:0/2738400558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78107920 msgr2=0x7fef78103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7659c700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78107920 0x7fef78103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7659c700 1 -- 172.21.15.98:0/2738400558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef78102330 msgr2=0x7fef78103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7659c700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef78102330 0x7fef78103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7659c700 1 -- 172.21.15.98:0/2738400558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef781a7620 con 0x7fef78101970 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef78102330 0x7fef78103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7659c700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78101970 0x7fef781034f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00d950 tx=0x7fef6c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef677fe700 1 -- 172.21.15.98:0/2738400558 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c016070 con 0x7fef78101970 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7cc18700 1 -- 172.21.15.98:0/2738400558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef781a7910 con 0x7fef78101970 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef677fe700 1 -- 172.21.15.98:0/2738400558 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef6c009df0 con 0x7fef78101970 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef677fe700 1 -- 172.21.15.98:0/2738400558 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c00f3c0 con 0x7fef78101970 2024-04-25T17:51:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.028+0000 7fef7cc18700 1 -- 172.21.15.98:0/2738400558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef781a7ea0 con 0x7fef78101970 2024-04-25T17:51:45.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.030+0000 7fef677fe700 1 -- 172.21.15.98:0/2738400558 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fef6c019070 con 0x7fef78101970 2024-04-25T17:51:45.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.031+0000 7fef677fe700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fef600a3db0 0x7fef600a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.031+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fef600a3db0 0x7fef600a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.031+0000 7fef677fe700 1 -- 172.21.15.98:0/2738400558 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fef6c0d0360 con 0x7fef78101970 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.031+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fef600a3db0 0x7fef600a6270 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fef68005950 tx=0x7fef680058e0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef874a1b80 1 Processor -- start 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef874a1b80 1 -- start start 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f78d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f79280 0x55e0c6f83c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f84150 0x55e0c6f86540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef874a1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e0c6ecbda0 con 0x55e0c6f78960 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef874a1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e0c6e729d0 con 0x55e0c6f79280 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef874a1b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e0c6ec7d60 con 0x55e0c6f84150 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef7659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f78d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef7659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f78d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56108/0 (socket says 172.21.15.98:56108) 2024-04-25T17:51:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.032+0000 7fef7659c700 1 -- 172.21.15.98:0/64683820 learned_addr learned my addr 172.21.15.98:0/64683820 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef76d9d700 1 --2- 172.21.15.98:0/64683820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f84150 0x55e0c6f86540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef75d9b700 1 --2- 172.21.15.98:0/64683820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f79280 0x55e0c6f83c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef7659c700 1 -- 172.21.15.98:0/64683820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f84150 msgr2=0x55e0c6f86540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef7659c700 1 --2- 172.21.15.98:0/64683820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f84150 0x55e0c6f86540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef7659c700 1 -- 172.21.15.98:0/64683820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f79280 msgr2=0x55e0c6f83c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef7659c700 1 --2- 172.21.15.98:0/64683820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f79280 0x55e0c6f83c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef7659c700 1 -- 172.21.15.98:0/64683820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0c6e19940 con 0x55e0c6f78960 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef7659c700 1 --2- 172.21.15.98:0/64683820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f78d40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0a4a40 tx=0x7fef6c09cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfce1eca0b59aa09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef64ff9700 1 -- 172.21.15.98:0/64683820 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c020040 con 0x55e0c6f78960 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef64ff9700 1 -- 172.21.15.98:0/64683820 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef6c0a4f70 con 0x55e0c6f78960 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.033+0000 7fef64ff9700 1 -- 172.21.15.98:0/64683820 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c0a4ae0 con 0x55e0c6f78960 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 -- 172.21.15.98:0/64683820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 msgr2=0x55e0c6f78d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 --2- 172.21.15.98:0/64683820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f78d40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fef6c0a4a40 tx=0x7fef6c09cad0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 -- 172.21.15.98:0/64683820 shutdown_connections 2024-04-25T17:51:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 --2- 172.21.15.98:0/64683820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f84150 0x55e0c6f86540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 --2- 172.21.15.98:0/64683820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f78d40 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 --2- 172.21.15.98:0/64683820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f79280 0x55e0c6f83c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 -- 172.21.15.98:0/64683820 >> 172.21.15.98:0/64683820 conn(0x55e0c6ea6700 msgr2=0x55e0c6f74e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 -- 172.21.15.98:0/64683820 shutdown_connections 2024-04-25T17:51:45.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 -- 172.21.15.98:0/64683820 wait complete. 2024-04-25T17:51:45.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 -- start start 2024-04-25T17:51:45.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.034+0000 7fef874a1b80 1 Processor -- start 2024-04-25T17:51:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f78960 0x55e0c6f838f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f80bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f84150 0x55e0c6f81130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef874a1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e0c6f86a80 con 0x55e0c6f79280 2024-04-25T17:51:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef874a1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e0c6e6ac40 con 0x55e0c6f78960 2024-04-25T17:51:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef874a1b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e0c6f86c00 con 0x55e0c6f84150 2024-04-25T17:51:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef874a1b80 10 client.0 _ll_register_callbacks cb 0x55e0c6e649c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:51:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef75d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f80bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef75d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f80bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56116/0 (socket says 172.21.15.98:56116) 2024-04-25T17:51:45.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef75d9b700 1 -- 172.21.15.98:0/2401474273 learned_addr learned my addr 172.21.15.98:0/2401474273 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef7659c700 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f78960 0x55e0c6f838f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef76d9d700 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f84150 0x55e0c6f81130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef75d9b700 1 -- 172.21.15.98:0/2401474273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f84150 msgr2=0x55e0c6f81130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f84150 0x55e0c6f81130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef75d9b700 1 -- 172.21.15.98:0/2401474273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f78960 msgr2=0x55e0c6f838f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f78960 0x55e0c6f838f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef75d9b700 1 -- 172.21.15.98:0/2401474273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0c6e832e0 con 0x55e0c6f79280 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f80bf0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fef6800fe30 tx=0x7fef68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947f6fa022f7881d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef5dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef5dffb700 1 -- 172.21.15.98:0/2401474273 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68023020 con 0x55e0c6f79280 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef5dffb700 1 -- 172.21.15.98:0/2401474273 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fef68003880 con 0x55e0c6f79280 2024-04-25T17:51:45.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.035+0000 7fef874a1b80 1 -- 172.21.15.98:0/2401474273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef6c0a4f70 con 0x55e0c6f79280 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef5dffb700 1 -- 172.21.15.98:0/2401474273 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fef6802e050 con 0x55e0c6f79280 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef874a1b80 10 client.16917 fetch_fsmap learned FSMap version 52 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef874a1b80 1 -- 172.21.15.98:0/2401474273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x55e0c6f913c0 con 0x55e0c6f79280 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef5dffb700 1 -- 172.21.15.98:0/2401474273 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef6c0a4f70 con 0x55e0c6f79280 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef5dffb700 1 -- 172.21.15.98:0/2401474273 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 52) ==== 8346+0+0 (secure 0 0 0) 0x7fef68011b00 con 0x55e0c6f79280 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef874a1b80 10 client.16917 fetch_fsmap finished waiting for FSMap version 52 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef874a1b80 10 client.16917 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef874a1b80 20 client.16917 populate_metadata read hostname 'smithi098' 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e0c6f93a00 0x55e0c6f95e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef874a1b80 4 client.16917 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:51:45.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef874a1b80 1 -- 172.21.15.98:0/2401474273 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0c6ed9590 con 0x55e0c6f93a00 2024-04-25T17:51:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.036+0000 7fef7659c700 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e0c6f93a00 0x55e0c6f95e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.037+0000 7fef7659c700 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e0c6f93a00 0x55e0c6f95e30 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e99e8f47ab2ac2f0 server_cookie=f4e89d64d3cbfb8c in_seq=0 out_seq=0 2024-04-25T17:51:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.037+0000 7fef5dffb700 10 client.16917 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.037+0000 7fef5dffb700 1 -- 172.21.15.98:0/2401474273 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0c6ed9590 con 0x55e0c6f93a00 2024-04-25T17:51:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.037+0000 7fef5dffb700 10 client.16917 handle_command_reply: tid=0 2024-04-25T17:51:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.037+0000 7fef874a1b80 1 client.16917 shutdown 2024-04-25T17:51:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.037+0000 7fef874a1b80 1 -- 172.21.15.98:0/2401474273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 msgr2=0x55e0c6f80bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.037+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f80bf0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fef6800fe30 tx=0x7fef68004750 comp rx=0 tx=0).stop 2024-04-25T17:51:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.038+0000 7fef874a1b80 1 -- 172.21.15.98:0/2401474273 shutdown_connections 2024-04-25T17:51:45.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.038+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e0c6f93a00 0x55e0c6f95e30 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.038+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f84150 0x55e0c6f81130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.038+0000 7fef5dffb700 0 client.16917 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.038+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f78960 0x55e0c6f838f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.038+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2401474273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f80bf0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.038+0000 7fef874a1b80 1 -- 172.21.15.98:0/2401474273 >> 172.21.15.98:0/2401474273 conn(0x55e0c6ea6700 msgr2=0x55e0c6f74a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.038+0000 7fef874a1b80 1 -- 172.21.15.98:0/2401474273 shutdown_connections 2024-04-25T17:51:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.038+0000 7fef874a1b80 1 -- 172.21.15.98:0/2401474273 wait complete. 2024-04-25T17:51:45.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.038+0000 7fef874a1b80 20 client.16917 trim_cache size 0 max 16384 2024-04-25T17:51:45.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef874a1b80 1 Processor -- start 2024-04-25T17:51:45.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef874a1b80 1 -- start start 2024-04-25T17:51:45.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f78960 0x55e0c6f74fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f9b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 0x55e0c6f9bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef874a1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef781b1e00 con 0x55e0c6f79280 2024-04-25T17:51:45.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef874a1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef781af660 con 0x55e0c6f84150 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef874a1b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fef781ad4d0 con 0x55e0c6f78960 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef75d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f9b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef75d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f9b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56120/0 (socket says 172.21.15.98:56120) 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef75d9b700 1 -- 172.21.15.98:0/2753154661 learned_addr learned my addr 172.21.15.98:0/2753154661 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef75d9b700 1 -- 172.21.15.98:0/2753154661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f78960 msgr2=0x55e0c6f74fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef7659c700 1 --2- 172.21.15.98:0/2753154661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f78960 0x55e0c6f74fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef76d9d700 1 --2- 172.21.15.98:0/2753154661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 0x55e0c6f9bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2753154661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f78960 0x55e0c6f74fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef75d9b700 1 -- 172.21.15.98:0/2753154661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 msgr2=0x55e0c6f9bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.057+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2753154661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 0x55e0c6f9bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef75d9b700 1 -- 172.21.15.98:0/2753154661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0c6f76350 con 0x55e0c6f79280 2024-04-25T17:51:45.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef76d9d700 1 --2- 172.21.15.98:0/2753154661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 0x55e0c6f9bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2753154661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f9b700 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fef68000c00 tx=0x7fef68005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f946ece75c8206 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef5ffff700 1 -- 172.21.15.98:0/2753154661 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68023040 con 0x55e0c6f79280 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef5ffff700 1 -- 172.21.15.98:0/2753154661 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef6800fe60 con 0x55e0c6f79280 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef5ffff700 1 -- 172.21.15.98:0/2753154661 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68004a20 con 0x55e0c6f79280 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef874a1b80 1 -- 172.21.15.98:0/2753154661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 msgr2=0x55e0c6f9b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2753154661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f9b700 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fef68000c00 tx=0x7fef68005a00 comp rx=0 tx=0).stop 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef874a1b80 1 -- 172.21.15.98:0/2753154661 shutdown_connections 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2753154661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f78960 0x55e0c6f74fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2753154661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 0x55e0c6f9bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2753154661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f79280 0x55e0c6f9b700 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.058+0000 7fef874a1b80 1 -- 172.21.15.98:0/2753154661 >> 172.21.15.98:0/2753154661 conn(0x55e0c6ea6700 msgr2=0x55e0c6f748c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 1 -- 172.21.15.98:0/2753154661 shutdown_connections 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 1 -- 172.21.15.98:0/2753154661 wait complete. 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 1 -- start start 2024-04-25T17:51:45.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 1 Processor -- start 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f8c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f79280 0x55e0c6f8a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 0x55e0c6f8aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef781adb20 con 0x55e0c6f78960 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef781aef90 con 0x55e0c6f84150 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fef781ae1f0 con 0x55e0c6f79280 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef874a1b80 10 client.0 _ll_register_callbacks cb 0x55e0c6e649c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef7659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f8c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef7659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f8c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56128/0 (socket says 172.21.15.98:56128) 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef7659c700 1 -- 172.21.15.98:0/2252640898 learned_addr learned my addr 172.21.15.98:0/2252640898 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f79280 0x55e0c6f8a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.059+0000 7fef76d9d700 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 0x55e0c6f8aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef7659c700 1 -- 172.21.15.98:0/2252640898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f79280 msgr2=0x55e0c6f8a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef7659c700 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f79280 0x55e0c6f8a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef7659c700 1 -- 172.21.15.98:0/2252640898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 msgr2=0x55e0c6f8aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef7659c700 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 0x55e0c6f8aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef7659c700 1 -- 172.21.15.98:0/2252640898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0c6f8b3e0 con 0x55e0c6f78960 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef7659c700 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f8c260 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fef6c000c00 tx=0x7fef6c0a0830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44dcb83cda158438 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef5dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef5dffb700 1 -- 172.21.15.98:0/2252640898 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c020040 con 0x55e0c6f78960 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef5dffb700 1 -- 172.21.15.98:0/2252640898 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fef6c09c560 con 0x55e0c6f78960 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef874a1b80 1 -- 172.21.15.98:0/2252640898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef6800fe60 con 0x55e0c6f78960 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef5dffb700 1 -- 172.21.15.98:0/2252640898 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fef6c07d050 con 0x55e0c6f78960 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef874a1b80 10 client.16923 fetch_fsmap learned FSMap version 52 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef874a1b80 1 -- 172.21.15.98:0/2252640898 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x55e0c6ffaf50 con 0x55e0c6f78960 2024-04-25T17:51:45.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef5dffb700 1 -- 172.21.15.98:0/2252640898 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef6800fe60 con 0x55e0c6f78960 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef5dffb700 1 -- 172.21.15.98:0/2252640898 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 52) ==== 8346+0+0 (secure 0 0 0) 0x7fef6c0a4ae0 con 0x55e0c6f78960 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef874a1b80 10 client.16923 fetch_fsmap finished waiting for FSMap version 52 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef874a1b80 10 client.16923 resolve_mds: validated gid 24461 aka daemon mds.b 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef874a1b80 20 client.16923 populate_metadata read hostname 'smithi098' 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e0c6ffb240 0x55e0c6ffd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef874a1b80 4 client.16923 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.060+0000 7fef874a1b80 1 -- 172.21.15.98:0/2252640898 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55e0c6ed3630 con 0x55e0c6ffb240 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e0c6ffb240 0x55e0c6ffd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef5dffb700 10 client.16923 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef75d9b700 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e0c6ffb240 0x55e0c6ffd630 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3ce3a45817234cf server_cookie=9320629c19d4b78d in_seq=0 out_seq=0 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef5dffb700 1 -- 172.21.15.98:0/2252640898 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55e0c6ed3630 con 0x55e0c6ffb240 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef5dffb700 10 client.16923 handle_command_reply: tid=0 2024-04-25T17:51:45.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef874a1b80 1 client.16923 shutdown 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef874a1b80 1 -- 172.21.15.98:0/2252640898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 msgr2=0x55e0c6f8c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f8c260 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fef6c000c00 tx=0x7fef6c0a0830 comp rx=0 tx=0).stop 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef874a1b80 1 -- 172.21.15.98:0/2252640898 shutdown_connections 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x55e0c6ffb240 0x55e0c6ffd630 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef5dffb700 0 client.16923 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0c6f84150 0x55e0c6f8aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0c6f78960 0x55e0c6f8c260 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.061+0000 7fef874a1b80 1 --2- 172.21.15.98:0/2252640898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0c6f79280 0x55e0c6f8a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef874a1b80 1 -- 172.21.15.98:0/2252640898 >> 172.21.15.98:0/2252640898 conn(0x55e0c6ea6700 msgr2=0x55e0c6f82420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef874a1b80 1 -- 172.21.15.98:0/2252640898 shutdown_connections 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef874a1b80 1 -- 172.21.15.98:0/2252640898 wait complete. 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef874a1b80 20 client.16923 trim_cache size 0 max 16384 2024-04-25T17:51:45.067 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:51:45.068 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 52, 2024-04-25T17:51:45.068 INFO:teuthology.orchestra.run.smithi098.stdout: "leader": 24461, 2024-04-25T17:51:45.068 INFO:teuthology.orchestra.run.smithi098.stdout: "set_version": 43, 2024-04-25T17:51:45.068 INFO:teuthology.orchestra.run.smithi098.stdout: "sets": {} 2024-04-25T17:51:45.068 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:51:45.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 -- 172.21.15.98:0/2738400558 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fef600a3db0 msgr2=0x7fef600a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fef600a3db0 0x7fef600a6270 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fef68005950 tx=0x7fef680058e0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 -- 172.21.15.98:0/2738400558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78101970 msgr2=0x7fef781034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78101970 0x7fef781034f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00d950 tx=0x7fef6c00dc60 comp rx=0 tx=0).stop 2024-04-25T17:51:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 -- 172.21.15.98:0/2738400558 shutdown_connections 2024-04-25T17:51:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fef600a3db0 0x7fef600a6270 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78107920 0x7fef78103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78101970 0x7fef781034f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 --2- 172.21.15.98:0/2738400558 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef78102330 0x7fef78103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 -- 172.21.15.98:0/2738400558 >> 172.21.15.98:0/2738400558 conn(0x7fef78078910 msgr2=0x7fef78105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 -- 172.21.15.98:0/2738400558 shutdown_connections 2024-04-25T17:51:45.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:45.062+0000 7fef7cc18700 1 -- 172.21.15.98:0/2738400558 wait complete. 2024-04-25T17:51:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.750+0000 7fdd78c43640 1 -- 172.21.15.98:0/2247492652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74101cc0 msgr2=0x7fdd74105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.750+0000 7fdd78c43640 1 --2- 172.21.15.98:0/2247492652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74101cc0 0x7fdd74105ab0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fdd68009900 tx=0x7fdd6802a410 comp rx=0 tx=0).stop 2024-04-25T17:51:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.750+0000 7fdd78c43640 1 -- 172.21.15.98:0/2247492652 shutdown_connections 2024-04-25T17:51:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.750+0000 7fdd78c43640 1 --2- 172.21.15.98:0/2247492652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd74106190 0x7fdd7410dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.750+0000 7fdd78c43640 1 --2- 172.21.15.98:0/2247492652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74101cc0 0x7fdd74105ab0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.750+0000 7fdd78c43640 1 --2- 172.21.15.98:0/2247492652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd74101310 0x7fdd741016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.750+0000 7fdd78c43640 1 -- 172.21.15.98:0/2247492652 >> 172.21.15.98:0/2247492652 conn(0x7fdd74077cc0 msgr2=0x7fdd740780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.750+0000 7fdd78c43640 1 -- 172.21.15.98:0/2247492652 shutdown_connections 2024-04-25T17:51:45.752 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.750+0000 7fdd78c43640 1 -- 172.21.15.98:0/2247492652 wait complete. 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd78c43640 1 Processor -- start 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd78c43640 1 -- start start 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd78c43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74101310 0x7fdd74199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd78c43640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd74106190 0x7fdd74199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd78c43640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd74199ff0 0x7fdd741a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd78c43640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd74111100 con 0x7fdd74101310 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd78c43640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd74110f80 con 0x7fdd74199ff0 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd78c43640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd74111280 con 0x7fdd74106190 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd7259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74101310 0x7fdd74199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd7259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74101310 0x7fdd74199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56154/0 (socket says 172.21.15.98:56154) 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.751+0000 7fdd7259c640 1 -- 172.21.15.98:0/2006220136 learned_addr learned my addr 172.21.15.98:0/2006220136 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd74106190 0x7fdd74199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd72d9d640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd74199ff0 0x7fdd741a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd7259c640 1 -- 172.21.15.98:0/2006220136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd74106190 msgr2=0x7fdd74199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd7259c640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd74106190 0x7fdd74199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd7259c640 1 -- 172.21.15.98:0/2006220136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd74199ff0 msgr2=0x7fdd741a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd7259c640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd74199ff0 0x7fdd741a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd7259c640 1 -- 172.21.15.98:0/2006220136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd741a4d70 con 0x7fdd74101310 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd74106190 0x7fdd74199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd7259c640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74101310 0x7fdd74199520 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400c890 tx=0x7fdd6400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd637fe640 1 -- 172.21.15.98:0/2006220136 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd64004220 con 0x7fdd74101310 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd637fe640 1 -- 172.21.15.98:0/2006220136 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd640043c0 con 0x7fdd74101310 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd78c43640 1 -- 172.21.15.98:0/2006220136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd741a5060 con 0x7fdd74101310 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd637fe640 1 -- 172.21.15.98:0/2006220136 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd64004a00 con 0x7fdd74101310 2024-04-25T17:51:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.752+0000 7fdd78c43640 1 -- 172.21.15.98:0/2006220136 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd741a5580 con 0x7fdd74101310 2024-04-25T17:51:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.754+0000 7fdd637fe640 1 -- 172.21.15.98:0/2006220136 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdd640079b0 con 0x7fdd74101310 2024-04-25T17:51:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.754+0000 7fdd7ba38740 1 Processor -- start 2024-04-25T17:51:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd7ba38740 1 -- start start 2024-04-25T17:51:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9eb95a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 0x564e9eba02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eba0800 0x564e9eba2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564e9eafa970 con 0x564e9eba0800 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564e9ea974a0 con 0x564e9eb95620 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564e9eaf6930 con 0x564e9eb95f40 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd637fe640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdd5c0a3dc0 0x7fdd5c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd71d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eba0800 0x564e9eba2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd71d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eba0800 0x564e9eba2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56168/0 (socket says 172.21.15.98:56168) 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd71d9b640 1 -- 172.21.15.98:0/3361160259 learned_addr learned my addr 172.21.15.98:0/3361160259 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd72d9d640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdd5c0a3dc0 0x7fdd5c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd7259c640 1 --2- 172.21.15.98:0/3361160259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 0x564e9eba02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd72d9d640 1 --2- 172.21.15.98:0/3361160259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9eb95a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd637fe640 1 -- 172.21.15.98:0/2006220136 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fdd640d0600 con 0x7fdd74101310 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd71d9b640 1 -- 172.21.15.98:0/3361160259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 msgr2=0x564e9eba02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/3361160259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 0x564e9eba02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd71d9b640 1 -- 172.21.15.98:0/3361160259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 msgr2=0x564e9eb95a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/3361160259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9eb95a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd71d9b640 1 -- 172.21.15.98:0/3361160259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e9e80d210 con 0x564e9eba0800 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd7259c640 1 --2- 172.21.15.98:0/3361160259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 0x564e9eba02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/3361160259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eba0800 0x564e9eba2bf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fdd68031440 tx=0x7fdd68032de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e1201c3082eb65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.755+0000 7fdd72d9d640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdd5c0a3dc0 0x7fdd5c0a6280 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c005fd0 tx=0x7fdd6c009b90 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd60ff9640 1 -- 172.21.15.98:0/3361160259 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6803f070 con 0x564e9eba0800 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd60ff9640 1 -- 172.21.15.98:0/3361160259 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd680316a0 con 0x564e9eba0800 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd60ff9640 1 -- 172.21.15.98:0/3361160259 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd68035070 con 0x564e9eba0800 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd7ba38740 1 -- 172.21.15.98:0/3361160259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eba0800 msgr2=0x564e9eba2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/3361160259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eba0800 0x564e9eba2bf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fdd68031440 tx=0x7fdd68032de0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd7ba38740 1 -- 172.21.15.98:0/3361160259 shutdown_connections 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/3361160259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eba0800 0x564e9eba2bf0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/3361160259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 0x564e9eba02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/3361160259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9eb95a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd7ba38740 1 -- 172.21.15.98:0/3361160259 >> 172.21.15.98:0/3361160259 conn(0x564e9eb91b30 msgr2=0x564e9eb91f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.756+0000 7fdd7ba38740 1 -- 172.21.15.98:0/3361160259 shutdown_connections 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7ba38740 1 -- 172.21.15.98:0/3361160259 wait complete. 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7ba38740 1 -- start start 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7ba38740 1 Processor -- start 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95620 0x564e9eba0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 0x564e9eb9d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eba0800 0x564e9eb9d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564e9eba33d0 con 0x564e9eb95620 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564e9ea950b0 con 0x564e9eba0800 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564e9eba3550 con 0x564e9eb95f40 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7ba38740 10 client.0 _ll_register_callbacks cb 0x564e9ea82d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95620 0x564e9eba0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95620 0x564e9eba0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56180/0 (socket says 172.21.15.98:56180) 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7259c640 1 -- 172.21.15.98:0/2971199244 learned_addr learned my addr 172.21.15.98:0/2971199244 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd72d9d640 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eba0800 0x564e9eb9d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 0x564e9eb9d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7259c640 1 -- 172.21.15.98:0/2971199244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 msgr2=0x564e9eb9d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7259c640 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 0x564e9eb9d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7259c640 1 -- 172.21.15.98:0/2971199244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eba0800 msgr2=0x564e9eb9d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7259c640 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eba0800 0x564e9eb9d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd7259c640 1 -- 172.21.15.98:0/2971199244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e9e8d9620 con 0x564e9eb95620 2024-04-25T17:51:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.757+0000 7fdd72d9d640 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eba0800 0x564e9eb9d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd7259c640 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95620 0x564e9eba0170 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fdd64098a00 tx=0x7fdd6401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac002e67cfcf1e09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2971199244 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd64005290 con 0x564e9eb95620 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2971199244 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fdd64098c10 con 0x564e9eb95620 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2971199244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd680316a0 con 0x564e9eb95620 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2971199244 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fdd6409cca0 con 0x564e9eb95620 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd7ba38740 10 client.16935 fetch_fsmap learned FSMap version 52 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2971199244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x564e9ebaf360 con 0x564e9eb95620 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2971199244 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd680316a0 con 0x564e9eb95620 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2971199244 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 52) ==== 8346+0+0 (secure 0 0 0) 0x7fdd6401ed20 con 0x564e9eb95620 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd7ba38740 10 client.16935 fetch_fsmap finished waiting for FSMap version 52 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd7ba38740 10 client.16935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd7ba38740 20 client.16935 populate_metadata read hostname 'smithi098' 2024-04-25T17:51:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564e9ebb07d0 0x564e9ebb2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd7ba38740 4 client.16935 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.758+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2971199244 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e9eb08390 con 0x564e9ebb07d0 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.759+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564e9ebb07d0 0x564e9ebb2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.759+0000 7fdd59ffb640 10 client.16935 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.759+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564e9ebb07d0 0x564e9ebb2be0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fc8e2151bc3d3fa server_cookie=77e6ac2232c52cb5 in_seq=0 out_seq=0 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2971199244 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e9eb08390 con 0x564e9ebb07d0 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd59ffb640 10 client.16935 handle_command_reply: tid=0 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 client.16935 shutdown 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2971199244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95620 msgr2=0x564e9eba0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95620 0x564e9eba0170 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fdd64098a00 tx=0x7fdd6401ecb0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2971199244 shutdown_connections 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564e9ebb07d0 0x564e9ebb2be0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eba0800 0x564e9eb9d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd59ffb640 0 client.16935 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eb95f40 0x564e9eb9d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2971199244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95620 0x564e9eba0170 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2971199244 >> 172.21.15.98:0/2971199244 conn(0x564e9eb91b30 msgr2=0x564e9eb97860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2971199244 shutdown_connections 2024-04-25T17:51:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2971199244 wait complete. 2024-04-25T17:51:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.760+0000 7fdd7ba38740 20 client.16935 trim_cache size 0 max 16384 2024-04-25T17:51:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd7ba38740 1 Processor -- start 2024-04-25T17:51:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd7ba38740 1 -- start start 2024-04-25T17:51:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9ebac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eb976e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eba0800 0x564e9eb9fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd741b03d0 con 0x564e9eb95f40 2024-04-25T17:51:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd741adc70 con 0x564e9eb95620 2024-04-25T17:51:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd741abae0 con 0x564e9eba0800 2024-04-25T17:51:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd71d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eb976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd71d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eb976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56188/0 (socket says 172.21.15.98:56188) 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd71d9b640 1 -- 172.21.15.98:0/1721086130 learned_addr learned my addr 172.21.15.98:0/1721086130 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd72d9d640 1 --2- 172.21.15.98:0/1721086130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eba0800 0x564e9eb9fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd71d9b640 1 -- 172.21.15.98:0/1721086130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eba0800 msgr2=0x564e9eb9fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/1721086130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eba0800 0x564e9eb9fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd71d9b640 1 -- 172.21.15.98:0/1721086130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 msgr2=0x564e9ebac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd7259c640 1 --2- 172.21.15.98:0/1721086130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9ebac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/1721086130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9ebac290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd71d9b640 1 -- 172.21.15.98:0/1721086130 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e9eb9b9b0 con 0x564e9eb95f40 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.776+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/1721086130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eb976e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fdd68009900 tx=0x7fdd68005310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c7f93480540d6e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd5bfff640 1 -- 172.21.15.98:0/1721086130 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd68052070 con 0x564e9eb95f40 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd5bfff640 1 -- 172.21.15.98:0/1721086130 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd68005380 con 0x564e9eb95f40 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd5bfff640 1 -- 172.21.15.98:0/1721086130 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6803f040 con 0x564e9eb95f40 2024-04-25T17:51:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd7ba38740 1 -- 172.21.15.98:0/1721086130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 msgr2=0x564e9eb976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/1721086130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eb976e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fdd68009900 tx=0x7fdd68005310 comp rx=0 tx=0).stop 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd7ba38740 1 -- 172.21.15.98:0/1721086130 shutdown_connections 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/1721086130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eba0800 0x564e9eb9fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/1721086130 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eb976e0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/1721086130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9ebac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd7ba38740 1 -- 172.21.15.98:0/1721086130 >> 172.21.15.98:0/1721086130 conn(0x564e9eb91b30 msgr2=0x564e9eb96fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd7ba38740 1 -- 172.21.15.98:0/1721086130 shutdown_connections 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.777+0000 7fdd7ba38740 1 -- 172.21.15.98:0/1721086130 wait complete. 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7ba38740 1 -- start start 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7ba38740 1 Processor -- start 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9eba41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eba4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7ba38740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eba0800 0x564e9eba4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdd741ac130 con 0x564e9eb95f40 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdd741ad5a0 con 0x564e9eb95620 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7ba38740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd741ac800 con 0x564e9eba0800 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7ba38740 10 client.0 _ll_register_callbacks cb 0x7fdd64098180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd71d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eba4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd71d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eba4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56196/0 (socket says 172.21.15.98:56196) 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd71d9b640 1 -- 172.21.15.98:0/2637109106 learned_addr learned my addr 172.21.15.98:0/2637109106 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7259c640 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9eba41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd71d9b640 1 -- 172.21.15.98:0/2637109106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eba0800 msgr2=0x564e9eba4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd72d9d640 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eba0800 0x564e9eba4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eba0800 0x564e9eba4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd71d9b640 1 -- 172.21.15.98:0/2637109106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 msgr2=0x564e9eba41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9eba41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd71d9b640 1 -- 172.21.15.98:0/2637109106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e9ec15d30 con 0x564e9eb95f40 2024-04-25T17:51:45.783 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd7259c640 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9eba41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd71d9b640 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eba4720 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fdd68050040 tx=0x7fdd68005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbd475db43c7dcd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.778+0000 7fdd59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2637109106 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6805b060 con 0x564e9eb95f40 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2637109106 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fdd68004240 con 0x564e9eb95f40 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2637109106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd68005380 con 0x564e9eb95f40 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2637109106 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fdd68052070 con 0x564e9eb95f40 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd7ba38740 10 client.16941 fetch_fsmap learned FSMap version 52 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2637109106 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x564e9ec17010 con 0x564e9eb95f40 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2637109106 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd68005380 con 0x564e9eb95f40 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2637109106 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 52) ==== 8346+0+0 (secure 0 0 0) 0x7fdd6804d050 con 0x564e9eb95f40 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd7ba38740 10 client.16941 fetch_fsmap finished waiting for FSMap version 52 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd7ba38740 10 client.16941 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd7ba38740 20 client.16941 populate_metadata read hostname 'smithi098' 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564e9ec172f0 0x564e9ec196e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd7ba38740 4 client.16941 mds_command: new command op to 24461 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T17:51:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2637109106 --> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564e9eb02200 con 0x564e9ec172f0 2024-04-25T17:51:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.779+0000 7fdd7259c640 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564e9ec172f0 0x564e9ec196e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.780+0000 7fdd59ffb640 10 client.16941 ms_handle_connect on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.780+0000 7fdd7259c640 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564e9ec172f0 0x564e9ec196e0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c2706743b96f0b6 server_cookie=e73cd625792ed6ab in_seq=0 out_seq=0 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd59ffb640 1 -- 172.21.15.98:0/2637109106 <== mds.0 v2:172.21.15.173:6838/2342732992 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564e9eb02200 con 0x564e9ec172f0 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd59ffb640 10 client.16941 handle_command_reply: tid=0 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 client.16941 shutdown 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2637109106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 msgr2=0x564e9eba4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eba4720 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fdd68050040 tx=0x7fdd68005a20 comp rx=0 tx=0).stop 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2637109106 shutdown_connections 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.173:6838/2342732992,v1:172.21.15.173:6839/2342732992] conn(0x564e9ec172f0 0x564e9ec196e0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e9eba0800 0x564e9eba4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564e9eb95f40 0x564e9eba4720 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd59ffb640 0 client.16941 ms_handle_reset on v2:172.21.15.173:6838/2342732992 2024-04-25T17:51:45.815 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 --2- 172.21.15.98:0/2637109106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564e9eb95620 0x564e9eba41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2637109106 >> 172.21.15.98:0/2637109106 conn(0x564e9eb91b30 msgr2=0x564e9eb97860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2637109106 shutdown_connections 2024-04-25T17:51:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 1 -- 172.21.15.98:0/2637109106 wait complete. 2024-04-25T17:51:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd7ba38740 20 client.16941 trim_cache size 0 max 16384 2024-04-25T17:51:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:{ 2024-04-25T17:51:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "message": "", 2024-04-25T17:51:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout: "return_code": 0 2024-04-25T17:51:45.816 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stdout:} 2024-04-25T17:51:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd78c43640 1 -- 172.21.15.98:0/2006220136 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdd5c0a3dc0 msgr2=0x7fdd5c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd78c43640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdd5c0a3dc0 0x7fdd5c0a6280 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c005fd0 tx=0x7fdd6c009b90 comp rx=0 tx=0).stop 2024-04-25T17:51:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd78c43640 1 -- 172.21.15.98:0/2006220136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74101310 msgr2=0x7fdd74199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.815+0000 7fdd78c43640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74101310 0x7fdd74199520 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400c890 tx=0x7fdd6400cd50 comp rx=0 tx=0).stop 2024-04-25T17:51:45.817 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.816+0000 7fdd78c43640 1 -- 172.21.15.98:0/2006220136 shutdown_connections 2024-04-25T17:51:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.816+0000 7fdd78c43640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdd5c0a3dc0 0x7fdd5c0a6280 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.816+0000 7fdd78c43640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd74199ff0 0x7fdd741a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.816+0000 7fdd78c43640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd74106190 0x7fdd74199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.816+0000 7fdd78c43640 1 --2- 172.21.15.98:0/2006220136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdd74101310 0x7fdd74199520 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.816+0000 7fdd78c43640 1 -- 172.21.15.98:0/2006220136 >> 172.21.15.98:0/2006220136 conn(0x7fdd74077cc0 msgr2=0x7fdd74100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.816+0000 7fdd78c43640 1 -- 172.21.15.98:0/2006220136 shutdown_connections 2024-04-25T17:51:45.818 INFO:teuthology.task.background_exec.ubuntu@smithi098.front.sepia.ceph.com.smithi098.stderr:2024-04-25T17:51:45.816+0000 7fdd78c43640 1 -- 172.21.15.98:0/2006220136 wait complete. 2024-04-25T17:51:45.879 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_3071830b3533a96301fd87d582ed5f17f0b618cd/qa/tasks/quiescer.py", line 411, in task stop_all_quiescers(ctx.ceph[cluster_name].thrashers) File "/home/teuthworker/src/github.com_ceph_ceph_3071830b3533a96301fd87d582ed5f17f0b618cd/qa/tasks/quiescer.py", line 370, in stop_all_quiescers raise RuntimeError(f"error during quiesce thrashing: {thrasher.exception}") RuntimeError: error during quiesce thrashing: Error releasing set 'd4deafbb': 1 (EPERM) 2024-04-25T17:51:45.880 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-25T17:51:45.891 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:51:45.899 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-25T17:51:45.910 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-25T17:51:45.910 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T17:51:45.910 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:51:45.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-25T17:51:45.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-25T17:51:46.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T17:51:46.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 -- 172.21.15.98:0/2770865858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb88106f70 msgr2=0x7ffb88107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 --2- 172.21.15.98:0/2770865858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb88106f70 0x7ffb88107350 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009900 tx=0x7ffb7802a410 comp rx=0 tx=0).stop 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 -- 172.21.15.98:0/2770865858 shutdown_connections 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 --2- 172.21.15.98:0/2770865858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb8806b120 0x7ffb8810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 --2- 172.21.15.98:0/2770865858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb8806a760 0x7ffb8806abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 --2- 172.21.15.98:0/2770865858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb88106f70 0x7ffb88107350 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 -- 172.21.15.98:0/2770865858 >> 172.21.15.98:0/2770865858 conn(0x7ffb88077dc0 msgr2=0x7ffb880781d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 -- 172.21.15.98:0/2770865858 shutdown_connections 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 -- 172.21.15.98:0/2770865858 wait complete. 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 Processor -- start 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 -- start start 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8806a760 0x7ffb88198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb8806b120 0x7ffb88198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb88106f70 0x7ffb8819cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb88112220 con 0x7ffb8806a760 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffb881120a0 con 0x7ffb8806b120 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.172+0000 7ffb8eecf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffb881123a0 con 0x7ffb88106f70 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8cc6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8806a760 0x7ffb88198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8cc6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8806a760 0x7ffb88198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56208/0 (socket says 172.21.15.98:56208) 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8cc6b640 1 -- 172.21.15.98:0/3897323504 learned_addr learned my addr 172.21.15.98:0/3897323504 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb87fff640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb8806b120 0x7ffb88198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8d46c640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb88106f70 0x7ffb8819cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8cc6b640 1 -- 172.21.15.98:0/3897323504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb88106f70 msgr2=0x7ffb8819cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8cc6b640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb88106f70 0x7ffb8819cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8cc6b640 1 -- 172.21.15.98:0/3897323504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb8806b120 msgr2=0x7ffb88198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8cc6b640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb8806b120 0x7ffb88198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8cc6b640 1 -- 172.21.15.98:0/3897323504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8819d670 con 0x7ffb8806a760 2024-04-25T17:51:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8d46c640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb88106f70 0x7ffb8819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:51:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8cc6b640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8806a760 0x7ffb88198680 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ffb78006010 tx=0x7ffb780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb85ffb640 1 -- 172.21.15.98:0/3897323504 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb7803f030 con 0x7ffb8806a760 2024-04-25T17:51:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8eecf640 1 -- 172.21.15.98:0/3897323504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb8819d900 con 0x7ffb8806a760 2024-04-25T17:51:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb85ffb640 1 -- 172.21.15.98:0/3897323504 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb78003800 con 0x7ffb8806a760 2024-04-25T17:51:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb85ffb640 1 -- 172.21.15.98:0/3897323504 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb78035070 con 0x7ffb8806a760 2024-04-25T17:51:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.173+0000 7ffb8eecf640 1 -- 172.21.15.98:0/3897323504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb881a51f0 con 0x7ffb8806a760 2024-04-25T17:51:46.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.174+0000 7ffb8eecf640 1 -- 172.21.15.98:0/3897323504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb8804fce0 con 0x7ffb8806a760 2024-04-25T17:51:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.175+0000 7ffb85ffb640 1 -- 172.21.15.98:0/3897323504 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffb78003cb0 con 0x7ffb8806a760 2024-04-25T17:51:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.176+0000 7ffb85ffb640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb700a3dc0 0x7ffb700a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.176+0000 7ffb85ffb640 1 -- 172.21.15.98:0/3897323504 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7ffb780f1390 con 0x7ffb8806a760 2024-04-25T17:51:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.176+0000 7ffb87fff640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb700a3dc0 0x7ffb700a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.176+0000 7ffb87fff640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb700a3dc0 0x7ffb700a6280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffb88100f80 tx=0x7ffb7c00a2d0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:46.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.180+0000 7ffb85ffb640 1 -- 172.21.15.98:0/3897323504 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb780bda30 con 0x7ffb8806a760 2024-04-25T17:51:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.381+0000 7ffb8eecf640 1 -- 172.21.15.98:0/3897323504 --> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7ffb880ff740 con 0x7ffb700a3dc0 2024-04-25T17:51:46.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.389+0000 7ffb85ffb640 1 -- 172.21.15.98:0/3897323504 <== mgr.15129 v2:172.21.15.98:6800/844767972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7ffb880ff740 con 0x7ffb700a3dc0 2024-04-25T17:51:46.389 INFO:teuthology.orchestra.run.smithi098.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T17:35:55", "first": "2024-04-25T17:36:00", "last": "2024-04-25T17:51:00", "last_pruned": "2024-04-25T17:51:00", "created_count": 16, "pruned_count": 10, "active": true} 2024-04-25T17:51:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.391+0000 7ffb8eecf640 1 -- 172.21.15.98:0/3897323504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb700a3dc0 msgr2=0x7ffb700a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.391+0000 7ffb8eecf640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb700a3dc0 0x7ffb700a6280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffb88100f80 tx=0x7ffb7c00a2d0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.391+0000 7ffb8eecf640 1 -- 172.21.15.98:0/3897323504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8806a760 msgr2=0x7ffb88198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.391+0000 7ffb8eecf640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8806a760 0x7ffb88198680 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ffb78006010 tx=0x7ffb780042f0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.392+0000 7ffb8eecf640 1 -- 172.21.15.98:0/3897323504 shutdown_connections 2024-04-25T17:51:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.392+0000 7ffb8eecf640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7ffb700a3dc0 0x7ffb700a6280 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.392+0000 7ffb8eecf640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb88106f70 0x7ffb8819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.392+0000 7ffb8eecf640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb8806b120 0x7ffb88198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.392+0000 7ffb8eecf640 1 --2- 172.21.15.98:0/3897323504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb8806a760 0x7ffb88198680 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.392+0000 7ffb8eecf640 1 -- 172.21.15.98:0/3897323504 >> 172.21.15.98:0/3897323504 conn(0x7ffb88077dc0 msgr2=0x7ffb880ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.392+0000 7ffb8eecf640 1 -- 172.21.15.98:0/3897323504 shutdown_connections 2024-04-25T17:51:46.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.392+0000 7ffb8eecf640 1 -- 172.21.15.98:0/3897323504 wait complete. 2024-04-25T17:51:46.402 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T17:51:46.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 -- 172.21.15.98:0/2045317212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0381063a0 msgr2=0x7fb03810df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 --2- 172.21.15.98:0/2045317212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0381063a0 0x7fb03810df70 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fb030009ca0 tx=0x7fb03002a8c0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 -- 172.21.15.98:0/2045317212 shutdown_connections 2024-04-25T17:51:46.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 --2- 172.21.15.98:0/2045317212 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0381063a0 0x7fb03810df70 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 --2- 172.21.15.98:0/2045317212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0381021a0 0x7fb038105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 --2- 172.21.15.98:0/2045317212 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0381017f0 0x7fb038101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 -- 172.21.15.98:0/2045317212 >> 172.21.15.98:0/2045317212 conn(0x7fb038077390 msgr2=0x7fb0380fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 -- 172.21.15.98:0/2045317212 shutdown_connections 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 -- 172.21.15.98:0/2045317212 wait complete. 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 Processor -- start 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.504+0000 7fb03cf6f640 1 -- start start 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb03cf6f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0381017f0 0x7fb038199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb03cf6f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0381063a0 0x7fb038199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb03cf6f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb03819a1f0 0x7fb0381a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb03cf6f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb038111310 con 0x7fb03819a1f0 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb03cf6f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb038111190 con 0x7fb0381063a0 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb03cf6f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb038111490 con 0x7fb0381017f0 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb036d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb03819a1f0 0x7fb0381a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb036d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb03819a1f0 0x7fb0381a47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56232/0 (socket says 172.21.15.98:56232) 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb036d9d640 1 -- 172.21.15.98:0/1735940827 learned_addr learned my addr 172.21.15.98:0/1735940827 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb02ffff640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0381063a0 0x7fb038199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb03659c640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0381017f0 0x7fb038199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb036d9d640 1 -- 172.21.15.98:0/1735940827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0381017f0 msgr2=0x7fb038199770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb036d9d640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0381017f0 0x7fb038199770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb036d9d640 1 -- 172.21.15.98:0/1735940827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0381063a0 msgr2=0x7fb038199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb036d9d640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0381063a0 0x7fb038199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb036d9d640 1 -- 172.21.15.98:0/1735940827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0381a4f70 con 0x7fb03819a1f0 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb03659c640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0381017f0 0x7fb038199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb036d9d640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb03819a1f0 0x7fb0381a47f0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb030005cc0 tx=0x7fb03002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb02f7fe640 1 -- 172.21.15.98:0/1735940827 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb030003bf0 con 0x7fb03819a1f0 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb03cf6f640 1 -- 172.21.15.98:0/1735940827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0381a5200 con 0x7fb03819a1f0 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb02f7fe640 1 -- 172.21.15.98:0/1735940827 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb030003d90 con 0x7fb03819a1f0 2024-04-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb02f7fe640 1 -- 172.21.15.98:0/1735940827 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03002fbc0 con 0x7fb03819a1f0 2024-04-25T17:51:46.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.505+0000 7fb03cf6f640 1 -- 172.21.15.98:0/1735940827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0381a5750 con 0x7fb03819a1f0 2024-04-25T17:51:46.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.506+0000 7fb03cf6f640 1 -- 172.21.15.98:0/1735940827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb03804fce0 con 0x7fb03819a1f0 2024-04-25T17:51:46.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.507+0000 7fb02f7fe640 1 -- 172.21.15.98:0/1735940827 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb03002fd60 con 0x7fb03819a1f0 2024-04-25T17:51:46.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.508+0000 7fb02f7fe640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb0240a3dc0 0x7fb0240a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.508+0000 7fb03659c640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb0240a3dc0 0x7fb0240a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.508+0000 7fb02f7fe640 1 -- 172.21.15.98:0/1735940827 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fb0300f1e10 con 0x7fb03819a1f0 2024-04-25T17:51:46.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.508+0000 7fb03659c640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb0240a3dc0 0x7fb0240a6280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb028005fd0 tx=0x7fb028005ea0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:46.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.513+0000 7fb02f7fe640 1 -- 172.21.15.98:0/1735940827 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb030004610 con 0x7fb03819a1f0 2024-04-25T17:51:46.522 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:46 smithi173 ceph-mon[32856]: pgmap v500: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 25 MiB/s wr, 1.97k op/s 2024-04-25T17:51:46.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:46 smithi098 ceph-mon[29192]: pgmap v500: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 25 MiB/s wr, 1.97k op/s 2024-04-25T17:51:46.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:46 smithi203 ceph-mon[32834]: pgmap v500: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 25 MiB/s wr, 1.97k op/s 2024-04-25T17:51:46.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.714+0000 7fb03cf6f640 1 -- 172.21.15.98:0/1735940827 --> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fb03806a5d0 con 0x7fb0240a3dc0 2024-04-25T17:51:46.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.722+0000 7fb02f7fe640 1 -- 172.21.15.98:0/1735940827 <== mgr.15129 v2:172.21.15.98:6800/844767972 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fb03806a5d0 con 0x7fb0240a3dc0 2024-04-25T17:51:46.721 INFO:teuthology.orchestra.run.smithi098.stdout:/ 1m 6m3h 2024-04-25T17:51:46.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 -- 172.21.15.98:0/1735940827 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb0240a3dc0 msgr2=0x7fb0240a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb0240a3dc0 0x7fb0240a6280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb028005fd0 tx=0x7fb028005ea0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 -- 172.21.15.98:0/1735940827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb03819a1f0 msgr2=0x7fb0381a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb03819a1f0 0x7fb0381a47f0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb030005cc0 tx=0x7fb03002f380 comp rx=0 tx=0).stop 2024-04-25T17:51:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 -- 172.21.15.98:0/1735940827 shutdown_connections 2024-04-25T17:51:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fb0240a3dc0 0x7fb0240a6280 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb03819a1f0 0x7fb0381a47f0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0381063a0 0x7fb038199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 --2- 172.21.15.98:0/1735940827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0381017f0 0x7fb038199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 -- 172.21.15.98:0/1735940827 >> 172.21.15.98:0/1735940827 conn(0x7fb038077390 msgr2=0x7fb038104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 -- 172.21.15.98:0/1735940827 shutdown_connections 2024-04-25T17:51:46.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.724+0000 7fb03cf6f640 1 -- 172.21.15.98:0/1735940827 wait complete. 2024-04-25T17:51:46.734 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T17:51:46.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T17:51:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.845+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2537409970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f98107300 msgr2=0x7f7f9810eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.845+0000 7f7f9dbd8640 1 --2- 172.21.15.98:0/2537409970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f98107300 0x7f7f9810eeb0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7f90009ca0 tx=0x7f7f9002a8c0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.845+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2537409970 shutdown_connections 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.845+0000 7f7f9dbd8640 1 --2- 172.21.15.98:0/2537409970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f98107300 0x7f7f9810eeb0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.845+0000 7f7f9dbd8640 1 --2- 172.21.15.98:0/2537409970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f9806b2b0 0x7f7f98106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.845+0000 7f7f9dbd8640 1 --2- 172.21.15.98:0/2537409970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9806a900 0x7f7f9806ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.845+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2537409970 >> 172.21.15.98:0/2537409970 conn(0x7f7f98077f20 msgr2=0x7f7f98078330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.845+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2537409970 shutdown_connections 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.845+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2537409970 wait complete. 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f9dbd8640 1 Processor -- start 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f9dbd8640 1 -- start start 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f9dbd8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f9806a900 0x7f7f981986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f9dbd8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f98107300 0x7f7f98198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f9dbd8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9819cf90 0x7f7f9819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f9dbd8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f98112210 con 0x7f7f98107300 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f9dbd8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f98112090 con 0x7f7f9806a900 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f9dbd8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7f98112390 con 0x7f7f9819cf90 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f96ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f98107300 0x7f7f98198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f96ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f98107300 0x7f7f98198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56254/0 (socket says 172.21.15.98:56254) 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.846+0000 7f7f96ffd640 1 -- 172.21.15.98:0/2511749540 learned_addr learned my addr 172.21.15.98:0/2511749540 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f97fff640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9819cf90 0x7f7f9819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f977fe640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f9806a900 0x7f7f981986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f96ffd640 1 -- 172.21.15.98:0/2511749540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9819cf90 msgr2=0x7f7f9819d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f96ffd640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9819cf90 0x7f7f9819d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f96ffd640 1 -- 172.21.15.98:0/2511749540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f9806a900 msgr2=0x7f7f981986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f96ffd640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f9806a900 0x7f7f981986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f96ffd640 1 -- 172.21.15.98:0/2511749540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9819da80 con 0x7f7f98107300 2024-04-25T17:51:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f97fff640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9819cf90 0x7f7f9819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:51:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f96ffd640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f98107300 0x7f7f98198c10 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00c970 tx=0x7f7f8c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f94ff9640 1 -- 172.21.15.98:0/2511749540 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c007bb0 con 0x7f7f98107300 2024-04-25T17:51:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f94ff9640 1 -- 172.21.15.98:0/2511749540 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f8c007d50 con 0x7f7f98107300 2024-04-25T17:51:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2511749540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9819dd70 con 0x7f7f98107300 2024-04-25T17:51:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f94ff9640 1 -- 172.21.15.98:0/2511749540 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c00f450 con 0x7f7f98107300 2024-04-25T17:51:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.847+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2511749540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f981a55b0 con 0x7f7f98107300 2024-04-25T17:51:46.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.848+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2511749540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f980691c0 con 0x7f7f98107300 2024-04-25T17:51:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.849+0000 7f7f94ff9640 1 -- 172.21.15.98:0/2511749540 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7f8c004eb0 con 0x7f7f98107300 2024-04-25T17:51:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.850+0000 7f7f94ff9640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7f800a3da0 0x7f7f800a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.850+0000 7f7f977fe640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7f800a3da0 0x7f7f800a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:46.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.850+0000 7f7f94ff9640 1 -- 172.21.15.98:0/2511749540 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7f7f8c0d0980 con 0x7f7f98107300 2024-04-25T17:51:46.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.850+0000 7f7f977fe640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7f800a3da0 0x7f7f800a6260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7f88005fd0 tx=0x7f7f88005850 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:46.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:46.853+0000 7f7f94ff9640 1 -- 172.21.15.98:0/2511749540 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7f8c014030 con 0x7f7f98107300 2024-04-25T17:51:47.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.031+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2511749540 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f98199a20 con 0x7f7f98107300 2024-04-25T17:51:47.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.033+0000 7f7f94ff9640 1 -- 172.21.15.98:0/2511749540 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) ==== 74+0+23157 (secure 0 0 0) 0x7f7f98199a20 con 0x7f7f98107300 2024-04-25T17:51:47.032 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:47.032 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":114,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:51:15.193591+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T17:32:30.267079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T17:32:33.956902+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":27,"snap_epoch":113,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/4145487482":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/674729727":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/1530074164":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/2167465649":"2024-04-26T17:35:54.302654+0000","172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3206373024":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/4145487482":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":27,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:51:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2511749540 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7f800a3da0 msgr2=0x7f7f800a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7f800a3da0 0x7f7f800a6260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7f88005fd0 tx=0x7f7f88005850 comp rx=0 tx=0).stop 2024-04-25T17:51:47.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2511749540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f98107300 msgr2=0x7f7f98198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f98107300 0x7f7f98198c10 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00c970 tx=0x7f7f8c00ce30 comp rx=0 tx=0).stop 2024-04-25T17:51:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2511749540 shutdown_connections 2024-04-25T17:51:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f7f800a3da0 0x7f7f800a6260 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9819cf90 0x7f7f9819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f98107300 0x7f7f98198c10 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 --2- 172.21.15.98:0/2511749540 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f9806a900 0x7f7f981986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2511749540 >> 172.21.15.98:0/2511749540 conn(0x7f7f98077f20 msgr2=0x7f7f980fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2511749540 shutdown_connections 2024-04-25T17:51:47.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.035+0000 7f7f9dbd8640 1 -- 172.21.15.98:0/2511749540 wait complete. 2024-04-25T17:51:47.046 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T17:27:36.969831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T17:32:30.267079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T17:32:33.956902+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 113, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T17:51:47.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T17:51:47.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 -- 172.21.15.98:0/1386775065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f767810c510 msgr2=0x7f767810e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 --2- 172.21.15.98:0/1386775065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f767810c510 0x7f767810e9b0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7670009ca0 tx=0x7f767002a9e0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 -- 172.21.15.98:0/1386775065 shutdown_connections 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 --2- 172.21.15.98:0/1386775065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f767810c510 0x7f767810e9b0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 --2- 172.21.15.98:0/1386775065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7678103af0 0x7f767810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 --2- 172.21.15.98:0/1386775065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76781031d0 0x7f76781035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 -- 172.21.15.98:0/1386775065 >> 172.21.15.98:0/1386775065 conn(0x7f7678077f50 msgr2=0x7f7678078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 -- 172.21.15.98:0/1386775065 shutdown_connections 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 -- 172.21.15.98:0/1386775065 wait complete. 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 Processor -- start 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.149+0000 7f767f4cf640 1 -- start start 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767f4cf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76781031d0 0x7f7678100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767f4cf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7678103af0 0x7f7678101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767f4cf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f767810c510 0x7f76781018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767f4cf640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f76781121a0 con 0x7f76781031d0 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767f4cf640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7678112020 con 0x7f7678103af0 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767f4cf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7678112320 con 0x7f767810c510 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767d26b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76781031d0 0x7f7678100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767d26b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76781031d0 0x7f7678100e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56284/0 (socket says 172.21.15.98:56284) 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767d26b640 1 -- 172.21.15.98:0/1099781989 learned_addr learned my addr 172.21.15.98:0/1099781989 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767da6c640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f767810c510 0x7f76781018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767ca6a640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7678103af0 0x7f7678101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767d26b640 1 -- 172.21.15.98:0/1099781989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f767810c510 msgr2=0x7f76781018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767d26b640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f767810c510 0x7f76781018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767d26b640 1 -- 172.21.15.98:0/1099781989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7678103af0 msgr2=0x7f7678101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767d26b640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7678103af0 0x7f7678101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767d26b640 1 -- 172.21.15.98:0/1099781989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76781a6c00 con 0x7f76781031d0 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767ca6a640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7678103af0 0x7f7678101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:51:47.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f767d26b640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76781031d0 0x7f7678100e20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f767400e9f0 tx=0x7f767400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f766e7fc640 1 -- 172.21.15.98:0/1099781989 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767400cd80 con 0x7f76781031d0 2024-04-25T17:51:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.150+0000 7f766e7fc640 1 -- 172.21.15.98:0/1099781989 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76740054d0 con 0x7f76781031d0 2024-04-25T17:51:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.151+0000 7f767f4cf640 1 -- 172.21.15.98:0/1099781989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76781a6ef0 con 0x7f76781031d0 2024-04-25T17:51:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.151+0000 7f766e7fc640 1 -- 172.21.15.98:0/1099781989 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674010690 con 0x7f76781031d0 2024-04-25T17:51:47.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.151+0000 7f767f4cf640 1 -- 172.21.15.98:0/1099781989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76781a73b0 con 0x7f76781031d0 2024-04-25T17:51:47.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.152+0000 7f767f4cf640 1 -- 172.21.15.98:0/1099781989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7678068d70 con 0x7f76781031d0 2024-04-25T17:51:47.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.153+0000 7f766e7fc640 1 -- 172.21.15.98:0/1099781989 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7674004e90 con 0x7f76781031d0 2024-04-25T17:51:47.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.153+0000 7f766e7fc640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f76640a3da0 0x7f76640a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.153+0000 7f767ca6a640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f76640a3da0 0x7f76640a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.153+0000 7f766e7fc640 1 -- 172.21.15.98:0/1099781989 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7f76740d0c20 con 0x7f76781031d0 2024-04-25T17:51:47.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.154+0000 7f767ca6a640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f76640a3da0 0x7f76640a6260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f76781028b0 tx=0x7f7668005f90 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:47.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.158+0000 7f766e7fc640 1 -- 172.21.15.98:0/1099781989 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f767409d2c0 con 0x7f76781031d0 2024-04-25T17:51:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.340+0000 7f767f4cf640 1 -- 172.21.15.98:0/1099781989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f76781076c0 con 0x7f76781031d0 2024-04-25T17:51:47.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.341+0000 7f766e7fc640 1 -- 172.21.15.98:0/1099781989 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) ==== 93+0+10 (secure 0 0 0) 0x7f767409cbc0 con 0x7f76781031d0 2024-04-25T17:51:47.341 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T17:51:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 -- 172.21.15.98:0/1099781989 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f76640a3da0 msgr2=0x7f76640a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f76640a3da0 0x7f76640a6260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f76781028b0 tx=0x7f7668005f90 comp rx=0 tx=0).stop 2024-04-25T17:51:47.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 -- 172.21.15.98:0/1099781989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76781031d0 msgr2=0x7f7678100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76781031d0 0x7f7678100e20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f767400e9f0 tx=0x7f767400edf0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 -- 172.21.15.98:0/1099781989 shutdown_connections 2024-04-25T17:51:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f76640a3da0 0x7f76640a6260 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f767810c510 0x7f76781018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7678103af0 0x7f7678101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 --2- 172.21.15.98:0/1099781989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f76781031d0 0x7f7678100e20 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 -- 172.21.15.98:0/1099781989 >> 172.21.15.98:0/1099781989 conn(0x7f7678077f50 msgr2=0x7f76780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 -- 172.21.15.98:0/1099781989 shutdown_connections 2024-04-25T17:51:47.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.343+0000 7f767f4cf640 1 -- 172.21.15.98:0/1099781989 wait complete. 2024-04-25T17:51:47.352 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:47 smithi098 ceph-mon[29192]: from='client.16947 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:51:47.352 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:47 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2511749540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:51:47.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T17:51:47.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/4016238735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c84106180 msgr2=0x7f5c8410dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 --2- 172.21.15.98:0/4016238735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c84106180 0x7f5c8410dd50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c009ca0 tx=0x7f5c7c02a8c0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/4016238735 shutdown_connections 2024-04-25T17:51:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 --2- 172.21.15.98:0/4016238735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c84106180 0x7f5c8410dd50 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 --2- 172.21.15.98:0/4016238735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c84101a60 0x7f5c84105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 --2- 172.21.15.98:0/4016238735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c841010b0 0x7f5c84101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/4016238735 >> 172.21.15.98:0/4016238735 conn(0x7f5c840771d0 msgr2=0x7f5c840775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/4016238735 shutdown_connections 2024-04-25T17:51:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/4016238735 wait complete. 2024-04-25T17:51:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 Processor -- start 2024-04-25T17:51:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.456+0000 7f5c8a3f0640 1 -- start start 2024-04-25T17:51:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8a3f0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c841010b0 0x7f5c84102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8a3f0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c84106180 0x7f5c84103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8a3f0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c841035c0 0x7f5c841a6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8a3f0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5c841110d0 con 0x7f5c841035c0 2024-04-25T17:51:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8a3f0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5c84110f50 con 0x7f5c841010b0 2024-04-25T17:51:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8a3f0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5c84111250 con 0x7f5c84106180 2024-04-25T17:51:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8898d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c841035c0 0x7f5c841a6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8898d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c841035c0 0x7f5c841a6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56312/0 (socket says 172.21.15.98:56312) 2024-04-25T17:51:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8898d640 1 -- 172.21.15.98:0/2373037526 learned_addr learned my addr 172.21.15.98:0/2373037526 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c83fff640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c841010b0 0x7f5c84102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c837fe640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c84106180 0x7f5c84103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8898d640 1 -- 172.21.15.98:0/2373037526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c84106180 msgr2=0x7f5c84103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8898d640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c84106180 0x7f5c84103080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8898d640 1 -- 172.21.15.98:0/2373037526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c841010b0 msgr2=0x7f5c84102b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8898d640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c841010b0 0x7f5c84102b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8898d640 1 -- 172.21.15.98:0/2373037526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c841a6f10 con 0x7f5c841035c0 2024-04-25T17:51:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c837fe640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c84106180 0x7f5c84103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:51:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c8898d640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c841035c0 0x7f5c841a6810 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c005cc0 tx=0x7f5c7c02f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:47.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.457+0000 7f5c817fa640 1 -- 172.21.15.98:0/2373037526 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c003cf0 con 0x7f5c841035c0 2024-04-25T17:51:47.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.458+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/2373037526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c841a71a0 con 0x7f5c841035c0 2024-04-25T17:51:47.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.458+0000 7f5c817fa640 1 -- 172.21.15.98:0/2373037526 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c7c0044f0 con 0x7f5c841035c0 2024-04-25T17:51:47.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.458+0000 7f5c817fa640 1 -- 172.21.15.98:0/2373037526 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c02fc30 con 0x7f5c841035c0 2024-04-25T17:51:47.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.458+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/2373037526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c841a7740 con 0x7f5c841035c0 2024-04-25T17:51:47.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.459+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/2373037526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c84102580 con 0x7f5c841035c0 2024-04-25T17:51:47.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.460+0000 7f5c817fa640 1 -- 172.21.15.98:0/2373037526 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5c7c0046a0 con 0x7f5c841035c0 2024-04-25T17:51:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.460+0000 7f5c817fa640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5c6c0a3d50 0x7f5c6c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.461+0000 7f5c83fff640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5c6c0a3d50 0x7f5c6c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.461+0000 7f5c817fa640 1 -- 172.21.15.98:0/2373037526 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7f5c7c0f1d90 con 0x7f5c841035c0 2024-04-25T17:51:47.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.461+0000 7f5c83fff640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5c6c0a3d50 0x7f5c6c0a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400abe0 tx=0x7f5c7400a2d0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:47.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.465+0000 7f5c817fa640 1 -- 172.21.15.98:0/2373037526 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c7c0be540 con 0x7f5c841035c0 2024-04-25T17:51:47.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.641+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/2373037526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5c8410a3c0 con 0x7f5c841035c0 2024-04-25T17:51:47.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.641+0000 7f5c817fa640 1 -- 172.21.15.98:0/2373037526 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) ==== 104+0+11 (secure 0 0 0) 0x7f5c7c0bde40 con 0x7f5c841035c0 2024-04-25T17:51:47.641 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T17:51:47.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/2373037526 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5c6c0a3d50 msgr2=0x7f5c6c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5c6c0a3d50 0x7f5c6c0a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400abe0 tx=0x7f5c7400a2d0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/2373037526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c841035c0 msgr2=0x7f5c841a6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c841035c0 0x7f5c841a6810 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c005cc0 tx=0x7f5c7c02f470 comp rx=0 tx=0).stop 2024-04-25T17:51:47.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/2373037526 shutdown_connections 2024-04-25T17:51:47.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5c6c0a3d50 0x7f5c6c0a6210 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c841035c0 0x7f5c841a6810 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c84106180 0x7f5c84103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 --2- 172.21.15.98:0/2373037526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c841010b0 0x7f5c84102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/2373037526 >> 172.21.15.98:0/2373037526 conn(0x7f5c840771d0 msgr2=0x7f5c8406a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:47.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/2373037526 shutdown_connections 2024-04-25T17:51:47.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.643+0000 7f5c8a3f0640 1 -- 172.21.15.98:0/2373037526 wait complete. 2024-04-25T17:51:47.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T17:51:47.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:47 smithi203 ceph-mon[32834]: from='client.16947 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:51:47.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:47 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2511749540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:51:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.758+0000 7f7521f5d640 1 -- 172.21.15.98:0/280848454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f751c101a40 msgr2=0x7f751c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.758+0000 7f7521f5d640 1 --2- 172.21.15.98:0/280848454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f751c101a40 0x7f751c105a80 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7510009900 tx=0x7f751002a410 comp rx=0 tx=0).stop 2024-04-25T17:51:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.758+0000 7f7521f5d640 1 -- 172.21.15.98:0/280848454 shutdown_connections 2024-04-25T17:51:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.758+0000 7f7521f5d640 1 --2- 172.21.15.98:0/280848454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c106160 0x7f751c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.758+0000 7f7521f5d640 1 --2- 172.21.15.98:0/280848454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f751c101a40 0x7f751c105a80 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.758+0000 7f7521f5d640 1 --2- 172.21.15.98:0/280848454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f751c101090 0x7f751c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.758+0000 7f7521f5d640 1 -- 172.21.15.98:0/280848454 >> 172.21.15.98:0/280848454 conn(0x7f751c077f50 msgr2=0x7f751c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.758+0000 7f7521f5d640 1 -- 172.21.15.98:0/280848454 shutdown_connections 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.758+0000 7f7521f5d640 1 -- 172.21.15.98:0/280848454 wait complete. 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.758+0000 7f7521f5d640 1 Processor -- start 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f7521f5d640 1 -- start start 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f7521f5d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c101090 0x7f751c197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f7521f5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f751c106160 0x7f751c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f7521f5d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f751c19bf00 0x7f751c19c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f7521f5d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f751c111080 con 0x7f751c106160 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f7521f5d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f751c110f00 con 0x7f751c19bf00 2024-04-25T17:51:47.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f7521f5d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f751c111200 con 0x7f751c101090 2024-04-25T17:51:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f751c106160 0x7f751c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f751c106160 0x7f751c197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56342/0 (socket says 172.21.15.98:56342) 2024-04-25T17:51:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751affd640 1 -- 172.21.15.98:0/1754583581 learned_addr learned my addr 172.21.15.98:0/1754583581 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751b7fe640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c101090 0x7f751c197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751bfff640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f751c19bf00 0x7f751c19c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751affd640 1 -- 172.21.15.98:0/1754583581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c101090 msgr2=0x7f751c197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751affd640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c101090 0x7f751c197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751affd640 1 -- 172.21.15.98:0/1754583581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f751c19bf00 msgr2=0x7f751c19c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751affd640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f751c19bf00 0x7f751c19c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751affd640 1 -- 172.21.15.98:0/1754583581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f751c19c9f0 con 0x7f751c106160 2024-04-25T17:51:47.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751bfff640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f751c19bf00 0x7f751c19c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:51:47.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.759+0000 7f751affd640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f751c106160 0x7f751c197b80 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7510032fd0 tx=0x7f7510004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:47.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.760+0000 7f7518ff9640 1 -- 172.21.15.98:0/1754583581 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751003f030 con 0x7f751c106160 2024-04-25T17:51:47.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.760+0000 7f7521f5d640 1 -- 172.21.15.98:0/1754583581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f751c19cc80 con 0x7f751c106160 2024-04-25T17:51:47.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.760+0000 7f7518ff9640 1 -- 172.21.15.98:0/1754583581 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7510004490 con 0x7f751c106160 2024-04-25T17:51:47.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.760+0000 7f7518ff9640 1 -- 172.21.15.98:0/1754583581 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7510035070 con 0x7f751c106160 2024-04-25T17:51:47.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.760+0000 7f7521f5d640 1 -- 172.21.15.98:0/1754583581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f751c1a4450 con 0x7f751c106160 2024-04-25T17:51:47.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.761+0000 7f7521f5d640 1 -- 172.21.15.98:0/1754583581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f751c068d70 con 0x7f751c106160 2024-04-25T17:51:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.762+0000 7f7518ff9640 1 -- 172.21.15.98:0/1754583581 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7510042030 con 0x7f751c106160 2024-04-25T17:51:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.762+0000 7f7518ff9640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75040a3d20 0x7f75040a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.763+0000 7f7518ff9640 1 -- 172.21.15.98:0/1754583581 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7f75100f1100 con 0x7f751c106160 2024-04-25T17:51:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.763+0000 7f751b7fe640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75040a3d20 0x7f75040a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:47.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.763+0000 7f751b7fe640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75040a3d20 0x7f75040a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f750c0058c0 tx=0x7f750c00a2d0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:47.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.766+0000 7f7518ff9640 1 -- 172.21.15.98:0/1754583581 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75100bd7a0 con 0x7f751c106160 2024-04-25T17:51:47.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:47 smithi173 ceph-mon[32856]: from='client.16947 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:51:47.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:47 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2511749540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:51:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.946+0000 7f7521f5d640 1 -- 172.21.15.98:0/1754583581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f751c101a40 con 0x7f751c106160 2024-04-25T17:51:47.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.947+0000 7f7518ff9640 1 -- 172.21.15.98:0/1754583581 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) ==== 100+0+11 (secure 0 0 0) 0x7f75100bd0a0 con 0x7f751c106160 2024-04-25T17:51:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T17:51:47.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.948+0000 7f7521f5d640 1 -- 172.21.15.98:0/1754583581 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75040a3d20 msgr2=0x7f75040a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.948+0000 7f7521f5d640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75040a3d20 0x7f75040a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f750c0058c0 tx=0x7f750c00a2d0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.948+0000 7f7521f5d640 1 -- 172.21.15.98:0/1754583581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f751c106160 msgr2=0x7f751c197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:47.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.948+0000 7f7521f5d640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f751c106160 0x7f751c197b80 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f7510032fd0 tx=0x7f7510004270 comp rx=0 tx=0).stop 2024-04-25T17:51:47.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.948+0000 7f7521f5d640 1 -- 172.21.15.98:0/1754583581 shutdown_connections 2024-04-25T17:51:47.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.948+0000 7f7521f5d640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f75040a3d20 0x7f75040a61e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.948+0000 7f7521f5d640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f751c19bf00 0x7f751c19c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.948+0000 7f7521f5d640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f751c106160 0x7f751c197b80 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.948+0000 7f7521f5d640 1 --2- 172.21.15.98:0/1754583581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f751c101090 0x7f751c197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:47.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.948+0000 7f7521f5d640 1 -- 172.21.15.98:0/1754583581 >> 172.21.15.98:0/1754583581 conn(0x7f751c077f50 msgr2=0x7f751c0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:47.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.949+0000 7f7521f5d640 1 -- 172.21.15.98:0/1754583581 shutdown_connections 2024-04-25T17:51:47.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:47.949+0000 7f7521f5d640 1 -- 172.21.15.98:0/1754583581 wait complete. 2024-04-25T17:51:47.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T17:51:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.057+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4112930849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb2c1004c0 msgr2=0x7fcb2c100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.057+0000 7fcb32fcc640 1 --2- 172.21.15.98:0/4112930849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb2c1004c0 0x7fcb2c100900 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcb18009930 tx=0x7fcb1802a420 comp rx=0 tx=0).stop 2024-04-25T17:51:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.057+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4112930849 shutdown_connections 2024-04-25T17:51:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.057+0000 7fcb32fcc640 1 --2- 172.21.15.98:0/4112930849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb2c100e40 0x7fcb2c10ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.057+0000 7fcb32fcc640 1 --2- 172.21.15.98:0/4112930849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb2c1004c0 0x7fcb2c100900 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:48.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.057+0000 7fcb32fcc640 1 --2- 172.21.15.98:0/4112930849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb2c1080d0 0x7fcb2c1084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.057+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4112930849 >> 172.21.15.98:0/4112930849 conn(0x7fcb2c077480 msgr2=0x7fcb2c0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.057+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4112930849 shutdown_connections 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.057+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4112930849 wait complete. 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb32fcc640 1 Processor -- start 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb32fcc640 1 -- start start 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb32fcc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb2c1004c0 0x7fcb2c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb32fcc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb2c100e40 0x7fcb2c19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb32fcc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb2c1080d0 0x7fcb2c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb32fcc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c112290 con 0x7fcb2c1004c0 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb32fcc640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c112110 con 0x7fcb2c1080d0 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb32fcc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcb2c112410 con 0x7fcb2c100e40 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb30d68640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb2c1004c0 0x7fcb2c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb30d68640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb2c1004c0 0x7fcb2c19e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56362/0 (socket says 172.21.15.98:56362) 2024-04-25T17:51:48.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb30d68640 1 -- 172.21.15.98:0/4274473264 learned_addr learned my addr 172.21.15.98:0/4274473264 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb31569640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb2c1080d0 0x7fcb2c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb30d68640 1 -- 172.21.15.98:0/4274473264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb2c100e40 msgr2=0x7fcb2c19eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb2bfff640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb2c100e40 0x7fcb2c19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb30d68640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb2c100e40 0x7fcb2c19eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb30d68640 1 -- 172.21.15.98:0/4274473264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb2c1080d0 msgr2=0x7fcb2c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb30d68640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb2c1080d0 0x7fcb2c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb30d68640 1 -- 172.21.15.98:0/4274473264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb2c198f50 con 0x7fcb2c1004c0 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb31569640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb2c1080d0 0x7fcb2c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.058+0000 7fcb30d68640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb2c1004c0 0x7fcb2c19e5d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000d770 tx=0x7fcb2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.059+0000 7fcb29ffb640 1 -- 172.21.15.98:0/4274473264 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb20017070 con 0x7fcb2c1004c0 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.059+0000 7fcb29ffb640 1 -- 172.21.15.98:0/4274473264 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb200054d0 con 0x7fcb2c1004c0 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.059+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4274473264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb2c199240 con 0x7fcb2c1004c0 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.059+0000 7fcb29ffb640 1 -- 172.21.15.98:0/4274473264 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb20004c30 con 0x7fcb2c1004c0 2024-04-25T17:51:48.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.059+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4274473264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb2c1a5550 con 0x7fcb2c1004c0 2024-04-25T17:51:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.060+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4274473264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb2c1037e0 con 0x7fcb2c1004c0 2024-04-25T17:51:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.061+0000 7fcb29ffb640 1 -- 172.21.15.98:0/4274473264 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcb20004e10 con 0x7fcb2c1004c0 2024-04-25T17:51:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.061+0000 7fcb29ffb640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcb1c0a3d20 0x7fcb1c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:51:48.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.061+0000 7fcb2bfff640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcb1c0a3d20 0x7fcb1c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:51:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.061+0000 7fcb29ffb640 1 -- 172.21.15.98:0/4274473264 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8513+0+0 (secure 0 0 0) 0x7fcb200d0310 con 0x7fcb2c1004c0 2024-04-25T17:51:48.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.062+0000 7fcb2bfff640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcb1c0a3d20 0x7fcb1c0a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb18006010 tx=0x7fcb18000bc0 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:51:48.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.066+0000 7fcb29ffb640 1 -- 172.21.15.98:0/4274473264 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb2009c9b0 con 0x7fcb2c1004c0 2024-04-25T17:51:48.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.294+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4274473264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fcb2c10a4e0 con 0x7fcb2c1004c0 2024-04-25T17:51:48.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.301+0000 7fcb29ffb640 1 -- 172.21.15.98:0/4274473264 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746058 (secure 0 0 0) 0x7fcb20096e20 con 0x7fcb2c1004c0 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 26, 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: "active_gid": 15129, 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: "active_name": "x", 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addrs": { 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T17:51:48.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6800", 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 844767972 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6801", 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 844767972 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addr": "172.21.15.98:6801/844767972", 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: "active_change": "2024-04-25T17:35:54.302851+0000", 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T17:51:48.312 INFO:teuthology.orchestra.run.smithi098.stdout: "available": true, 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "standbys": [ 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24733, 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "z", 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540701547738038271, 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:48.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.316 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.317 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.319 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.320 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.321 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.322 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.323 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.324 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.325 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-04-25T17:51:48.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T17:51:48.328 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T17:51:48.329 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T17:51:48.330 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T17:51:48.331 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T17:51:48.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.333 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T17:51:48.334 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.335 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.337 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.339 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.340 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T17:51:48.341 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-04-25T17:51:48.342 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.343 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:48.344 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.345 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.346 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.347 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.349 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-04-25T17:51:48.352 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.353 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.354 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.355 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T17:51:48.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T17:51:48.358 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.360 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.362 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.364 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.365 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.372 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.377 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-04-25T17:51:48.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-04-25T17:51:48.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-04-25T17:51:48.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.382 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.385 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.387 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.388 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.389 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.390 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.391 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T17:51:48.392 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-04-25T17:51:48.394 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.395 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T17:51:48.396 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.397 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.398 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T17:51:48.400 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.403 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.405 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.406 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-04-25T17:51:48.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.409 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.412 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.413 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T17:51:48.414 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.417 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.423 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.424 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.425 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.430 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.433 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.434 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.435 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.436 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "[]", 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.438 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "{}", 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.445 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T17:51:48.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T17:51:48.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T17:51:48.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T17:51:48.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T17:51:48.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T17:51:48.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T17:51:48.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.460 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.465 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.467 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.469 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.470 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.472 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T17:51:48.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.474 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.477 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.477 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.477 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.477 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.477 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.477 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.477 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-04-25T17:51:48.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.478 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.479 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.482 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.483 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.486 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T17:51:48.489 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.490 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T17:51:48.491 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T17:51:48.492 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.495 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.496 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.497 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.498 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.500 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.501 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.502 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-04-25T17:51:48.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-04-25T17:51:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:48.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.509 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.510 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.511 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.512 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-04-25T17:51:48.514 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.515 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-04-25T17:51:48.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.518 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.519 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.524 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.526 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.527 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-04-25T17:51:48.529 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-04-25T17:51:48.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.532 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-04-25T17:51:48.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.538 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.539 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.540 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.543 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-04-25T17:51:48.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-04-25T17:51:48.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-04-25T17:51:48.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.548 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.549 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.553 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.556 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.560 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.561 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.563 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.564 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.565 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.567 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.568 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-04-25T17:51:48.570 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.571 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.572 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.573 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.574 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.575 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.576 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.578 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.584 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.585 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T17:51:48.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.588 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.590 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.593 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.594 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.595 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-04-25T17:51:48.596 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-04-25T17:51:48.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-04-25T17:51:48.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.600 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.602 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.603 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.605 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-04-25T17:51:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.609 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.610 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.612 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.613 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.615 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-04-25T17:51:48.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T17:51:48.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.619 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.620 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.621 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.622 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.623 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.624 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.625 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.626 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.627 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.628 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.629 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.630 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.631 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.632 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.633 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.636 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.637 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.638 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.642 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.646 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.647 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.648 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.649 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.650 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.651 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.652 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.654 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-04-25T17:51:48.655 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-04-25T17:51:48.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.657 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-04-25T17:51:48.658 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.659 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.661 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.662 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.663 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.666 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.667 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.669 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.670 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.671 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.672 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.674 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-04-25T17:51:48.676 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.677 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.678 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.679 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.680 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.684 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.685 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.686 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.687 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.688 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.689 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.690 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.691 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.692 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.694 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.696 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.697 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.698 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.699 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.700 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.701 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.703 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T17:51:48.704 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.704 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.704 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.704 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.704 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.704 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.706 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.708 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.709 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.710 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.711 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.712 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.713 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.714 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-04-25T17:51:48.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-04-25T17:51:48.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.717 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.718 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.720 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.721 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.723 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.725 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-04-25T17:51:48.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-04-25T17:51:48.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.729 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:48.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24773, 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "y", 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540701547738038271, 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.731 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.732 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.733 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.735 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.736 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-04-25T17:51:48.737 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-04-25T17:51:48.738 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.740 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.741 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.743 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.745 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.746 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T17:51:48.747 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.751 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.753 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.754 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.755 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.757 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.758 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.759 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.764 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.765 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.766 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.767 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.768 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T17:51:48.769 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-04-25T17:51:48.770 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.771 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.772 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T17:51:48.773 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:48.774 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.775 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.776 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.777 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.778 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.779 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.780 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.782 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-04-25T17:51:48.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-04-25T17:51:48.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-04-25T17:51:48.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.786 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.788 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.789 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.790 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-04-25T17:51:48.791 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.792 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.793 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.794 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-04-25T17:51:48.795 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.797 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.798 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.799 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.800 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.801 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.802 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.803 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.804 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T17:51:48.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T17:51:48.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-04-25T17:51:48.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.808 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-04-25T17:51:48.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-04-25T17:51:48.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-04-25T17:51:48.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-04-25T17:51:48.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.815 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.817 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-04-25T17:51:48.818 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.819 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.820 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T17:51:48.821 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T17:51:48.822 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T17:51:48.823 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.824 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.825 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.826 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.828 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.829 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.830 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.831 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T17:51:48.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T17:51:48.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.834 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.835 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.836 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.837 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.839 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.840 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.841 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.842 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.843 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.844 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.845 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.847 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.848 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.850 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.851 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.853 INFO:teuthology.orchestra.run.smithi098.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "[]", 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.854 INFO:teuthology.orchestra.run.smithi098.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "{}", 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T17:51:48.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T17:51:48.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.857 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.858 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.860 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.861 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.863 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.864 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.865 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.866 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.867 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.868 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.869 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.870 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.871 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.872 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.875 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.876 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.879 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-04-25T17:51:48.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T17:51:48.881 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.883 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.884 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.885 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.886 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.889 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.889 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.889 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.889 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-04-25T17:51:48.889 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.891 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.892 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.895 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.898 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.899 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.900 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.902 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.903 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T17:51:48.904 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.905 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.906 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.907 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.908 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.909 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.910 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.911 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.914 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.915 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.916 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.917 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.918 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-04-25T17:51:48.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.922 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.923 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.924 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.925 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.926 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.927 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-04-25T17:51:48.928 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.929 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.930 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.933 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.934 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.935 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.936 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.938 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.939 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.940 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.941 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.942 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.943 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.944 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.945 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.947 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.949 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.950 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.952 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.954 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.954 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.954 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.954 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.954 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.954 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.955 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.956 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.957 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T17:51:48.958 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.959 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.960 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.964 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.966 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.967 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.968 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.970 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.971 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.972 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.973 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.974 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.975 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.976 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T17:51:48.979 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.980 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.981 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.982 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.984 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.985 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.987 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.989 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.990 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.991 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.992 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.993 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.994 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-04-25T17:51:48.996 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:48.997 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:48.999 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.000 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.001 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.003 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.005 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.006 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.007 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.008 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.009 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.010 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.011 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.012 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-04-25T17:51:49.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-04-25T17:51:49.014 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.016 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.018 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.019 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.020 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-04-25T17:51:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.023 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.024 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.025 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-04-25T17:51:49.026 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T17:51:49.027 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.029 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.030 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.031 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.032 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:49.033 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.034 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.035 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-04-25T17:51:49.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T17:51:49.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.039 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.040 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.041 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.042 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.043 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.044 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.046 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.047 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.048 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-04-25T17:51:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.051 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.052 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.053 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.054 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-04-25T17:51:49.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.057 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.058 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.059 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.060 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-04-25T17:51:49.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-04-25T17:51:49.064 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-04-25T17:51:49.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-04-25T17:51:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.067 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.068 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.069 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.070 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-04-25T17:51:49.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-04-25T17:51:49.074 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.076 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.077 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.078 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.079 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.080 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.081 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.082 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.083 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.084 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.085 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.086 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.087 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.088 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-04-25T17:51:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-04-25T17:51:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-04-25T17:51:49.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T17:51:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.093 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.094 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.095 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.098 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.099 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.100 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.102 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.103 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.104 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.105 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.106 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.107 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.108 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.109 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.110 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.113 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.114 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.115 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.116 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-04-25T17:51:49.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-04-25T17:51:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-04-25T17:51:49.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.121 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.122 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.124 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.126 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.129 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.130 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.132 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.134 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-04-25T17:51:49.135 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.136 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.137 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.138 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.141 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.142 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.142 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.142 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.142 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.142 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.142 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.142 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.144 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-04-25T17:51:49.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: "modules": [ 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard", 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: "iostat", 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: "nfs", 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: "restful", 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: "snap_schedule" 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-04-25T17:51:49.147 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.148 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.149 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.150 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.151 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.152 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-04-25T17:51:49.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-04-25T17:51:49.154 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.155 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.156 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.157 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.158 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.159 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.160 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-04-25T17:51:49.161 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-04-25T17:51:49.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-04-25T17:51:49.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:49.164 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.165 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.166 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.167 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:49.168 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.169 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.170 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.172 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.173 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T17:51:49.174 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-04-25T17:51:49.175 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.176 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:49.177 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:49.178 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.179 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-04-25T17:51:49.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-04-25T17:51:49.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-04-25T17:51:49.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.183 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.184 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.185 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.186 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-04-25T17:51:49.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-04-25T17:51:49.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.189 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.190 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.191 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.193 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T17:51:49.193 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.193 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.193 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.193 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-04-25T17:51:49.193 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.193 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.193 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.193 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-04-25T17:51:49.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-04-25T17:51:49.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.194 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.194 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T17:51:49.194 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.194 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.195 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.195 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:49.195 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.195 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.195 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.195 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-04-25T17:51:49.195 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-04-25T17:51:49.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.196 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.196 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T17:51:49.196 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.196 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.197 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.198 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.199 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-04-25T17:51:49.200 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-04-25T17:51:49.201 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:49.202 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.203 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.204 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.207 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.208 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-04-25T17:51:49.209 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-04-25T17:51:49.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.211 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.213 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T17:51:49.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:49.215 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.216 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.218 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-04-25T17:51:49.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.220 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.221 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.222 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.224 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-04-25T17:51:49.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-04-25T17:51:49.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.227 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-04-25T17:51:49.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-04-25T17:51:49.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.231 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.232 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.233 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.234 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.235 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.236 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.237 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T17:51:49.238 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T17:51:49.239 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T17:51:49.240 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.242 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.243 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.245 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.246 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.247 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.249 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T17:51:49.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.251 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.255 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.256 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.257 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.258 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.259 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.261 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.264 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-04-25T17:51:49.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T17:51:49.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T17:51:49.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.269 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.270 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "[]", 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "{}", 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.272 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.273 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.275 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.276 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.279 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T17:51:49.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.281 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-04-25T17:51:49.284 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.286 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.287 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.288 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.289 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.290 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.291 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.292 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.294 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-04-25T17:51:49.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-04-25T17:51:49.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.294 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.294 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.294 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.294 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.294 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.294 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.298 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.298 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.298 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.298 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.298 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-04-25T17:51:49.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-04-25T17:51:49.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-04-25T17:51:49.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.300 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.301 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.303 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.305 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.306 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.307 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.308 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.309 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.310 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.311 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.312 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.313 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.314 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.315 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.316 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.317 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.319 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.320 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.321 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T17:51:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T17:51:49.323 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.324 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.325 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.326 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.328 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.330 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.331 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.332 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.333 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T17:51:49.334 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.335 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.336 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.337 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.338 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.339 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.340 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.342 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.345 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.352 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.353 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.354 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.355 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.356 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.357 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.358 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.359 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.360 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.362 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-04-25T17:51:49.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.367 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.368 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-04-25T17:51:49.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.370 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.372 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.373 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.375 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-04-25T17:51:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-04-25T17:51:49.377 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.379 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.382 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.383 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.384 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.387 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.388 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.389 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.390 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.391 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.392 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.393 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.396 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.397 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.398 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.400 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.401 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.403 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.404 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.408 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.411 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.415 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.416 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.417 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.418 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.419 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-04-25T17:51:49.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T17:51:49.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.423 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.424 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.425 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.427 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-04-25T17:51:49.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-04-25T17:51:49.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.434 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.436 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.438 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T17:51:49.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.441 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.447 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.448 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.449 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.451 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.459 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.460 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.461 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.462 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.464 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.465 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.467 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.468 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.469 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.471 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.472 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.473 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.474 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.475 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.477 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.478 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.479 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.482 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.483 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.484 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-04-25T17:51:49.485 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.486 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.487 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.490 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.494 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.495 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.498 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.499 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.501 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.501 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.501 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.503 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.504 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.506 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.507 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.510 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.511 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.512 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.514 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.516 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T17:51:49.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.518 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T17:51:49.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.524 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-04-25T17:51:49.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-04-25T17:51:49.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-04-25T17:51:49.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.528 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.530 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.531 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.535 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.538 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.539 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.541 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.542 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.544 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-04-25T17:51:49.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.548 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T17:51:49.549 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T17:51:49.550 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.551 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.552 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T17:51:49.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T17:51:49.556 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T17:51:49.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-04-25T17:51:49.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T17:51:49.559 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T17:51:49.560 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: "services": { 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard": "https://172.21.15.98:8443/" 2024-04-25T17:51:49.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "always_on_modules": { 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus": [ 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.562 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific": [ 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy": [ 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-04-25T17:51:49.563 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "reef": [ 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-04-25T17:51:49.564 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "squid": [ 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-04-25T17:51:49.565 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: "last_failure_osd_epoch": 92, 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: "active_clients": [ 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 592347275 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.566 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "libcephsqlite", 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3363371338 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T17:51:49.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1580606063 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 785238152 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.568 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3455529295 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T17:51:49.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T17:51:49.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T17:51:49.570 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-04-25T17:51:49.570 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2798126155 2024-04-25T17:51:49.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.570 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:49.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:49.570 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T17:51:49.570 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:51:49.571 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:48 smithi173 ceph-mon[32856]: pgmap v501: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 25 MiB/s wr, 1.98k op/s 2024-04-25T17:51:49.571 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:48 smithi173 ceph-mon[32856]: from='client.16953 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:51:49.572 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:48 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1099781989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T17:51:49.572 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:48 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2373037526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T17:51:49.572 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:48 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1754583581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T17:51:49.572 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:48 smithi203 ceph-mon[32834]: pgmap v501: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 25 MiB/s wr, 1.98k op/s 2024-04-25T17:51:49.572 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:48 smithi203 ceph-mon[32834]: from='client.16953 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:51:49.572 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:48 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1099781989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T17:51:49.572 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:48 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2373037526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T17:51:49.572 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:48 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1754583581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T17:51:49.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4274473264 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcb1c0a3d20 msgr2=0x7fcb1c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:49.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcb1c0a3d20 0x7fcb1c0a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb18006010 tx=0x7fcb18000bc0 comp rx=0 tx=0).stop 2024-04-25T17:51:49.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4274473264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb2c1004c0 msgr2=0x7fcb2c19e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:51:49.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb2c1004c0 0x7fcb2c19e5d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcb2000d770 tx=0x7fcb2000dc30 comp rx=0 tx=0).stop 2024-04-25T17:51:49.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4274473264 shutdown_connections 2024-04-25T17:51:49.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fcb1c0a3d20 0x7fcb1c0a61e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:49.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb2c1080d0 0x7fcb2c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:49.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb2c100e40 0x7fcb2c19eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:49.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 --2- 172.21.15.98:0/4274473264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb2c1004c0 0x7fcb2c19e5d0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:51:49.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4274473264 >> 172.21.15.98:0/4274473264 conn(0x7fcb2c077480 msgr2=0x7fcb2c105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:51:49.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.306+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4274473264 shutdown_connections 2024-04-25T17:51:49.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:51:48.307+0000 7fcb32fcc640 1 -- 172.21.15.98:0/4274473264 wait complete. 2024-04-25T17:51:49.580 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T17:51:49.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T17:51:49.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:48 smithi098 ceph-mon[29192]: pgmap v501: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 25 MiB/s wr, 1.98k op/s 2024-04-25T17:51:49.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:48 smithi098 ceph-mon[29192]: from='client.16953 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T17:51:49.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:48 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1099781989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T17:51:49.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:48 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2373037526' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T17:51:49.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:48 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1754583581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T17:51:49.606 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:49 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4274473264' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T17:51:49.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:49 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/4274473264' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T17:51:49.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:49 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/4274473264' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T17:51:49.830 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:51:50.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:50 smithi098 ceph-mon[29192]: pgmap v502: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 6.6 MiB/s wr, 599 op/s 2024-04-25T17:51:50.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:50 smithi203 ceph-mon[32834]: pgmap v502: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 6.6 MiB/s wr, 599 op/s 2024-04-25T17:51:50.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:50 smithi173 ceph-mon[32856]: pgmap v502: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 6.6 MiB/s wr, 599 op/s 2024-04-25T17:51:51.577 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:51:51.577 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:51:51.577 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 4, 2024-04-25T17:51:51.577 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 4, 2024-04-25T17:51:51.577 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 2944, 2024-04-25T17:51:51.577 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 448, 2024-04-25T17:51:51.577 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001572081, 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000039027, 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000540113, 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3, 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000679707, 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000226569 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:51.578 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 3, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 241, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 480, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 276656, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 6, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.017198489, 2024-04-25T17:51:51.579 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.011964131, 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.002288037, 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 238, 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.013231847, 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000055595 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 276656 2024-04-25T17:51:51.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 324, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 313, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1618864, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 193200, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.052170252, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.018777819, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.033927884, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.006520885, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 312, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.015742563, 2024-04-25T17:51:51.581 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000050456 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1618864, 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 193200 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.582 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:51:51.583 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:51:51.584 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 147947, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 29, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-04-25T17:51:51.585 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:51.586 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 327, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1573152, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 327, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 327, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1573152, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:51.587 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:51.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:51.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:51.588 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:51:52.199 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-25T17:51:52.199 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-25T17:51:52.199 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-25T17:51:52.199 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-25T17:51:52.199 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-25T17:51:52.199 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T17:51:52.200 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T17:51:52.483 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:51:52.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:52 smithi098 ceph-mon[29192]: pgmap v503: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 3.1 MiB/s wr, 336 op/s 2024-04-25T17:51:52.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:52 smithi203 ceph-mon[32834]: pgmap v503: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 3.1 MiB/s wr, 336 op/s 2024-04-25T17:51:52.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:52 smithi173 ceph-mon[32856]: pgmap v503: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 3.1 MiB/s wr, 336 op/s 2024-04-25T17:51:54.124 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:51:54.124 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001083033, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000240433, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:54.125 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 335, 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 322, 2024-04-25T17:51:54.126 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1623712, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 198128, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.055452914, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.019900658, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.036437157, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.006991362, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 322, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.015419844, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000047887 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:54.127 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1623712, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 198128 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 3, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 245, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 480, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 277360, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 6, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.018126091, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.012879198, 2024-04-25T17:51:54.128 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.002182606, 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 242, 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.012309174, 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000050864 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 277360 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-04-25T17:51:54.129 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-04-25T17:51:54.131 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 147947, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 29, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-04-25T17:51:54.132 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-04-25T17:51:54.133 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 334, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1574062, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 334, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T17:51:54.134 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 334, 2024-04-25T17:51:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1574062, 2024-04-25T17:51:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T17:51:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:54.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:54.135 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:51:54.438 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:54 smithi098 ceph-mon[29192]: pgmap v504: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 3.1 MiB/s wr, 345 op/s 2024-04-25T17:51:54.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:54 smithi203 ceph-mon[32834]: pgmap v504: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 3.1 MiB/s wr, 345 op/s 2024-04-25T17:51:54.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:54 smithi173 ceph-mon[32856]: pgmap v504: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 3.1 MiB/s wr, 345 op/s 2024-04-25T17:51:54.782 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T17:51:54.782 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T17:51:54.783 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-25T17:51:54.783 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-25T17:51:54.783 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-25T17:51:54.783 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T17:51:54.783 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T17:51:55.021 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:51:56.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:56 smithi098 ceph-mon[29192]: pgmap v505: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 52 KiB/s wr, 22 op/s 2024-04-25T17:51:56.573 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:56 smithi173 ceph-mon[32856]: pgmap v505: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 52 KiB/s wr, 22 op/s 2024-04-25T17:51:56.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:56 smithi203 ceph-mon[32834]: pgmap v505: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 52 KiB/s wr, 22 op/s 2024-04-25T17:51:56.714 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:51:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:51:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 331, 2024-04-25T17:51:56.714 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 318, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1622624, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 195568, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.056397696, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.020747564, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.035118192, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.006748706, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 318, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.014330087, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000045063 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:56.715 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1622624, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 195568 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 3, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 243, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 480, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 277008, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 6, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.018296530, 2024-04-25T17:51:56.716 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.012632456, 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.002839164, 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 240, 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.019117732, 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000079657 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 277008 2024-04-25T17:51:56.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.000413026, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000338766, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:56.718 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.719 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:51:56.720 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-04-25T17:51:56.721 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 147970, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 29, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-04-25T17:51:56.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-04-25T17:51:56.723 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 330, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1573542, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 330, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 330, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1573542, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:51:57.466 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-25T17:51:57.466 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-25T17:51:57.466 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-25T17:51:57.466 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-25T17:51:57.466 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-25T17:51:57.466 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T17:51:57.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T17:51:57.708 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:51:58.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:51:58 smithi098 ceph-mon[29192]: pgmap v506: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 56 KiB/s wr, 25 op/s 2024-04-25T17:51:58.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:51:58 smithi203 ceph-mon[32834]: pgmap v506: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 56 KiB/s wr, 25 op/s 2024-04-25T17:51:58.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:51:58 smithi173 ceph-mon[32856]: pgmap v506: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 56 KiB/s wr, 25 op/s 2024-04-25T17:51:59.826 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T17:51:59.826 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001060959, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000226484, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:59.827 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 333, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 322, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 1621312, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 198960, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-04-25T17:51:59.828 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.059081578, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.019223574, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.041626106, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.006649466, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 321, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.013680168, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000042617 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1621312, 2024-04-25T17:51:59.829 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 198960 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 6, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 248, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3264, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 199728, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 5, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.018419755, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.013710649, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001978894, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:51:59.830 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 246, 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.015956893, 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000064865 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 199728 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-04-25T17:51:59.831 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-04-25T17:51:59.832 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-04-25T17:51:59.833 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148731, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 31, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-04-25T17:51:59.834 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T17:51:59.835 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 335, 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1574295, 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 335, 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 335, 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1574295, 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T17:51:59.836 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T17:52:00.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:00 smithi098 ceph-mon[29192]: pgmap v507: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-04-25T17:52:00.549 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T17:52:00.549 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T17:52:00.550 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-25T17:52:00.550 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-25T17:52:00.550 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-25T17:52:00.550 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T17:52:00.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T17:52:00.578 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:00 smithi173 ceph-mon[32856]: pgmap v507: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-04-25T17:52:00.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:00 smithi203 ceph-mon[32834]: pgmap v507: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 27 KiB/s wr, 13 op/s 2024-04-25T17:52:00.803 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:52:01.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:01 smithi173 ceph-mon[32856]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:52:01.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:01 smithi098 ceph-mon[29192]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:52:01.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:01 smithi203 ceph-mon[32834]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T17:52:02.626 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:02 smithi173 ceph-mon[32856]: pgmap v508: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-04-25T17:52:02.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:02 smithi098 ceph-mon[29192]: pgmap v508: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-04-25T17:52:02.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:02 smithi203 ceph-mon[32834]: pgmap v508: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-04-25T17:52:02.754 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 2489, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 2485, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 2380463, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 8267377, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 176, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 8, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.265007120, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.077780636, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.120701257, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.052757015, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2481, 2024-04-25T17:52:02.755 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.388243754, 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000156486 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 464, 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000082945, 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000178 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 1904032, 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 680496 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 12105, 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 14085, 2024-04-25T17:52:02.756 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 4756557, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 9003582, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 10, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 6, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.950116255, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.367092911, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.598063900, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.160483600, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 14086, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1.144450350, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000081247 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:02.757 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 22278, 2024-04-25T17:52:02.758 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.003846805, 2024-04-25T17:52:02.758 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000172 2024-04-25T17:52:02.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.758 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 2853360, 2024-04-25T17:52:02.758 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 5448272 2024-04-25T17:52:02.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.758 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:52:02.758 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 8605, 2024-04-25T17:52:02.758 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 8816, 2024-04-25T17:52:02.758 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 2892715, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 20115362, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 27, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 6, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.568181744, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.220800454, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.265333804, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.106808213, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 8814, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1.063756084, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000120689 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 11073, 2024-04-25T17:52:02.759 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.001873396, 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000169 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4828, 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 3.065515196, 2024-04-25T17:52:02.760 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000634945 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 145, 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.008477187, 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000058463 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 3026, 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 2854, 2024-04-25T17:52:02.761 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2854, 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 35.412768898, 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.012408118 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 172, 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 3, 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 671, 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 5, 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-25T17:52:02.762 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 289, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 114, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 141, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 34, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 126, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 1454, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 65, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 65, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 16, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 5268, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 3899, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 170, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 24, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 2, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-25T17:52:02.763 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 204, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 10967, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 43, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 1407, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 17, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 396, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 995, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 1169604344, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 6, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-25T17:52:02.764 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 28, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 1013, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 2081, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 2437, 2024-04-25T17:52:02.765 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 1268, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 62, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 637 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 24, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 63, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 16, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 161, 2024-04-25T17:52:02.766 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 36, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 110, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 3, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 11, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 1670, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 43, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 5, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-25T17:52:02.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 4716, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 4699, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 4699, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 17, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 238, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 71, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 70, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 70, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 1, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 1, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-25T17:52:02.768 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 1, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 23627339, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 23742164, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 0, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2470, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 16.615798254, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.006727043 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 0 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 226, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 1576, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 1350, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 64, 2024-04-25T17:52:02.769 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 110, 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 46, 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 300, 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 2210, 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 1910, 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 65, 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 1360, 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 1295, 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 72036, 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 231800 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-25T17:52:02.770 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 4289, 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 715, 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 3026, 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 304, 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 1898, 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1056, 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 3.489244554, 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.003304208 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 214, 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.112112496, 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000523890 2024-04-25T17:52:02.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 121, 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.403235295, 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.003332523 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 524, 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 27.977356741, 2024-04-25T17:52:02.772 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.053391902 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.774 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-25T17:52:02.774 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.774 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.774 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.774 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-25T17:52:02.774 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.774 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.774 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.775 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-25T17:52:02.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 17, 2024-04-25T17:52:02.775 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002944236, 2024-04-25T17:52:02.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000173190 2024-04-25T17:52:02.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.775 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-25T17:52:02.775 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 37, 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.336035369, 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.009082037 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 17, 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.293014586, 2024-04-25T17:52:02.776 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.017236152 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 501, 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.212990751, 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000425131 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 62, 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.107452875, 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001733110 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-25T17:52:02.777 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 138, 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1.999666273, 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.014490335 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.011144801, 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.005572400 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 11, 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.070594978, 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.006417725 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.001037945, 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001037945 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-25T17:52:02.780 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 7, 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.019747187, 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002821026 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-25T17:52:02.781 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 146, 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.376190811, 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002576649 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.782 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 2, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 4, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 2, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 2, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 164, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 82, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 962, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.783 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 360, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 360, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:52:02.784 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:52:02.785 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 592533, 2024-04-25T17:52:02.786 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 394, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 8032, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 167, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-25T17:52:02.787 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 1020525, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 25947, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 5524, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 23420988, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 32, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 5524, 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-25T17:52:02.788 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5524, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 75.548320642, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.013676379 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5524, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 9388 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 5524, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 18, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 5506, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 585, 2024-04-25T17:52:02.789 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 1198, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 15, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 2485, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 272, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 4, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 571, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 3, 2024-04-25T17:52:02.790 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 2396, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-25T17:52:02.791 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 114, 2024-04-25T17:52:02.792 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 25, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 12, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 12, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 1713, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 6, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 140 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 1, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.793 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 1998 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 17, 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 9, 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 50, 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 36, 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:02.794 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 23195, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 7545847, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 23195, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 23195, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 7545847, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.795 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 5524, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 23424856, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 5524, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 23424856, 2024-04-25T17:52:02.796 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 5524, 2024-04-25T17:52:02.797 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 5524, 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 5524, 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 5524, 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:02.798 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 36, 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 3916, 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 36, 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 15, 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 3916, 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.799 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55d26a04a500": { 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 60, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 4716, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 19547860, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 4716, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2470, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 19547800, 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:02.800 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:02.801 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:02.801 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:02.801 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:02.801 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:02.801 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T17:52:03.656 INFO:tasks.check_counter:Daemon mds.b mds.exported=43 2024-04-25T17:52:03.657 INFO:tasks.check_counter:Daemon mds.b mds.imported=17 2024-04-25T17:52:03.657 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-25T17:52:03.657 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=17 2024-04-25T17:52:03.657 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=11 2024-04-25T17:52:03.657 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T17:52:03.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T17:52:03.948 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:52:04.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:04 smithi173 ceph-mon[32856]: pgmap v509: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-04-25T17:52:04.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:04 smithi098 ceph-mon[29192]: pgmap v509: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-04-25T17:52:04.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:04 smithi203 ceph-mon[32834]: pgmap v509: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 23 KiB/s wr, 14 op/s 2024-04-25T17:52:05.820 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T17:52:05.820 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:52:05.820 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 929, 2024-04-25T17:52:05.820 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 931, 2024-04-25T17:52:05.820 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 4436277, 2024-04-25T17:52:05.820 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 295486, 2024-04-25T17:52:05.820 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 7, 2024-04-25T17:52:05.820 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.077829186, 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.029696355, 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.020144362, 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.020221955, 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 929, 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.047404055, 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000051026 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 1115, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 1360, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 4473804, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 943635, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 7, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.104199732, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.035480373, 2024-04-25T17:52:05.822 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.026303251, 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.031457189, 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1357, 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.054361985, 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000040060 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 589696 2024-04-25T17:52:05.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 1889, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 1847, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 2429686, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 679272, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.220586954, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.068746878, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.084252338, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.063293333, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1847, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.089590565, 2024-04-25T17:52:05.824 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000048505 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 1779680, 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 204816 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-25T17:52:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5946, 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.163935895, 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000027570 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-25T17:52:05.827 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 1235, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 416, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 819, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 640, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 9, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-04-25T17:52:05.828 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-04-25T17:52:05.829 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T17:52:05.830 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-25T17:52:05.831 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.832 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 1642, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 5, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-04-25T17:52:05.833 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 12762764, 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 12759371, 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 1642 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 1184, 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 1589, 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 405, 2024-04-25T17:52:05.834 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 54, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 60, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 6, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 1235, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 1661, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 426, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 60104, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 215416 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-04-25T17:52:05.835 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.837 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.839 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-25T17:52:05.840 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.840 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.840 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.840 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-25T17:52:05.840 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.840 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.840 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.841 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.842 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.845 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-25T17:52:05.845 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.845 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.845 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.845 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-25T17:52:05.845 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.845 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.845 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 2, 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 5, 2024-04-25T17:52:05.847 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 3, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 4, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 3381, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:52:05.848 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:52:05.849 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:52:05.850 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 244633, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 1272, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 8032, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 167, 2024-04-25T17:52:05.851 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 3684883, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 34672, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 3534, 2024-04-25T17:52:05.852 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 21, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 3, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 3534, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 3534, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 9.379794520, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002654158 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 3534, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 3535 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 3534, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 3534, 2024-04-25T17:52:05.853 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 2179, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 1353, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-25T17:52:05.854 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 1, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-25T17:52:05.855 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 115, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 26, 2024-04-25T17:52:05.856 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 6, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 7, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 1, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-04-25T17:52:05.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 3929, 2024-04-25T17:52:05.858 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 10963229, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 3929, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 3929, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 10963229, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:05.859 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 3534, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 8568465, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 3534, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 8568465, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.860 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 3534, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 3534, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 3534, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 3534, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.861 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:05.862 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x5571961006e0": { 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T17:52:05.863 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:05.864 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:05.865 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T17:52:06.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:06 smithi173 ceph-mon[32856]: pgmap v510: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.4 KiB/s wr, 6 op/s 2024-04-25T17:52:06.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:06 smithi173 ceph-mon[32856]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T17:52:06.541 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-25T17:52:06.541 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-25T17:52:06.541 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-25T17:52:06.541 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-25T17:52:06.541 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T17:52:06.541 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T17:52:06.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T17:52:06.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:06 smithi098 ceph-mon[29192]: pgmap v510: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.4 KiB/s wr, 6 op/s 2024-04-25T17:52:06.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:06 smithi098 ceph-mon[29192]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T17:52:06.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:06 smithi203 ceph-mon[32834]: pgmap v510: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.4 KiB/s wr, 6 op/s 2024-04-25T17:52:06.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:06 smithi203 ceph-mon[32834]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T17:52:06.780 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:52:08.510 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:08 smithi173 ceph-mon[32856]: pgmap v512: 97 pgs: 97 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 68 KiB/s wr, 30 op/s 2024-04-25T17:52:08.537 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T17:52:08.537 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:52:08.537 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 905, 2024-04-25T17:52:08.537 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 937, 2024-04-25T17:52:08.537 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 308465, 2024-04-25T17:52:08.537 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 4545321, 2024-04-25T17:52:08.537 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 17, 2024-04-25T17:52:08.537 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 2, 2024-04-25T17:52:08.537 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.086705885, 2024-04-25T17:52:08.537 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.034183054, 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.022558224, 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.022554825, 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 936, 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.058533990, 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000062536 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 84, 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000015317, 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000182 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.538 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 8736, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 15056 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 3802, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 5649, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 2899082, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 7522638, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 11, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 6, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.331891736, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.142679149, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.206990545, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.020205599, 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:08.539 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5648, 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.658895295, 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000116659 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 6185, 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.001542123, 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000249 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 2784080, 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 3128816 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 5669, 2024-04-25T17:52:08.540 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 5748, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 3005483, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 3270644, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 10, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 6, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.364716989, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.173605288, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.242163473, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.006866005, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5745, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.256839877, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000044706 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:08.541 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 10535, 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.001782841, 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000169 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 589344 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-25T17:52:08.542 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2113, 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1.435526347, 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000679378 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 18, 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000576446, 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000032024 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:08.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 1159, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 1081, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1081, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 25.689662904, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.023764720 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 78, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 8, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 19, 2024-04-25T17:52:08.544 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 1, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 1253, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 86, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 1141, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 26, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 1129, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 474, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 869, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 869, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 15, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 2187, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 2028, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 69, 2024-04-25T17:52:08.545 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 24, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 2, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 5, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 3211, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 7, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 423, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 24, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 1277, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 1027, 2024-04-25T17:52:08.546 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 1626971436, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 6, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 25, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 1133, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 1609, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 2, 2024-04-25T17:52:08.547 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 1621, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 263, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 53, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 72 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 33, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 16, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 4, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-25T17:52:08.548 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 17, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 3, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 6, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 5, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 11, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 8207, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 7, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-25T17:52:08.549 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 1, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 1643, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 1643, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 5, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-25T17:52:08.550 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 5, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 12766790, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 0, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1110, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 4.544054841, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.004093743 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 0 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.551 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 1225, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 1650, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 425, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 78, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 83, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 5, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 1257, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 1745, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 488, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 869, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 1446, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 577, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 66836, 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 223608 2024-04-25T17:52:08.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 1376, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 573, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 1159, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 56, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 1209, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002534185, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002534185 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 277, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 15.341231549, 2024-04-25T17:52:08.553 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.055383507 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 10, 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.069239227, 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.006923922 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 517, 2024-04-25T17:52:08.554 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 5.283630181, 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.010219787 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.555 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-25T17:52:08.556 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.557 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.557 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.557 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-25T17:52:08.557 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.557 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-25T17:52:08.558 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 216, 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.286083965, 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001324462 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 34, 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.095039650, 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002795283 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 10, 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 4.410777745, 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.441077774 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.559 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.161033143, 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.080516571 2024-04-25T17:52:08.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.562 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 14, 2024-04-25T17:52:08.563 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.040093259, 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002863804 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 2, 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 5, 2024-04-25T17:52:08.564 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 3, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 2, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 108, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 54, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 966, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:52:08.565 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:52:08.566 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:52:08.567 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 503539, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 1182, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 8032, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 167, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:52:08.568 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 4203524, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 44174, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 1347, 2024-04-25T17:52:08.569 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 8707367, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 33, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 1347, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1347, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 5.315883058, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.003946461 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1347, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1510 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 1347, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 9, 2024-04-25T17:52:08.570 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 1338, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 0, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 10, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 1, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 1111, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 64, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 18, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-25T17:52:08.571 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 1, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 20, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 2, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-25T17:52:08.572 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-25T17:52:08.573 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 115, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 26, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 11, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 12, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 1, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 218, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 16, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 49 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 1, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-25T17:52:08.574 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 8528 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 3, 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 3, 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 3, 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 3, 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-25T17:52:08.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 10372, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 5125799, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 10372, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 10372, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 5125799, 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:08.576 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 1347, 2024-04-25T17:52:08.577 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 8707662, 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1347, 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 8707662, 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:08.578 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T17:52:08.582 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T17:52:08.582 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 1347, 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 1347, 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1347, 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1347, 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-25T17:52:08.583 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 3, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 423, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 3, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 423, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.584 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x5647543cc500": { 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 1643, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 8572486, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 1643, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1110, 2024-04-25T17:52:08.585 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 8572486, 2024-04-25T17:52:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:08.586 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T17:52:08.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:08 smithi098 ceph-mon[29192]: pgmap v512: 97 pgs: 97 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 68 KiB/s wr, 30 op/s 2024-04-25T17:52:08.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:08 smithi203 ceph-mon[32834]: pgmap v512: 97 pgs: 97 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 68 KiB/s wr, 30 op/s 2024-04-25T17:52:09.353 INFO:tasks.check_counter:Daemon mds.h mds.exported=7 2024-04-25T17:52:09.353 INFO:tasks.check_counter:Daemon mds.h mds.imported=24 2024-04-25T17:52:09.353 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-25T17:52:09.353 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-25T17:52:09.353 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T17:52:09.353 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T17:52:09.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T17:52:09.591 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T17:52:10.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:10 smithi098 ceph-mon[29192]: pgmap v513: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 69 KiB/s wr, 31 op/s 2024-04-25T17:52:10.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:10 smithi098 ceph-mon[29192]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T17:52:10.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:10 smithi203 ceph-mon[32834]: pgmap v513: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 69 KiB/s wr, 31 op/s 2024-04-25T17:52:10.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:10 smithi203 ceph-mon[32834]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T17:52:10.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:10 smithi173 ceph-mon[32856]: pgmap v513: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 69 KiB/s wr, 31 op/s 2024-04-25T17:52:10.758 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:10 smithi173 ceph-mon[32856]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T17:52:11.393 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T17:52:11.393 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:52:11.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 0, 2024-04-25T17:52:11.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 2, 2024-04-25T17:52:11.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 0, 2024-04-25T17:52:11.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 0, 2024-04-25T17:52:11.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-04-25T17:52:11.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 0, 2024-04-25T17:52:11.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.001018276, 2024-04-25T17:52:11.393 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.000260625, 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.394 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 337, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 326, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 1622400, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 201520, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 2, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 1, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.059356033, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.019237789, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.039918838, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.007711165, 2024-04-25T17:52:11.395 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 325, 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.015587194, 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000047960 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 1622400, 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 201520 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:52:11.396 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 7, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 253, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 3424, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 278864, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.019221996, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.010102233, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.006328243, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 251, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.023424267, 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000093323 2024-04-25T17:52:11.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3424, 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 278864 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 0, 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-25T17:52:11.398 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:52:11.399 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:52:11.400 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 147977, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 29, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:52:11.401 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 0, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 0, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 928, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 2, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T17:52:11.402 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 340, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 1574842, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 340, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 340, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1574842, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T17:52:11.403 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T17:52:11.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T17:52:11.404 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:11.404 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T17:52:11.404 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T17:52:12.521 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:12 smithi173 ceph-mon[32856]: pgmap v515: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 82 KiB/s wr, 33 op/s 2024-04-25T17:52:12.579 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-25T17:52:12.579 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-25T17:52:12.579 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-25T17:52:12.579 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-25T17:52:12.579 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-25T17:52:12.579 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T17:52:12.580 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T17:52:12.606 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:12 smithi203 ceph-mon[32834]: pgmap v515: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 82 KiB/s wr, 33 op/s 2024-04-25T17:52:12.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:12 smithi098 ceph-mon[29192]: pgmap v515: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 82 KiB/s wr, 33 op/s 2024-04-25T17:52:12.832 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:52:14.610 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:14 smithi203 ceph-mon[32834]: pgmap v516: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 111 KiB/s wr, 39 op/s 2024-04-25T17:52:14.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:14 smithi098 ceph-mon[29192]: pgmap v516: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 111 KiB/s wr, 39 op/s 2024-04-25T17:52:14.745 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 5, 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 7, 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 2444, 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 1606, 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 7, 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.002770915, 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.000106526, 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.001553766, 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000108757, 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:14.746 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 5, 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.003003995, 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000600799 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:52:14.747 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 789, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 782, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 2773890, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 270119, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 4, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.076688859, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.023586190, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.025910854, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.022359757, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 781, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.047258227, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000060509 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.748 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 211168, 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 52864 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 300, 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 360, 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 75696, 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 289490, 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 5, 2024-04-25T17:52:14.749 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-25T17:52:14.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.031738321, 2024-04-25T17:52:14.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.009697119, 2024-04-25T17:52:14.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.008085952, 2024-04-25T17:52:14.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.010738500, 2024-04-25T17:52:14.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:14.750 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 359, 2024-04-25T17:52:14.750 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.014027860, 2024-04-25T17:52:14.750 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000039074 2024-04-25T17:52:14.750 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.750 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 195680 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T17:52:14.751 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1646, 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.049427899, 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000030029 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.752 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-25T17:52:14.753 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 217, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 142, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 75, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 14, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 12, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-25T17:52:14.754 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 995, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 1169604415, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 6, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T17:52:14.755 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-25T17:52:14.756 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-25T17:52:14.757 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-25T17:52:14.758 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 515, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 2, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 16, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-25T17:52:14.759 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 23627339, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 23774767, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 23749212, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 515 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 139, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 261, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 122, 2024-04-25T17:52:14.760 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 20, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 25, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 5, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 218, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 333, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 115, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 52080, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215416 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-04-25T17:52:14.761 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-25T17:52:14.762 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-25T17:52:14.763 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.764 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.765 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.766 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.767 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.768 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.769 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-25T17:52:14.770 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-25T17:52:14.771 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 2, 2024-04-25T17:52:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 2, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 0, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 0, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2164, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:52:14.773 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-25T17:52:14.774 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-25T17:52:14.775 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 213206, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 267, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 8032, 2024-04-25T17:52:14.776 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 167, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 605825, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 14786, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-25T17:52:14.777 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 983, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 26, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 4, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 983, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 983, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 4.565887205, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.004644849 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 983, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 984 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.778 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 983, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 983, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 588, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 392, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-25T17:52:14.779 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 2, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-25T17:52:14.780 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-25T17:52:14.781 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 116, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 10, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 7, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 7, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-25T17:52:14.782 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-04-25T17:52:14.783 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 1090, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 2745611, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 1090, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1090, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2745611, 2024-04-25T17:52:14.784 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:14.785 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 983, 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 2359318, 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 983, 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2359318, 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:14.786 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 983, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 983, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 983, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 983, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.787 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.788 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55d428fe86e0": { 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:14.789 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:14.790 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:14.790 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:14.790 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:14.790 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:14.790 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-25T17:52:14.790 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:14 smithi173 ceph-mon[32856]: pgmap v516: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 111 KiB/s wr, 39 op/s 2024-04-25T17:52:15.564 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-25T17:52:15.565 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-25T17:52:15.565 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-25T17:52:15.565 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-25T17:52:15.565 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T17:52:15.565 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T17:52:15.565 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T17:52:15.863 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:52:16.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:16 smithi203 ceph-mon[32834]: pgmap v517: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 96 KiB/s wr, 34 op/s 2024-04-25T17:52:16.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:16 smithi098 ceph-mon[29192]: pgmap v517: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 96 KiB/s wr, 34 op/s 2024-04-25T17:52:16.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:16 smithi173 ceph-mon[32856]: pgmap v517: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 96 KiB/s wr, 34 op/s 2024-04-25T17:52:17.630 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-25T17:52:17.630 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:52:17.630 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 3230, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 3230, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 13502169, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 1026323, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.267498555, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.085194490, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.072692333, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.089133625, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3229, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.126841113, 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000039281 2024-04-25T17:52:17.631 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 753, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 713, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 5665728, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 319412, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 4, 2024-04-25T17:52:17.632 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.095387690, 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.028961785, 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.054787886, 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.018918258, 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 712, 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.039764047, 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000055848 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.633 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1777744, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 205824 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 3, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 249, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 588, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 591806, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 7, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.013984628, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.008848643, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.002520899, 2024-04-25T17:52:17.634 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000041838, 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 246, 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.020844938, 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000084735 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 591488 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.635 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 6147, 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.154433988, 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000025123 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T17:52:17.636 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-25T17:52:17.637 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 743, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 66, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 677, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 649, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-04-25T17:52:17.638 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 9, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-04-25T17:52:17.639 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T17:52:17.640 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-25T17:52:17.641 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-25T17:52:17.642 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-25T17:52:17.643 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 2744, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 6, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-25T17:52:17.644 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 20576306, 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 20557704, 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 2744 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 322, 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 1079, 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 757, 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 43, 2024-04-25T17:52:17.645 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 62, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 19, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 743, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 1351, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 608, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 54916, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215416 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-04-25T17:52:17.646 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.647 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.650 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-25T17:52:17.651 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.651 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.651 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.651 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.651 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.652 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.653 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.654 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.655 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-25T17:52:17.656 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-25T17:52:17.657 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 2, 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 3, 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 1, 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 3, 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-04-25T17:52:17.658 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2760, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:52:17.659 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:52:17.660 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 209748, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 543, 2024-04-25T17:52:17.661 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 8032, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 167, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 1525330, 2024-04-25T17:52:17.662 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 33969, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 3584, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 21, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 3, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 3584, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3584, 2024-04-25T17:52:17.663 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 7.043825727, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001965353 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3584, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 3585 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 3584, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 3584, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 2134, 2024-04-25T17:52:17.664 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 1448, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 1, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-25T17:52:17.665 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-25T17:52:17.666 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 116, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 27, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 5, 2024-04-25T17:52:17.667 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 6, 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 1, 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-25T17:52:17.668 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:17.669 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 3982, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 18787031, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 3982, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 3982, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 18787031, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.670 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 3584, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 16382007, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 3584, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 16382007, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.671 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 3584, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 3584, 2024-04-25T17:52:17.672 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 3584, 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 3584, 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T17:52:17.673 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55ee130b06e0": { 2024-04-25T17:52:17.674 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:17.675 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:17.676 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:17.676 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:17.676 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-25T17:52:18.418 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-25T17:52:18.418 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-25T17:52:18.418 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-25T17:52:18.418 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-25T17:52:18.418 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T17:52:18.418 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T17:52:18.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T17:52:18.666 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:52:18.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:18 smithi098 ceph-mon[29192]: pgmap v518: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 31 KiB/s wr, 9 op/s 2024-04-25T17:52:18.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:18 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:52:18.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:18 smithi203 ceph-mon[32834]: pgmap v518: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 31 KiB/s wr, 9 op/s 2024-04-25T17:52:18.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:18 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:52:18.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:18 smithi173 ceph-mon[32856]: pgmap v518: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 31 KiB/s wr, 9 op/s 2024-04-25T17:52:18.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:18 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T17:52:19.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:19 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:52:19.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:19 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:52:19.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:19 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:52:19.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:19 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:52:19.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:19 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:52:19.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:19 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:52:19.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:19 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T17:52:19.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:19 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T17:52:19.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:19 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' 2024-04-25T17:52:20.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:20 smithi203 ceph-mon[32834]: pgmap v519: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 9 op/s 2024-04-25T17:52:20.492 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-25T17:52:20.492 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:52:20.492 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 5327, 2024-04-25T17:52:20.492 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 7504, 2024-04-25T17:52:20.493 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1689947, 2024-04-25T17:52:20.493 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 16506886, 2024-04-25T17:52:20.493 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 19, 2024-04-25T17:52:20.493 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 9, 2024-04-25T17:52:20.493 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.415498024, 2024-04-25T17:52:20.493 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.185201843, 2024-04-25T17:52:20.493 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.220314880, 2024-04-25T17:52:20.493 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.037551318, 2024-04-25T17:52:20.493 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:20.493 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 7496, 2024-04-25T17:52:20.494 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.352049292, 2024-04-25T17:52:20.494 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000046964 2024-04-25T17:52:20.494 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.494 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:20.494 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 8738, 2024-04-25T17:52:20.494 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001456965, 2024-04-25T17:52:20.494 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000166 2024-04-25T17:52:20.494 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.494 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1362688, 2024-04-25T17:52:20.494 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 3427216 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 6414, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 6376, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 4070466, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 2841319, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 7, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.376670112, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.155375667, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.260705187, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.016475895, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 6379, 2024-04-25T17:52:20.495 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.267076706, 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000041868 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 11085, 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001888191, 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000170 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1794480, 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 654704 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 2747, 2024-04-25T17:52:20.496 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2963, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1180267, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 5305977, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 10, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.172207704, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.067963971, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.096116909, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.018550836, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2963, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.108202193, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000036517 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.497 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:20.498 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4145, 2024-04-25T17:52:20.498 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000651864, 2024-04-25T17:52:20.498 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000157 2024-04-25T17:52:20.498 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.498 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 7280, 2024-04-25T17:52:20.498 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 172256 2024-04-25T17:52:20.499 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.499 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-25T17:52:20.499 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-25T17:52:20.499 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-25T17:52:20.499 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.500 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-25T17:52:20.500 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T17:52:20.500 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T17:52:20.500 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2569, 2024-04-25T17:52:20.500 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1.416665720, 2024-04-25T17:52:20.500 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000551446 2024-04-25T17:52:20.501 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:20.501 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.501 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-25T17:52:20.501 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T17:52:20.501 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T17:52:20.501 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 683, 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.032790432, 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000048009 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 1804, 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 1688, 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1688, 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 44.755077356, 2024-04-25T17:52:20.502 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.026513671 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 116, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 5, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 19, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 1, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 486, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 58, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 383, 2024-04-25T17:52:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 45, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 403, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 662, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 134, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 135, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 11, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 3999, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 3226, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 107, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 54, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 2, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 163, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 876, 2024-04-25T17:52:20.504 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 22, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 556, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 31, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 713, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 1027, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 1626971436, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 6, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T17:52:20.505 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 23, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 1088, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 1830, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 1828, 2024-04-25T17:52:20.506 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 457, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 199, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 1153 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 54, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 32, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 11, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-25T17:52:20.507 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 291, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 223, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 54, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 6, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 11, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 2190, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 23, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-25T17:52:20.508 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 1, 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 2745, 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 2745, 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-25T17:52:20.509 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 6, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 6, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 20594901, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1251, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 6.863946866, 2024-04-25T17:52:20.510 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.005486768 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 339, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 1107, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 768, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 63, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 80, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 17, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 502, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 1438, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 936, 2024-04-25T17:52:20.511 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 135, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 834, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 699, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 64420, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223608 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 3124, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 562, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 1804, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 207, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 1596, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 240, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 39.139292681, 2024-04-25T17:52:20.512 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.163080386 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 143, 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.353387245, 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.002471239 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 70, 2024-04-25T17:52:20.513 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.292175262, 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.004173932 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 604, 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 2.860444320, 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.004735834 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.514 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2, 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000289830, 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000144915 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-25T17:52:20.516 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.516 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.516 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.516 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.516 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-25T17:52:20.517 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 242, 2024-04-25T17:52:20.518 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.183235691, 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000757172 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 36, 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.021675299, 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000602091 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 270, 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1.664225868, 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.006163799 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.519 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.520 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.521 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.522 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-25T17:52:20.523 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 81, 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.240351160, 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.002967298 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 2, 2024-04-25T17:52:20.524 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 3, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 1, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 2, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 337, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 168, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 981, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:52:20.525 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-25T17:52:20.526 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-25T17:52:20.527 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 814572, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 599, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 88, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 1, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-25T17:52:20.528 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 8032, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 167, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 1554299, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 42522, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-25T17:52:20.529 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-25T17:52:20.530 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 2048, 2024-04-25T17:52:20.530 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 16588131, 2024-04-25T17:52:20.530 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 33, 2024-04-25T17:52:20.530 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 2048, 2024-04-25T17:52:20.530 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-25T17:52:20.530 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2048, 2024-04-25T17:52:20.531 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 11.508714753, 2024-04-25T17:52:20.531 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.005619489 2024-04-25T17:52:20.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.531 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-25T17:52:20.531 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-25T17:52:20.531 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2048, 2024-04-25T17:52:20.531 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 2340 2024-04-25T17:52:20.531 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.532 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 2048, 2024-04-25T17:52:20.532 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 87, 2024-04-25T17:52:20.532 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 1961, 2024-04-25T17:52:20.532 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-25T17:52:20.532 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-25T17:52:20.532 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-04-25T17:52:20.532 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-04-25T17:52:20.532 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 82, 2024-04-25T17:52:20.532 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 1333, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 136, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 1, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 238, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 2, 2024-04-25T17:52:20.533 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-25T17:52:20.534 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 116, 2024-04-25T17:52:20.535 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 27, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 12, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 12, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 387, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 10, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 121 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-25T17:52:20.536 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 2416 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 10, 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 10, 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 223, 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 223, 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-25T17:52:20.537 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 14484, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 5433247, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 14484, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 14484, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 5433247, 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:20.538 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:20.539 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 2048, 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 16611342, 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2048, 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 16611342, 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-25T17:52:20.540 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 2048, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 2048, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2048, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2048, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.541 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 223, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 23291, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 223, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 82, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 23291, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.542 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55e1d41ea500": { 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 2745, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 16400597, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 2745, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T17:52:20.543 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1251, 2024-04-25T17:52:20.544 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 16400597, 2024-04-25T17:52:20.544 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:20.544 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:20.544 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:20.544 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:20.544 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:20.544 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:20.544 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-25T17:52:20.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:20 smithi098 ceph-mon[29192]: pgmap v519: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 9 op/s 2024-04-25T17:52:20.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:20 smithi173 ceph-mon[32856]: pgmap v519: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 9 op/s 2024-04-25T17:52:21.267 INFO:tasks.check_counter:Daemon mds.i mds.exported=22 2024-04-25T17:52:21.267 INFO:tasks.check_counter:Daemon mds.i mds.imported=31 2024-04-25T17:52:21.267 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-25T17:52:21.268 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-25T17:52:21.268 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T17:52:21.268 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T17:52:21.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T17:52:21.506 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T17:52:22.665 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:22 smithi203 ceph-mon[32834]: pgmap v520: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 42 KiB/s wr, 9 op/s 2024-04-25T17:52:22.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:22 smithi098 ceph-mon[29192]: pgmap v520: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 42 KiB/s wr, 9 op/s 2024-04-25T17:52:22.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:22 smithi173 ceph-mon[32856]: pgmap v520: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 42 KiB/s wr, 9 op/s 2024-04-25T17:52:23.245 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-25T17:52:23.245 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T17:52:23.245 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 345, 2024-04-25T17:52:23.245 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 332, 2024-04-25T17:52:23.245 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1626432, 2024-04-25T17:52:23.245 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 204528, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 4, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 0, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.059966545, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.019414693, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.039695166, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.007866298, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 332, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.017393407, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000052389 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:23.246 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1626432, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 204528 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 3, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 254, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 480, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 278944, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.014458628, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.010010973, 2024-04-25T17:52:23.247 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.001633880, 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 251, 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.024609750, 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000098046 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 278944 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.248 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 0, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 0, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 0, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 2, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 1, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.000369720, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.000278722, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:23.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 0, 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-25T17:52:23.250 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-25T17:52:23.251 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-25T17:52:23.252 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 147920, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 27, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 0, 2024-04-25T17:52:23.253 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 0, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 928, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 2, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T17:52:23.254 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 344, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 1575362, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 344, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 344, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1575362, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T17:52:23.255 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-25T17:52:23.928 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-25T17:52:23.928 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-25T17:52:23.928 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-25T17:52:23.929 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-25T17:52:23.929 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-25T17:52:23.929 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T17:52:23.940 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T17:52:23.940 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:52:23.940 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T17:52:23.961 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:52:23.961 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T17:52:24.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd blocklist ls 2024-04-25T17:52:24.310 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:52:24.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:24 smithi098 ceph-mon[29192]: pgmap v521: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 67 KiB/s wr, 11 op/s 2024-04-25T17:52:24.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:24 smithi203 ceph-mon[32834]: pgmap v521: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 67 KiB/s wr, 11 op/s 2024-04-25T17:52:24.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:24 smithi173 ceph-mon[32856]: pgmap v521: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 67 KiB/s wr, 11 op/s 2024-04-25T17:52:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.237+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/2426787008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3810cfe0 msgr2=0x7f8a3810f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.237+0000 7f8a3e5a0700 1 --2- 172.21.15.98:0/2426787008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3810cfe0 0x7f8a3810f480 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000b0a0 tx=0x7f8a3000b3b0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.237+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/2426787008 shutdown_connections 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.237+0000 7f8a3e5a0700 1 --2- 172.21.15.98:0/2426787008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a38103c00 0x7f8a38103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.237+0000 7f8a3e5a0700 1 --2- 172.21.15.98:0/2426787008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3810cfe0 0x7f8a3810f480 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.237+0000 7f8a3e5a0700 1 --2- 172.21.15.98:0/2426787008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a38104520 0x7f8a3810ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.237+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/2426787008 >> 172.21.15.98:0/2426787008 conn(0x7f8a380fd3f0 msgr2=0x7f8a380ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.237+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/2426787008 shutdown_connections 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.237+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/2426787008 wait complete. 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3e5a0700 1 Processor -- start 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3e5a0700 1 -- start start 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3e5a0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a38103c00 0x7f8a38101860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3e5a0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a38104520 0x7f8a38101da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:26.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3e5a0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3810cfe0 0x7f8a381022e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3e5a0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a38112bb0 con 0x7f8a3810cfe0 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3e5a0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8a38112a30 con 0x7f8a38103c00 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3e5a0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8a38112d30 con 0x7f8a38104520 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3dd9f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3810cfe0 0x7f8a381022e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3dd9f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3810cfe0 0x7f8a381022e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33170/0 (socket says 172.21.15.98:33170) 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3dd9f700 1 -- 172.21.15.98:0/3746244102 learned_addr learned my addr 172.21.15.98:0/3746244102 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3cd9d700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a38104520 0x7f8a38101da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3d59e700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a38103c00 0x7f8a38101860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3dd9f700 1 -- 172.21.15.98:0/3746244102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a38104520 msgr2=0x7f8a38101da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3dd9f700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a38104520 0x7f8a38101da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3dd9f700 1 -- 172.21.15.98:0/3746244102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a38103c00 msgr2=0x7f8a38101860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:26.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3dd9f700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a38103c00 0x7f8a38101860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3dd9f700 1 -- 172.21.15.98:0/3746244102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a381a7510 con 0x7f8a3810cfe0 2024-04-25T17:52:26.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.238+0000 7f8a3dd9f700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3810cfe0 0x7f8a381022e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000b830 tx=0x7f8a30004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:26.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.239+0000 7f8a2e7fc700 1 -- 172.21.15.98:0/3746244102 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30021070 con 0x7f8a3810cfe0 2024-04-25T17:52:26.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.239+0000 7f8a2e7fc700 1 -- 172.21.15.98:0/3746244102 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a30007e30 con 0x7f8a3810cfe0 2024-04-25T17:52:26.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.239+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/3746244102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a381a77a0 con 0x7f8a3810cfe0 2024-04-25T17:52:26.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.239+0000 7f8a2e7fc700 1 -- 172.21.15.98:0/3746244102 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30004700 con 0x7f8a3810cfe0 2024-04-25T17:52:26.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.239+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/3746244102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a381a7d30 con 0x7f8a3810cfe0 2024-04-25T17:52:26.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.240+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/3746244102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a38069730 con 0x7f8a3810cfe0 2024-04-25T17:52:26.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.241+0000 7f8a2e7fc700 1 -- 172.21.15.98:0/3746244102 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8a30003d70 con 0x7f8a3810cfe0 2024-04-25T17:52:26.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.241+0000 7f8a2e7fc700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8a240a3e30 0x7f8a240a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:26.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.241+0000 7f8a3d59e700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8a240a3e30 0x7f8a240a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:26.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.241+0000 7f8a2e7fc700 1 -- 172.21.15.98:0/3746244102 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8513+0+0 (secure 0 0 0) 0x7f8a3000f020 con 0x7f8a3810cfe0 2024-04-25T17:52:26.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.242+0000 7f8a3d59e700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8a240a3e30 0x7f8a240a62f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a34005950 tx=0x7f8a3400b410 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:26.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.245+0000 7f8a2e7fc700 1 -- 172.21.15.98:0/3746244102 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a3002a020 con 0x7f8a3810cfe0 2024-04-25T17:52:26.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:26 smithi098 ceph-mon[29192]: pgmap v522: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2024-04-25T17:52:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.462+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/3746244102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8a38102d90 con 0x7f8a3810cfe0 2024-04-25T17:52:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.462+0000 7f8a2e7fc700 1 -- 172.21.15.98:0/3746244102 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v116) ==== 81+0+943 (secure 0 0 0) 0x7f8a3009e770 con 0x7f8a3810cfe0 2024-04-25T17:52:26.462 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/4145487482 2024-04-26T17:35:54.302654+0000 2024-04-25T17:52:26.462 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/674729727 2024-04-26T17:35:54.302654+0000 2024-04-25T17:52:26.462 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1530074164 2024-04-26T17:35:54.302654+0000 2024-04-25T17:52:26.462 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2167465649 2024-04-26T17:35:54.302654+0000 2024-04-25T17:52:26.462 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/3170662380 2024-04-26T17:25:08.883124+0000 2024-04-25T17:52:26.462 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/3170662380 2024-04-26T17:25:08.883124+0000 2024-04-25T17:52:26.463 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1795364490 2024-04-26T17:25:08.883124+0000 2024-04-25T17:52:26.463 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/773164127 2024-04-26T17:25:08.883124+0000 2024-04-25T17:52:26.463 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3206373024 2024-04-26T17:35:54.302654+0000 2024-04-25T17:52:26.463 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3412860846 2024-04-26T17:24:41.622373+0000 2024-04-25T17:52:26.463 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/3033733589 2024-04-26T17:24:41.622373+0000 2024-04-25T17:52:26.463 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3143366689 2024-04-26T17:24:41.622373+0000 2024-04-25T17:52:26.464 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/4145487482 2024-04-26T17:35:54.302654+0000 2024-04-25T17:52:26.464 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3075118794 2024-04-26T17:24:41.622373+0000 2024-04-25T17:52:26.464 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/3033733589 2024-04-26T17:24:41.622373+0000 2024-04-25T17:52:26.464 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/672531975 2024-04-26T17:25:08.883124+0000 2024-04-25T17:52:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/3746244102 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8a240a3e30 msgr2=0x7f8a240a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8a240a3e30 0x7f8a240a62f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a34005950 tx=0x7f8a3400b410 comp rx=0 tx=0).stop 2024-04-25T17:52:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/3746244102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3810cfe0 msgr2=0x7f8a381022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3810cfe0 0x7f8a381022e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000b830 tx=0x7f8a30004ce0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/3746244102 shutdown_connections 2024-04-25T17:52:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f8a240a3e30 0x7f8a240a62f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3810cfe0 0x7f8a381022e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a38103c00 0x7f8a38101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 --2- 172.21.15.98:0/3746244102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a38104520 0x7f8a38101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/3746244102 >> 172.21.15.98:0/3746244102 conn(0x7f8a380fd3f0 msgr2=0x7f8a380fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/3746244102 shutdown_connections 2024-04-25T17:52:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:26.464+0000 7f8a3e5a0700 1 -- 172.21.15.98:0/3746244102 wait complete. 2024-04-25T17:52:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:listed 16 entries 2024-04-25T17:52:26.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:26 smithi203 ceph-mon[32834]: pgmap v522: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2024-04-25T17:52:26.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:26 smithi173 ceph-mon[32856]: pgmap v522: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2024-04-25T17:52:27.202 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T17:52:27.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:52:27.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /tmp/cephfs 2024-04-25T17:52:27.371 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:52:27.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:52:27.372 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-04-25T17:52:27.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:52:27.428 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-25T17:52:27.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:27 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3746244102' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T17:52:27.466 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-04-25T17:52:27.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:52:27.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-04-25T17:52:27.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:52:27.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-04-25T17:52:27.648 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T17:52:27.648 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T17:52:27.648 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T17:52:27.667 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056107,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17754 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/808f77d0e5f3b5ffaae9445778b45b5eb252935f63148806f040ad710c787588/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDNWLMIERYILTCEMBIUR5IS2JU:/var/lib/containers/storage/overlay/l/WRP4UC5PAX5GPC3RKOGKVPRTK2,upperdir=/var/lib/containers/storage/overlay/808f77d0e5f3b5ffaae9445778b45b5eb252935f63148806f040ad710c787588/diff,workdir=/var/lib/containers/storage/overlay/808f77d0e5f3b5ffaae9445778b45b5eb252935f63148806f040ad710c787588/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d96b3b4a7c5b585459964b5abcb4c9c0272b5ef751f48fe47cb4adc9ae6b45dc/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDNWLMIERYILTCEMBIUR5IS2JU:/var/lib/containers/storage/overlay/l/WRP4UC5PAX5GPC3RKOGKVPRTK2,upperdir=/var/lib/containers/storage/overlay/d96b3b4a7c5b585459964b5abcb4c9c0272b5ef751f48fe47cb4adc9ae6b45dc/diff,workdir=/var/lib/containers/storage/overlay/d96b3b4a7c5b585459964b5abcb4c9c0272b5ef751f48fe47cb4adc9ae6b45dc/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/56e8da9146f62c4293d22c8875e3d0ab4a9b0e7c0808a37a714478262d6f2e8e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDNWLMIERYILTCEMBIUR5IS2JU:/var/lib/containers/storage/overlay/l/WRP4UC5PAX5GPC3RKOGKVPRTK2,upperdir=/var/lib/containers/storage/overlay/56e8da9146f62c4293d22c8875e3d0ab4a9b0e7c0808a37a714478262d6f2e8e/diff,workdir=/var/lib/containers/storage/overlay/56e8da9146f62c4293d22c8875e3d0ab4a9b0e7c0808a37a714478262d6f2e8e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/dca77b3bbe1c2a422b310a7a5dc4c2e623d3ce0f238cbdd8f0edd2f8bd969e04/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDNWLMIERYILTCEMBIUR5IS2JU:/var/lib/containers/storage/overlay/l/WRP4UC5PAX5GPC3RKOGKVPRTK2,upperdir=/var/lib/containers/storage/overlay/dca77b3bbe1c2a422b310a7a5dc4c2e623d3ce0f238cbdd8f0edd2f8bd969e04/diff,workdir=/var/lib/containers/storage/overlay/dca77b3bbe1c2a422b310a7a5dc4c2e623d3ce0f238cbdd8f0edd2f8bd969e04/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/80de66b2d3081fcf1dc3c589cdce18ef7cf5736d32a6827deb7504d06ff841fb/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDNWLMIERYILTCEMBIUR5IS2JU:/var/lib/containers/storage/overlay/l/WRP4UC5PAX5GPC3RKOGKVPRTK2,upperdir=/var/lib/containers/storage/overlay/80de66b2d3081fcf1dc3c589cdce18ef7cf5736d32a6827deb7504d06ff841fb/diff,workdir=/var/lib/containers/storage/overlay/80de66b2d3081fcf1dc3c589cdce18ef7cf5736d32a6827deb7504d06ff841fb/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f4c7eda7a708c93f56e05841f6a280c2eee136b4b16799175efc448536d0908e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDNWLMIERYILTCEMBIUR5IS2JU:/var/lib/containers/storage/overlay/l/WRP4UC5PAX5GPC3RKOGKVPRTK2,upperdir=/var/lib/containers/storage/overlay/f4c7eda7a708c93f56e05841f6a280c2eee136b4b16799175efc448536d0908e/diff,workdir=/var/lib/containers/storage/overlay/f4c7eda7a708c93f56e05841f6a280c2eee136b4b16799175efc448536d0908e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/359cf689c4b9d91817ff1f695d03ef6b3a45fc4c9dde010b53fb5224176e7403/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDNWLMIERYILTCEMBIUR5IS2JU:/var/lib/containers/storage/overlay/l/WRP4UC5PAX5GPC3RKOGKVPRTK2,upperdir=/var/lib/containers/storage/overlay/359cf689c4b9d91817ff1f695d03ef6b3a45fc4c9dde010b53fb5224176e7403/diff,workdir=/var/lib/containers/storage/overlay/359cf689c4b9d91817ff1f695d03ef6b3a45fc4c9dde010b53fb5224176e7403/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/745b86e55a308ba9df046d15b8d6409d68cf8f8d59236aeeb9b02e28aa696722/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDNWLMIERYILTCEMBIUR5IS2JU:/var/lib/containers/storage/overlay/l/WRP4UC5PAX5GPC3RKOGKVPRTK2,upperdir=/var/lib/containers/storage/overlay/745b86e55a308ba9df046d15b8d6409d68cf8f8d59236aeeb9b02e28aa696722/diff,workdir=/var/lib/containers/storage/overlay/745b86e55a308ba9df046d15b8d6409d68cf8f8d59236aeeb9b02e28aa696722/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/05ad0cc5349dd883784c121f738f2de9905adc2a77047f304277bfa81bf892f4/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDNWLMIERYILTCEMBIUR5IS2JU:/var/lib/containers/storage/overlay/l/WRP4UC5PAX5GPC3RKOGKVPRTK2,upperdir=/var/lib/containers/storage/overlay/05ad0cc5349dd883784c121f738f2de9905adc2a77047f304277bfa81bf892f4/diff,workdir=/var/lib/containers/storage/overlay/05ad0cc5349dd883784c121f738f2de9905adc2a77047f304277bfa81bf892f4/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a1485658a8b7ec7e45f05e312ff1894745551355b2172ed2eeb51fd4a53a424c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/IDNWLMIERYILTCEMBIUR5IS2JU:/var/lib/containers/storage/overlay/l/WRP4UC5PAX5GPC3RKOGKVPRTK2,upperdir=/var/lib/containers/storage/overlay/a1485658a8b7ec7e45f05e312ff1894745551355b2172ed2eeb51fd4a53a424c/diff,workdir=/var/lib/containers/storage/overlay/a1485658a8b7ec7e45f05e312ff1894745551355b2172ed2eeb51fd4a53a424c/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-25T17:52:27.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:52:27.667 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-25T17:52:27.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:27 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3746244102' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T17:52:27.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T17:52:27.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-04-25T17:52:27.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:27 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3746244102' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T17:52:27.792 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2024-04-25T17:52:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:52:27.793 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T17:52:27.809 INFO:tasks.cephadm:Teardown begin 2024-04-25T17:52:27.809 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T17:52:27.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:52:28.073 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:52:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:28 smithi098 ceph-mon[29192]: pgmap v523: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2024-04-25T17:52:28.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:28 smithi203 ceph-mon[32834]: pgmap v523: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2024-04-25T17:52:28.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:28 smithi173 ceph-mon[32856]: pgmap v523: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2024-04-25T17:52:29.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.687+0000 7fa919ba2700 1 -- 172.21.15.98:0/1822778100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa914102b00 msgr2=0x7fa914102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.687+0000 7fa919ba2700 1 --2- 172.21.15.98:0/1822778100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa914102b00 0x7fa914102f80 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fa908009990 tx=0x7fa908009ca0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.687+0000 7fa919ba2700 1 -- 172.21.15.98:0/1822778100 shutdown_connections 2024-04-25T17:52:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.687+0000 7fa919ba2700 1 --2- 172.21.15.98:0/1822778100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9141034c0 0x7fa91410f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.687+0000 7fa919ba2700 1 --2- 172.21.15.98:0/1822778100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa914108a10 0x7fa914108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.687+0000 7fa919ba2700 1 --2- 172.21.15.98:0/1822778100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa914102b00 0x7fa914102f80 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.687+0000 7fa919ba2700 1 -- 172.21.15.98:0/1822778100 >> 172.21.15.98:0/1822778100 conn(0x7fa914079c00 msgr2=0x7fa91407a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.688+0000 7fa919ba2700 1 -- 172.21.15.98:0/1822778100 shutdown_connections 2024-04-25T17:52:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.688+0000 7fa919ba2700 1 -- 172.21.15.98:0/1822778100 wait complete. 2024-04-25T17:52:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.688+0000 7fa919ba2700 1 Processor -- start 2024-04-25T17:52:29.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.688+0000 7fa919ba2700 1 -- start start 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.688+0000 7fa919ba2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa914102b00 0x7fa914199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.688+0000 7fa919ba2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9141034c0 0x7fa914199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa919ba2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa914108a10 0x7fa91419d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa919ba2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa914112c10 con 0x7fa914102b00 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa919ba2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa914112a90 con 0x7fa9141034c0 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa919ba2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa914112d90 con 0x7fa914108a10 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa9137fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa914102b00 0x7fa914199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa9137fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa914102b00 0x7fa914199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33194/0 (socket says 172.21.15.98:33194) 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa9137fe700 1 -- 172.21.15.98:0/2162341160 learned_addr learned my addr 172.21.15.98:0/2162341160 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa913fff700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa914108a10 0x7fa91419d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa912ffd700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9141034c0 0x7fa914199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa9137fe700 1 -- 172.21.15.98:0/2162341160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa914108a10 msgr2=0x7fa91419d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa9137fe700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa914108a10 0x7fa91419d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa9137fe700 1 -- 172.21.15.98:0/2162341160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9141034c0 msgr2=0x7fa914199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa9137fe700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9141034c0 0x7fa914199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa9137fe700 1 -- 172.21.15.98:0/2162341160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91419e030 con 0x7fa914102b00 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa912ffd700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9141034c0 0x7fa914199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa9137fe700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa914102b00 0x7fa914199020 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa90400c9b0 tx=0x7fa90400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa910ff9700 1 -- 172.21.15.98:0/2162341160 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa904007b00 con 0x7fa914102b00 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa910ff9700 1 -- 172.21.15.98:0/2162341160 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa904007ca0 con 0x7fa914102b00 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa919ba2700 1 -- 172.21.15.98:0/2162341160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91419e320 con 0x7fa914102b00 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.689+0000 7fa910ff9700 1 -- 172.21.15.98:0/2162341160 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa904018660 con 0x7fa914102b00 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.690+0000 7fa919ba2700 1 -- 172.21.15.98:0/2162341160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9141a5ba0 con 0x7fa914102b00 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.690+0000 7fa919ba2700 1 -- 172.21.15.98:0/2162341160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa914069730 con 0x7fa914102b00 2024-04-25T17:52:29.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.692+0000 7fa910ff9700 1 -- 172.21.15.98:0/2162341160 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa90400f450 con 0x7fa914102b00 2024-04-25T17:52:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.692+0000 7fa910ff9700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8fc0a3db0 0x7fa8fc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.692+0000 7fa912ffd700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8fc0a3db0 0x7fa8fc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.692+0000 7fa910ff9700 1 -- 172.21.15.98:0/2162341160 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8513+0+0 (secure 0 0 0) 0x7fa9040d0a10 con 0x7fa914102b00 2024-04-25T17:52:29.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.693+0000 7fa912ffd700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8fc0a3db0 0x7fa8fc0a6270 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa908006010 tx=0x7fa90800b540 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:29.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.696+0000 7fa910ff9700 1 -- 172.21.15.98:0/2162341160 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa904014030 con 0x7fa914102b00 2024-04-25T17:52:29.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.940+0000 7fa919ba2700 1 -- 172.21.15.98:0/2162341160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa91419a380 con 0x7fa914102b00 2024-04-25T17:52:29.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.941+0000 7fa910ff9700 1 -- 172.21.15.98:0/2162341160 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11878 (secure 0 0 0) 0x7fa90409d140 con 0x7fa914102b00 2024-04-25T17:52:29.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:52:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:51:59.917812+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26026":{"gid":26026,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/3925531495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":3925531495},{"type":"v1","addr":"172.21.15.203:6837","nonce":3925531495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:52:29.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 -- 172.21.15.98:0/2162341160 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8fc0a3db0 msgr2=0x7fa8fc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:29.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8fc0a3db0 0x7fa8fc0a6270 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa908006010 tx=0x7fa90800b540 comp rx=0 tx=0).stop 2024-04-25T17:52:29.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 -- 172.21.15.98:0/2162341160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa914102b00 msgr2=0x7fa914199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:29.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa914102b00 0x7fa914199020 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa90400c9b0 tx=0x7fa90400ccc0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 -- 172.21.15.98:0/2162341160 shutdown_connections 2024-04-25T17:52:29.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fa8fc0a3db0 0x7fa8fc0a6270 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa914108a10 0x7fa91419d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa914102b00 0x7fa914199020 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 --2- 172.21.15.98:0/2162341160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9141034c0 0x7fa914199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:29.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 -- 172.21.15.98:0/2162341160 >> 172.21.15.98:0/2162341160 conn(0x7fa914079c00 msgr2=0x7fa914100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:29.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 -- 172.21.15.98:0/2162341160 shutdown_connections 2024-04-25T17:52:29.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:29.943+0000 7fa919ba2700 1 -- 172.21.15.98:0/2162341160 wait complete. 2024-04-25T17:52:29.943 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 53 2024-04-25T17:52:30.559 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:30 smithi098 ceph-mon[29192]: pgmap v524: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2024-04-25T17:52:30.559 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:30 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2162341160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:52:30.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T17:52:30.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:30 smithi203 ceph-mon[32834]: pgmap v524: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2024-04-25T17:52:30.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:30 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2162341160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:52:30.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:30 smithi173 ceph-mon[32856]: pgmap v524: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2024-04-25T17:52:30.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:30 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2162341160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:52:30.832 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:52:32.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:32 smithi098 ceph-mon[29192]: pgmap v525: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 6 op/s 2024-04-25T17:52:32.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.541+0000 7f5c0742f700 1 -- 172.21.15.98:0/1288050365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c00108a30 msgr2=0x7f5c00108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:32.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.541+0000 7f5c0742f700 1 --2- 172.21.15.98:0/1288050365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c00108a30 0x7f5c00108e10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc009990 tx=0x7f5bfc009ca0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.541+0000 7f5c0742f700 1 -- 172.21.15.98:0/1288050365 shutdown_connections 2024-04-25T17:52:32.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.541+0000 7f5c0742f700 1 --2- 172.21.15.98:0/1288050365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c0006b0f0 0x7f5c0010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.541+0000 7f5c0742f700 1 --2- 172.21.15.98:0/1288050365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c00108a30 0x7f5c00108e10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.541+0000 7f5c0742f700 1 --2- 172.21.15.98:0/1288050365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c0006a750 0x7f5c0006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.541+0000 7f5c0742f700 1 -- 172.21.15.98:0/1288050365 >> 172.21.15.98:0/1288050365 conn(0x7f5c000788f0 msgr2=0x7f5c00078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.541+0000 7f5c0742f700 1 -- 172.21.15.98:0/1288050365 shutdown_connections 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.542+0000 7f5c0742f700 1 -- 172.21.15.98:0/1288050365 wait complete. 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.542+0000 7f5c0742f700 1 Processor -- start 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.542+0000 7f5c0742f700 1 -- start start 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.542+0000 7f5c0742f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c0006a750 0x7f5c001990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.542+0000 7f5c0742f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c0006b0f0 0x7f5c00199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.542+0000 7f5c0742f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c00108a30 0x7f5c0019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.542+0000 7f5c0742f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5c00112c00 con 0x7f5c0006a750 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.542+0000 7f5c0742f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5c00112a80 con 0x7f5c0006b0f0 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.542+0000 7f5c0742f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5c00112d80 con 0x7f5c00108a30 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c051cb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c0006a750 0x7f5c001990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:32.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c051cb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c0006a750 0x7f5c001990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33202/0 (socket says 172.21.15.98:33202) 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c051cb700 1 -- 172.21.15.98:0/2169998971 learned_addr learned my addr 172.21.15.98:0/2169998971 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c059cc700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c00108a30 0x7f5c0019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c049ca700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c0006b0f0 0x7f5c00199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c051cb700 1 -- 172.21.15.98:0/2169998971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c00108a30 msgr2=0x7f5c0019d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c051cb700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c00108a30 0x7f5c0019d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c051cb700 1 -- 172.21.15.98:0/2169998971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c0006b0f0 msgr2=0x7f5c00199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c051cb700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c0006b0f0 0x7f5c00199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c051cb700 1 -- 172.21.15.98:0/2169998971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0019e0b0 con 0x7f5c0006a750 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c051cb700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c0006a750 0x7f5c001990f0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc000c00 tx=0x7f5bfc0057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5bf67fc700 1 -- 172.21.15.98:0/2169998971 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc027070 con 0x7f5c0006a750 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c0742f700 1 -- 172.21.15.98:0/2169998971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0019e340 con 0x7f5c0006a750 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5bf67fc700 1 -- 172.21.15.98:0/2169998971 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bfc003f20 con 0x7f5c0006a750 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5bf67fc700 1 -- 172.21.15.98:0/2169998971 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc022bd0 con 0x7f5c0006a750 2024-04-25T17:52:32.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.543+0000 7f5c0742f700 1 -- 172.21.15.98:0/2169998971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c00101940 con 0x7f5c0006a750 2024-04-25T17:52:32.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.544+0000 7f5c0742f700 1 -- 172.21.15.98:0/2169998971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c00104170 con 0x7f5c0006a750 2024-04-25T17:52:32.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.545+0000 7f5bf67fc700 1 -- 172.21.15.98:0/2169998971 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5bfc02a070 con 0x7f5c0006a750 2024-04-25T17:52:32.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.546+0000 7f5bf67fc700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5bec0a3db0 0x7f5bec0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:32.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.546+0000 7f5bf67fc700 1 -- 172.21.15.98:0/2169998971 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8513+0+0 (secure 0 0 0) 0x7f5bfc0d1ed0 con 0x7f5c0006a750 2024-04-25T17:52:32.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.546+0000 7f5c049ca700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5bec0a3db0 0x7f5bec0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:32.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.546+0000 7f5c049ca700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5bec0a3db0 0x7f5bec0a6270 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005ea0 tx=0x7f5bf0005e30 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:32.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.550+0000 7f5bf67fc700 1 -- 172.21.15.98:0/2169998971 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5bfc031020 con 0x7f5c0006a750 2024-04-25T17:52:32.714 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:32 smithi203 ceph-mon[32834]: pgmap v525: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 6 op/s 2024-04-25T17:52:32.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:32 smithi173 ceph-mon[32856]: pgmap v525: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 6 op/s 2024-04-25T17:52:32.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.798+0000 7f5c0742f700 1 -- 172.21.15.98:0/2169998971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c0019a400 con 0x7f5c0006a750 2024-04-25T17:52:32.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.799+0000 7f5bf67fc700 1 -- 172.21.15.98:0/2169998971 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11878 (secure 0 0 0) 0x7f5bfc013070 con 0x7f5c0006a750 2024-04-25T17:52:32.799 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:52:32.799 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14520,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/682650272","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":682650272},{"type":"v1","addr":"172.21.15.98:6835","nonce":682650272}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3048794856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":3048794856},{"type":"v1","addr":"172.21.15.98:6837","nonce":3048794856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14577,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/672802539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":672802539},{"type":"v1","addr":"172.21.15.98:6839","nonce":672802539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/121413521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":121413521},{"type":"v1","addr":"172.21.15.98:6841","nonce":121413521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2408762956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2408762956},{"type":"v1","addr":"172.21.15.203:6833","nonce":2408762956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2248833890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2248833890},{"type":"v1","addr":"172.21.15.173:6833","nonce":2248833890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T17:32:37.173463+0000","modified":"2024-04-25T17:51:59.917812+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24461,"mds_1":24445,"mds_2":24440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24422":{"gid":24422,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6835/2662617129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2662617129},{"type":"v1","addr":"172.21.15.173:6835","nonce":2662617129}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2803298626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2803298626},{"type":"v1","addr":"172.21.15.203:6835","nonce":2803298626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6837/1096095017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":1096095017},{"type":"v1","addr":"172.21.15.173:6837","nonce":1096095017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6839/1340539385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":1340539385},{"type":"v1","addr":"172.21.15.203:6839","nonce":1340539385}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24461":{"gid":24461,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.173:6839/2342732992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2342732992},{"type":"v1","addr":"172.21.15.173:6839","nonce":2342732992}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26026":{"gid":26026,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6837/3925531495","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":3925531495},{"type":"v1","addr":"172.21.15.203:6837","nonce":3925531495}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24461,"qdb_cluster":[24445,24440,24461]},"id":1}]} 2024-04-25T17:52:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 -- 172.21.15.98:0/2169998971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5bec0a3db0 msgr2=0x7f5bec0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5bec0a3db0 0x7f5bec0a6270 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005ea0 tx=0x7f5bf0005e30 comp rx=0 tx=0).stop 2024-04-25T17:52:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 -- 172.21.15.98:0/2169998971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c0006a750 msgr2=0x7f5c001990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c0006a750 0x7f5c001990f0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc000c00 tx=0x7f5bfc0057a0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 -- 172.21.15.98:0/2169998971 shutdown_connections 2024-04-25T17:52:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5bec0a3db0 0x7f5bec0a6270 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c00108a30 0x7f5c0019d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c0006a750 0x7f5c001990f0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 --2- 172.21.15.98:0/2169998971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c0006b0f0 0x7f5c00199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 -- 172.21.15.98:0/2169998971 >> 172.21.15.98:0/2169998971 conn(0x7f5c000788f0 msgr2=0x7f5c000ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 -- 172.21.15.98:0/2169998971 shutdown_connections 2024-04-25T17:52:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:32.802+0000 7f5c0742f700 1 -- 172.21.15.98:0/2169998971 wait complete. 2024-04-25T17:52:32.803 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 53 2024-04-25T17:52:33.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:33 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/2169998971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:52:33.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T17:52:33.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:33 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/2169998971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:52:33.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:33 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/2169998971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T17:52:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:52:34.611 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:34 smithi098 ceph-mon[29192]: pgmap v526: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 11 op/s 2024-04-25T17:52:34.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:34 smithi203 ceph-mon[32834]: pgmap v526: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 11 op/s 2024-04-25T17:52:34.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:34 smithi173 ceph-mon[32856]: pgmap v526: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 11 op/s 2024-04-25T17:52:35.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.492+0000 7fba2727d700 1 -- 172.21.15.98:0/3335879132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba20103450 msgr2=0x7fba2010f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.492+0000 7fba2727d700 1 --2- 172.21.15.98:0/3335879132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba20103450 0x7fba2010f8b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fba1800b0a0 tx=0x7fba1800b3b0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.492+0000 7fba2727d700 1 -- 172.21.15.98:0/3335879132 shutdown_connections 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.492+0000 7fba2727d700 1 --2- 172.21.15.98:0/3335879132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba20103450 0x7fba2010f8b0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.492+0000 7fba2727d700 1 --2- 172.21.15.98:0/3335879132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba20108a50 0x7fba20108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.492+0000 7fba2727d700 1 --2- 172.21.15.98:0/3335879132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba20102a90 0x7fba20102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.492+0000 7fba2727d700 1 -- 172.21.15.98:0/3335879132 >> 172.21.15.98:0/3335879132 conn(0x7fba20079ba0 msgr2=0x7fba20079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2727d700 1 -- 172.21.15.98:0/3335879132 shutdown_connections 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2727d700 1 -- 172.21.15.98:0/3335879132 wait complete. 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2727d700 1 Processor -- start 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2727d700 1 -- start start 2024-04-25T17:52:35.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2727d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba20102a90 0x7fba20074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2727d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba20103450 0x7fba20075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2727d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba20108a50 0x7fba200757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2727d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba20112c20 con 0x7fba20108a50 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2727d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fba20112aa0 con 0x7fba20102a90 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2727d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba20112da0 con 0x7fba20103450 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2581a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba20108a50 0x7fba200757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2581a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba20108a50 0x7fba200757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52266/0 (socket says 172.21.15.98:52266) 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba2581a700 1 -- 172.21.15.98:0/1728543613 learned_addr learned my addr 172.21.15.98:0/1728543613 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.493+0000 7fba24818700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba20103450 0x7fba20075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba25019700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba20102a90 0x7fba20074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:35.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba2581a700 1 -- 172.21.15.98:0/1728543613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba20103450 msgr2=0x7fba20075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba2581a700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba20103450 0x7fba20075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba2581a700 1 -- 172.21.15.98:0/1728543613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba20102a90 msgr2=0x7fba20074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba2581a700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba20102a90 0x7fba20074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba2581a700 1 -- 172.21.15.98:0/1728543613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba201a75e0 con 0x7fba20108a50 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba25019700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba20102a90 0x7fba20074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba2581a700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba20108a50 0x7fba200757c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fba1800b830 tx=0x7fba18004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba127fc700 1 -- 172.21.15.98:0/1728543613 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18021070 con 0x7fba20108a50 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba127fc700 1 -- 172.21.15.98:0/1728543613 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba18007e30 con 0x7fba20108a50 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba127fc700 1 -- 172.21.15.98:0/1728543613 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba180047d0 con 0x7fba20108a50 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba2727d700 1 -- 172.21.15.98:0/1728543613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba201a7870 con 0x7fba20108a50 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.494+0000 7fba2727d700 1 -- 172.21.15.98:0/1728543613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba201a7db0 con 0x7fba20108a50 2024-04-25T17:52:35.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.496+0000 7fba2727d700 1 -- 172.21.15.98:0/1728543613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba20069730 con 0x7fba20108a50 2024-04-25T17:52:35.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.496+0000 7fba127fc700 1 -- 172.21.15.98:0/1728543613 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fba18007980 con 0x7fba20108a50 2024-04-25T17:52:35.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.497+0000 7fba127fc700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fba0c0a3e30 0x7fba0c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:35.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.497+0000 7fba25019700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fba0c0a3e30 0x7fba0c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:35.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.497+0000 7fba127fc700 1 -- 172.21.15.98:0/1728543613 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8513+0+0 (secure 0 0 0) 0x7fba180d1840 con 0x7fba20108a50 2024-04-25T17:52:35.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.498+0000 7fba25019700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fba0c0a3e30 0x7fba0c0a62f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fba14005e10 tx=0x7fba14005d60 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:35.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.502+0000 7fba127fc700 1 -- 172.21.15.98:0/1728543613 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba1809df70 con 0x7fba20108a50 2024-04-25T17:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.702+0000 7fba2727d700 1 -- 172.21.15.98:0/1728543613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fba2002df80 con 0x7fba20108a50 2024-04-25T17:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.703+0000 7fba127fc700 1 -- 172.21.15.98:0/1728543613 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+23157 (secure 0 0 0) 0x7fba1809d870 con 0x7fba20108a50 2024-04-25T17:52:35.703 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T17:52:35.703 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":116,"fsid":"79754014-0328-11ef-bc93-c7b262605968","created":"2024-04-25T17:24:14.957466+0000","modified":"2024-04-25T17:52:10.207529+0000","last_up_change":"2024-04-25T17:31:29.172572+0000","last_in_change":"2024-04-25T17:31:08.306773+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T17:27:36.969831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T17:32:30.267079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T17:32:33.956902+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"115","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":29,"snap_epoch":115,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"91254e6f-c19b-4a74-8b41-718ef0f5eb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6803","nonce":4172359021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6805","nonce":4172359021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6809","nonce":4172359021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":4172359021},{"type":"v1","addr":"172.21.15.98:6807","nonce":4172359021}]},"public_addr":"172.21.15.98:6803/4172359021","cluster_addr":"172.21.15.98:6805/4172359021","heartbeat_back_addr":"172.21.15.98:6809/4172359021","heartbeat_front_addr":"172.21.15.98:6807/4172359021","state":["exists","up"]},{"osd":1,"uuid":"1450194d-3e02-4d6a-ac19-364cb679d83b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6801","nonce":3686689866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6803","nonce":3686689866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6807","nonce":3686689866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":3686689866},{"type":"v1","addr":"172.21.15.173:6805","nonce":3686689866}]},"public_addr":"172.21.15.173:6801/3686689866","cluster_addr":"172.21.15.173:6803/3686689866","heartbeat_back_addr":"172.21.15.173:6807/3686689866","heartbeat_front_addr":"172.21.15.173:6805/3686689866","state":["exists","up"]},{"osd":2,"uuid":"bf18400e-90d7-46f8-af4a-1abca0ed8066","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6801","nonce":2010273295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6803","nonce":2010273295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6807","nonce":2010273295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2010273295},{"type":"v1","addr":"172.21.15.203:6805","nonce":2010273295}]},"public_addr":"172.21.15.203:6801/2010273295","cluster_addr":"172.21.15.203:6803/2010273295","heartbeat_back_addr":"172.21.15.203:6807/2010273295","heartbeat_front_addr":"172.21.15.203:6805/2010273295","state":["exists","up"]},{"osd":3,"uuid":"e2b72e19-266c-42bf-9cce-045bba17c014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6811","nonce":2541442286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6813","nonce":2541442286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6817","nonce":2541442286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2541442286},{"type":"v1","addr":"172.21.15.98:6815","nonce":2541442286}]},"public_addr":"172.21.15.98:6811/2541442286","cluster_addr":"172.21.15.98:6813/2541442286","heartbeat_back_addr":"172.21.15.98:6817/2541442286","heartbeat_front_addr":"172.21.15.98:6815/2541442286","state":["exists","up"]},{"osd":4,"uuid":"2773c8f9-a178-40dc-9da0-51a8ac5d163b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6809","nonce":2870689522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6811","nonce":2870689522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6815","nonce":2870689522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2870689522},{"type":"v1","addr":"172.21.15.173:6813","nonce":2870689522}]},"public_addr":"172.21.15.173:6809/2870689522","cluster_addr":"172.21.15.173:6811/2870689522","heartbeat_back_addr":"172.21.15.173:6815/2870689522","heartbeat_front_addr":"172.21.15.173:6813/2870689522","state":["exists","up"]},{"osd":5,"uuid":"21751edb-9f9a-4bce-a13a-2e728141a500","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6809","nonce":2904529843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6811","nonce":2904529843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6815","nonce":2904529843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2904529843},{"type":"v1","addr":"172.21.15.203:6813","nonce":2904529843}]},"public_addr":"172.21.15.203:6809/2904529843","cluster_addr":"172.21.15.203:6811/2904529843","heartbeat_back_addr":"172.21.15.203:6815/2904529843","heartbeat_front_addr":"172.21.15.203:6813/2904529843","state":["exists","up"]},{"osd":6,"uuid":"73964ed9-657d-4097-b943-83b96cc1d04e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6819","nonce":3355912265}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6821","nonce":3355912265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6825","nonce":3355912265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3355912265},{"type":"v1","addr":"172.21.15.98:6823","nonce":3355912265}]},"public_addr":"172.21.15.98:6819/3355912265","cluster_addr":"172.21.15.98:6821/3355912265","heartbeat_back_addr":"172.21.15.98:6825/3355912265","heartbeat_front_addr":"172.21.15.98:6823/3355912265","state":["exists","up"]},{"osd":7,"uuid":"6e248182-1de5-462a-a6ab-ec33e5ef7e7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6817","nonce":3192706308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6819","nonce":3192706308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6823","nonce":3192706308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3192706308},{"type":"v1","addr":"172.21.15.173:6821","nonce":3192706308}]},"public_addr":"172.21.15.173:6817/3192706308","cluster_addr":"172.21.15.173:6819/3192706308","heartbeat_back_addr":"172.21.15.173:6823/3192706308","heartbeat_front_addr":"172.21.15.173:6821/3192706308","state":["exists","up"]},{"osd":8,"uuid":"c486a634-e021-4c49-974f-56246017c27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6817","nonce":2633569866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6819","nonce":2633569866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6823","nonce":2633569866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":2633569866},{"type":"v1","addr":"172.21.15.203:6821","nonce":2633569866}]},"public_addr":"172.21.15.203:6817/2633569866","cluster_addr":"172.21.15.203:6819/2633569866","heartbeat_back_addr":"172.21.15.203:6823/2633569866","heartbeat_front_addr":"172.21.15.203:6821/2633569866","state":["exists","up"]},{"osd":9,"uuid":"650eada2-90dc-441b-a6e3-05fde6577647","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6827","nonce":3104319246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6829","nonce":3104319246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6833","nonce":3104319246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3104319246},{"type":"v1","addr":"172.21.15.98:6831","nonce":3104319246}]},"public_addr":"172.21.15.98:6827/3104319246","cluster_addr":"172.21.15.98:6829/3104319246","heartbeat_back_addr":"172.21.15.98:6833/3104319246","heartbeat_front_addr":"172.21.15.98:6831/3104319246","state":["exists","up"]},{"osd":10,"uuid":"cc1b690a-95fa-4a33-a0d4-185446d78a09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6825","nonce":3555470519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6827","nonce":3555470519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6831","nonce":3555470519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3555470519},{"type":"v1","addr":"172.21.15.173:6829","nonce":3555470519}]},"public_addr":"172.21.15.173:6825/3555470519","cluster_addr":"172.21.15.173:6827/3555470519","heartbeat_back_addr":"172.21.15.173:6831/3555470519","heartbeat_front_addr":"172.21.15.173:6829/3555470519","state":["exists","up"]},{"osd":11,"uuid":"44f36bfb-b366-4c58-a814-c4ddda0f32f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6825","nonce":2267615274}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6827","nonce":2267615274}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6831","nonce":2267615274}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2267615274},{"type":"v1","addr":"172.21.15.203:6829","nonce":2267615274}]},"public_addr":"172.21.15.203:6825/2267615274","cluster_addr":"172.21.15.203:6827/2267615274","heartbeat_back_addr":"172.21.15.203:6831/2267615274","heartbeat_front_addr":"172.21.15.203:6829/2267615274","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:26:45.123697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:09.207435+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:33.519281+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:27:59.138023+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:24.361012+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:28:49.766372+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:15.521150+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:29:41.643211+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:07.199725+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:30:33.713238+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:00.092035+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T17:31:27.231108+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/4145487482":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/674729727":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/1530074164":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/2167465649":"2024-04-26T17:35:54.302654+0000","172.21.15.98:6801/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:6800/3170662380":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/1795364490":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/773164127":"2024-04-26T17:25:08.883124+0000","172.21.15.98:0/3206373024":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/3412860846":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6801/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/3143366689":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/4145487482":"2024-04-26T17:35:54.302654+0000","172.21.15.98:0/3075118794":"2024-04-26T17:24:41.622373+0000","172.21.15.98:6800/3033733589":"2024-04-26T17:24:41.622373+0000","172.21.15.98:0/672531975":"2024-04-26T17:25:08.883124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":16,"length":1},{"begin":29,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 -- 172.21.15.98:0/1728543613 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fba0c0a3e30 msgr2=0x7fba0c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fba0c0a3e30 0x7fba0c0a62f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fba14005e10 tx=0x7fba14005d60 comp rx=0 tx=0).stop 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 -- 172.21.15.98:0/1728543613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba20108a50 msgr2=0x7fba200757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba20108a50 0x7fba200757c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fba1800b830 tx=0x7fba18004ce0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 -- 172.21.15.98:0/1728543613 shutdown_connections 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fba0c0a3e30 0x7fba0c0a62f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba20108a50 0x7fba200757c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba20102a90 0x7fba20074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 --2- 172.21.15.98:0/1728543613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba20103450 0x7fba20075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 -- 172.21.15.98:0/1728543613 >> 172.21.15.98:0/1728543613 conn(0x7fba20079ba0 msgr2=0x7fba201017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 -- 172.21.15.98:0/1728543613 shutdown_connections 2024-04-25T17:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:35.705+0000 7fba2727d700 1 -- 172.21.15.98:0/1728543613 wait complete. 2024-04-25T17:52:36.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs fail cephfs 2024-04-25T17:52:36.646 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:52:36.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:36 smithi098 ceph-mon[29192]: pgmap v527: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2024-04-25T17:52:36.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:36 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/1728543613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:52:36.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:36 smithi203 ceph-mon[32834]: pgmap v527: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2024-04-25T17:52:36.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:36 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/1728543613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:52:36.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:36 smithi173 ceph-mon[32856]: pgmap v527: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2024-04-25T17:52:36.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:36 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/1728543613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T17:52:38.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.481+0000 7fdbdb92c700 1 -- 172.21.15.98:0/3487479050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd4107970 msgr2=0x7fdbd406a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:38.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.481+0000 7fdbdb92c700 1 --2- 172.21.15.98:0/3487479050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd4107970 0x7fdbd406a750 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0009990 tx=0x7fdbd0009ca0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.481+0000 7fdbdb92c700 1 -- 172.21.15.98:0/3487479050 shutdown_connections 2024-04-25T17:52:38.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.481+0000 7fdbdb92c700 1 --2- 172.21.15.98:0/3487479050 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbd4105480 0x7fdbd4105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.481+0000 7fdbdb92c700 1 --2- 172.21.15.98:0/3487479050 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd4107970 0x7fdbd406a750 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.481+0000 7fdbdb92c700 1 --2- 172.21.15.98:0/3487479050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbd406ac90 0x7fdbd406b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.481+0000 7fdbdb92c700 1 -- 172.21.15.98:0/3487479050 >> 172.21.15.98:0/3487479050 conn(0x7fdbd40788f0 msgr2=0x7fdbd4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.481+0000 7fdbdb92c700 1 -- 172.21.15.98:0/3487479050 shutdown_connections 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.481+0000 7fdbdb92c700 1 -- 172.21.15.98:0/3487479050 wait complete. 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.481+0000 7fdbdb92c700 1 Processor -- start 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbdb92c700 1 -- start start 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbdb92c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd406ac90 0x7fdbd41a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbdb92c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbd4105480 0x7fdbd41a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbdb92c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbd4107970 0x7fdbd419f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbdb92c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41192a0 con 0x7fdbd406ac90 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbdb92c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd4119120 con 0x7fdbd4107970 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbdb92c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd4119420 con 0x7fdbd4105480 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd96c8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd406ac90 0x7fdbd41a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd96c8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd406ac90 0x7fdbd41a5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52280/0 (socket says 172.21.15.98:52280) 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd96c8700 1 -- 172.21.15.98:0/4106031372 learned_addr learned my addr 172.21.15.98:0/4106031372 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:52:38.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd8ec7700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbd4105480 0x7fdbd41a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd9ec9700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbd4107970 0x7fdbd419f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd96c8700 1 -- 172.21.15.98:0/4106031372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbd4105480 msgr2=0x7fdbd41a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd96c8700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbd4105480 0x7fdbd41a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd96c8700 1 -- 172.21.15.98:0/4106031372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbd4107970 msgr2=0x7fdbd419f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd96c8700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbd4107970 0x7fdbd419f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd96c8700 1 -- 172.21.15.98:0/4106031372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd419fd60 con 0x7fdbd406ac90 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd9ec9700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbd4107970 0x7fdbd419f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbd96c8700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd406ac90 0x7fdbd41a5440 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0000c00 tx=0x7fdbd00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbca7fc700 1 -- 172.21.15.98:0/4106031372 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0026070 con 0x7fdbd406ac90 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.482+0000 7fdbdb92c700 1 -- 172.21.15.98:0/4106031372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd419fff0 con 0x7fdbd406ac90 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.483+0000 7fdbca7fc700 1 -- 172.21.15.98:0/4106031372 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbd0003f20 con 0x7fdbd406ac90 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.483+0000 7fdbdb92c700 1 -- 172.21.15.98:0/4106031372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd41ac310 con 0x7fdbd406ac90 2024-04-25T17:52:38.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.483+0000 7fdbca7fc700 1 -- 172.21.15.98:0/4106031372 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0030060 con 0x7fdbd406ac90 2024-04-25T17:52:38.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.483+0000 7fdbdb92c700 1 -- 172.21.15.98:0/4106031372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbd4069730 con 0x7fdbd406ac90 2024-04-25T17:52:38.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.485+0000 7fdbca7fc700 1 -- 172.21.15.98:0/4106031372 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdbd0029070 con 0x7fdbd406ac90 2024-04-25T17:52:38.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.485+0000 7fdbca7fc700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbc00a3db0 0x7fdbc00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:38.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.486+0000 7fdbd8ec7700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbc00a3db0 0x7fdbc00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:38.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.486+0000 7fdbca7fc700 1 -- 172.21.15.98:0/4106031372 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8513+0+0 (secure 0 0 0) 0x7fdbd00d16f0 con 0x7fdbd406ac90 2024-04-25T17:52:38.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.486+0000 7fdbd8ec7700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbc00a3db0 0x7fdbc00a6270 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4079190 tx=0x7fdbc400a300 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:38.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.489+0000 7fdbca7fc700 1 -- 172.21.15.98:0/4106031372 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdbd0013070 con 0x7fdbd406ac90 2024-04-25T17:52:38.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:38 smithi098 ceph-mon[29192]: pgmap v528: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 10 op/s 2024-04-25T17:52:38.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-mon[32834]: pgmap v528: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 10 op/s 2024-04-25T17:52:38.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.733+0000 7fdbdb92c700 1 -- 172.21.15.98:0/4106031372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fdbd41a0b30 con 0x7fdbd406ac90 2024-04-25T17:52:38.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.737+0000 7fdbca7fc700 1 -- 172.21.15.98:0/4106031372 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v54) ==== 161+0+0 (secure 0 0 0) 0x7fdbd009de20 con 0x7fdbd406ac90 2024-04-25T17:52:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 -- 172.21.15.98:0/4106031372 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbc00a3db0 msgr2=0x7fdbc00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:38.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbc00a3db0 0x7fdbc00a6270 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdbd4079190 tx=0x7fdbc400a300 comp rx=0 tx=0).stop 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 -- 172.21.15.98:0/4106031372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd406ac90 msgr2=0x7fdbd41a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd406ac90 0x7fdbd41a5440 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0000c00 tx=0x7fdbd00057c0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 -- 172.21.15.98:0/4106031372 shutdown_connections 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7fdbc00a3db0 0x7fdbc00a6270 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbd4107970 0x7fdbd419f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdbd406ac90 0x7fdbd41a5440 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 --2- 172.21.15.98:0/4106031372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbd4105480 0x7fdbd41a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 -- 172.21.15.98:0/4106031372 >> 172.21.15.98:0/4106031372 conn(0x7fdbd40788f0 msgr2=0x7fdbd4103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 -- 172.21.15.98:0/4106031372 shutdown_connections 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:38.740+0000 7fdbdb92c700 1 -- 172.21.15.98:0/4106031372 wait complete. 2024-04-25T17:52:38.740 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T17:52:38.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-mon[32856]: pgmap v528: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 10 op/s 2024-04-25T17:52:39.113 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:38 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a[29188]: 2024-04-25T17:52:38.734+0000 7fc90c5c6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T17:52:39.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[70008]: ignoring --setuser ceph since I am not root 2024-04-25T17:52:39.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[70008]: ignoring --setgroup ceph since I am not root 2024-04-25T17:52:39.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-mds[70012]: -- 172.21.15.203:0/1552715617 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55699a26ba40 con 0x55699a23e800 2024-04-25T17:52:39.212 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[70008]: starting mds.c at 2024-04-25T17:52:39.212 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i[63871]: ignoring --setuser ceph since I am not root 2024-04-25T17:52:39.213 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i[63871]: ignoring --setgroup ceph since I am not root 2024-04-25T17:52:39.213 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-mds[63875]: -- 172.21.15.203:0/1955725095 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bb45297a40 con 0x55bb4526ac00 2024-04-25T17:52:39.213 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i[63871]: starting mds.i at 2024-04-25T17:52:39.213 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-f[63249]: ignoring --setuser ceph since I am not root 2024-04-25T17:52:39.213 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-f[63249]: ignoring --setgroup ceph since I am not root 2024-04-25T17:52:39.213 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-mds[63253]: -- 172.21.15.203:0/2096602424 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c0ace9a40 con 0x559c0acbd000 2024-04-25T17:52:39.213 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:52:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-f[63249]: starting mds.f at 2024-04-25T17:52:39.271 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b[63135]: ignoring --setuser ceph since I am not root 2024-04-25T17:52:39.271 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b[63135]: ignoring --setgroup ceph since I am not root 2024-04-25T17:52:39.271 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-mds[63139]: -- 172.21.15.173:0/3055322583 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55986c421a40 con 0x55986c3f4c00 2024-04-25T17:52:39.271 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b[63135]: starting mds.b at 2024-04-25T17:52:39.272 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h[62828]: ignoring --setuser ceph since I am not root 2024-04-25T17:52:39.272 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h[62828]: ignoring --setgroup ceph since I am not root 2024-04-25T17:52:39.272 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-mds[62832]: -- 172.21.15.173:0/3770113732 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fcd7065a40 con 0x55fcd7038800 2024-04-25T17:52:39.272 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h[62828]: starting mds.h at 2024-04-25T17:52:39.272 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-e[62517]: ignoring --setuser ceph since I am not root 2024-04-25T17:52:39.272 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-e[62517]: ignoring --setgroup ceph since I am not root 2024-04-25T17:52:39.272 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-mds[62521]: -- 172.21.15.173:0/1310596359 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5584f4695a40 con 0x5584f4669000 2024-04-25T17:52:39.272 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:52:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-e[62517]: starting mds.e at 2024-04-25T17:52:39.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T17:52:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:52:39.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:39 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4106031372' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T17:52:39.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:39 smithi098 ceph-mon[29192]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T17:52:39.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:39 smithi098 ceph-mon[29192]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T17:52:39.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:39 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/4106031372' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T17:52:39.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:39 smithi098 ceph-mon[29192]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T17:52:39.705 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:39 smithi098 ceph-mon[29192]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T17:52:39.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:39 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/4106031372' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T17:52:39.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:39 smithi203 ceph-mon[32834]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T17:52:39.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:39 smithi203 ceph-mon[32834]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T17:52:39.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:39 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/4106031372' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T17:52:39.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:39 smithi203 ceph-mon[32834]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T17:52:39.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:39 smithi203 ceph-mon[32834]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T17:52:39.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:39 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/4106031372' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T17:52:39.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:39 smithi173 ceph-mon[32856]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T17:52:39.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:39 smithi173 ceph-mon[32856]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T17:52:39.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:39 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/4106031372' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T17:52:39.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:39 smithi173 ceph-mon[32856]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T17:52:39.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:39 smithi173 ceph-mon[32856]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T17:52:40.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: pgmap v529: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.173:6834/4108057166,v1:172.21.15.173:6837/4108057166] up:boot 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.173:6835/3013770651,v1:172.21.15.173:6838/3013770651] up:boot 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.173:6836/616248895,v1:172.21.15.173:6839/616248895] up:boot 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.203:6834/3327169792,v1:172.21.15.203:6835/3327169792] up:boot 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.203:6836/1219289101,v1:172.21.15.203:6838/1219289101] up:boot 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: mds.? [v2:172.21.15.203:6837/4244144401,v1:172.21.15.203:6839/4244144401] up:boot 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T17:52:40.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:40 smithi098 ceph-mon[29192]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: pgmap v529: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.173:6834/4108057166,v1:172.21.15.173:6837/4108057166] up:boot 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.173:6835/3013770651,v1:172.21.15.173:6838/3013770651] up:boot 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.173:6836/616248895,v1:172.21.15.173:6839/616248895] up:boot 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.203:6834/3327169792,v1:172.21.15.203:6835/3327169792] up:boot 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.203:6836/1219289101,v1:172.21.15.203:6838/1219289101] up:boot 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: mds.? [v2:172.21.15.203:6837/4244144401,v1:172.21.15.203:6839/4244144401] up:boot 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T17:52:40.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:52:40.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T17:52:40.713 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:40 smithi203 ceph-mon[32834]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T17:52:40.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: pgmap v529: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2024-04-25T17:52:40.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.173:6834/4108057166,v1:172.21.15.173:6837/4108057166] up:boot 2024-04-25T17:52:40.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.173:6835/3013770651,v1:172.21.15.173:6838/3013770651] up:boot 2024-04-25T17:52:40.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.173:6836/616248895,v1:172.21.15.173:6839/616248895] up:boot 2024-04-25T17:52:40.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.203:6834/3327169792,v1:172.21.15.203:6835/3327169792] up:boot 2024-04-25T17:52:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.203:6836/1219289101,v1:172.21.15.203:6838/1219289101] up:boot 2024-04-25T17:52:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: mds.? [v2:172.21.15.203:6837/4244144401,v1:172.21.15.203:6839/4244144401] up:boot 2024-04-25T17:52:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-25T17:52:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T17:52:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T17:52:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T17:52:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T17:52:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T17:52:40.772 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:40 smithi173 ceph-mon[32856]: from='mgr.15129 172.21.15.98:0/3368013070' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T17:52:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.597+0000 7f06f4340700 1 -- 172.21.15.98:0/197347517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06ec107990 msgr2=0x7f06ec107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.597+0000 7f06f4340700 1 --2- 172.21.15.98:0/197347517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06ec107990 0x7f06ec107d70 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f06e8009990 tx=0x7f06e8009ca0 comp rx=0 tx=0).stop 2024-04-25T17:52:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.597+0000 7f06f4340700 1 -- 172.21.15.98:0/197347517 shutdown_connections 2024-04-25T17:52:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.597+0000 7f06f4340700 1 --2- 172.21.15.98:0/197347517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec102300 0x7f06ec10e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.597+0000 7f06f4340700 1 --2- 172.21.15.98:0/197347517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06ec107990 0x7f06ec107d70 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.597+0000 7f06f4340700 1 --2- 172.21.15.98:0/197347517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06ec101940 0x7f06ec101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.597+0000 7f06f4340700 1 -- 172.21.15.98:0/197347517 >> 172.21.15.98:0/197347517 conn(0x7f06ec0788f0 msgr2=0x7f06ec078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.597+0000 7f06f4340700 1 -- 172.21.15.98:0/197347517 shutdown_connections 2024-04-25T17:52:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.597+0000 7f06f4340700 1 -- 172.21.15.98:0/197347517 wait complete. 2024-04-25T17:52:41.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f4340700 1 Processor -- start 2024-04-25T17:52:41.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f4340700 1 -- start start 2024-04-25T17:52:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f4340700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec101940 0x7f06ec198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f4340700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06ec102300 0x7f06ec1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f4340700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06ec107990 0x7f06ec19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:41.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f4340700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec111b00 con 0x7f06ec107990 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f4340700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec111980 con 0x7f06ec101940 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f4340700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec111c80 con 0x7f06ec102300 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f28dd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06ec107990 0x7f06ec19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f28dd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06ec107990 0x7f06ec19c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52310/0 (socket says 172.21.15.98:52310) 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f28dd700 1 -- 172.21.15.98:0/3508006599 learned_addr learned my addr 172.21.15.98:0/3508006599 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f20dc700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec101940 0x7f06ec198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.598+0000 7f06f18db700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06ec102300 0x7f06ec1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06f28dd700 1 -- 172.21.15.98:0/3508006599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06ec102300 msgr2=0x7f06ec1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06f28dd700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06ec102300 0x7f06ec1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06f28dd700 1 -- 172.21.15.98:0/3508006599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec101940 msgr2=0x7f06ec198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06f28dd700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec101940 0x7f06ec198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06f28dd700 1 -- 172.21.15.98:0/3508006599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06ec19d0a0 con 0x7f06ec107990 2024-04-25T17:52:41.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06f18db700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06ec102300 0x7f06ec1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T17:52:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06f20dc700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec101940 0x7f06ec198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:52:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06f28dd700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06ec107990 0x7f06ec19c960 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f06e4007ee0 tx=0x7f06e400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06e37fe700 1 -- 172.21.15.98:0/3508006599 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e400dbd0 con 0x7f06ec107990 2024-04-25T17:52:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06f4340700 1 -- 172.21.15.98:0/3508006599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06ec19d390 con 0x7f06ec107990 2024-04-25T17:52:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06e37fe700 1 -- 172.21.15.98:0/3508006599 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06e400dd70 con 0x7f06ec107990 2024-04-25T17:52:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.599+0000 7f06f4340700 1 -- 172.21.15.98:0/3508006599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06ec1a4c10 con 0x7f06ec107990 2024-04-25T17:52:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.600+0000 7f06e37fe700 1 -- 172.21.15.98:0/3508006599 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4005990 con 0x7f06ec107990 2024-04-25T17:52:41.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.600+0000 7f06f4340700 1 -- 172.21.15.98:0/3508006599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d0005330 con 0x7f06ec107990 2024-04-25T17:52:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.601+0000 7f06e37fe700 1 -- 172.21.15.98:0/3508006599 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f06e401a020 con 0x7f06ec107990 2024-04-25T17:52:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.602+0000 7f06e37fe700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06d80a3c90 0x7f06d80a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.602+0000 7f06f20dc700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06d80a3c90 0x7f06d80a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.602+0000 7f06e37fe700 1 -- 172.21.15.98:0/3508006599 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8727+0+0 (secure 0 0 0) 0x7f06e40d04a0 con 0x7f06ec107990 2024-04-25T17:52:41.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.603+0000 7f06f20dc700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06d80a3c90 0x7f06d80a6150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f06e8006010 tx=0x7f06e800b540 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:41.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.606+0000 7f06e37fe700 1 -- 172.21.15.98:0/3508006599 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f06e4010070 con 0x7f06ec107990 2024-04-25T17:52:41.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:41.849+0000 7f06f4340700 1 -- 172.21.15.98:0/3508006599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f06d00058f0 con 0x7f06ec107990 2024-04-25T17:52:42.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.385+0000 7f06e37fe700 1 -- 172.21.15.98:0/3508006599 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v56) ==== 104+0+0 (secure 0 0 0) 0x7f06e409ca10 con 0x7f06ec107990 2024-04-25T17:52:42.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.387+0000 7f06f4340700 1 -- 172.21.15.98:0/3508006599 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06d80a3c90 msgr2=0x7f06d80a6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:42.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.387+0000 7f06f4340700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06d80a3c90 0x7f06d80a6150 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f06e8006010 tx=0x7f06e800b540 comp rx=0 tx=0).stop 2024-04-25T17:52:42.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.387+0000 7f06f4340700 1 -- 172.21.15.98:0/3508006599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06ec107990 msgr2=0x7f06ec19c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:42.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.387+0000 7f06f4340700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06ec107990 0x7f06ec19c960 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f06e4007ee0 tx=0x7f06e400d3b0 comp rx=0 tx=0).stop 2024-04-25T17:52:42.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.387+0000 7f06f4340700 1 -- 172.21.15.98:0/3508006599 shutdown_connections 2024-04-25T17:52:42.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.388+0000 7f06f4340700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f06d80a3c90 0x7f06d80a6150 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:42.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.388+0000 7f06f4340700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06ec107990 0x7f06ec19c960 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:42.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.388+0000 7f06f4340700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06ec101940 0x7f06ec198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:42.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.388+0000 7f06f4340700 1 --2- 172.21.15.98:0/3508006599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06ec102300 0x7f06ec1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:42.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.388+0000 7f06f4340700 1 -- 172.21.15.98:0/3508006599 >> 172.21.15.98:0/3508006599 conn(0x7f06ec0788f0 msgr2=0x7f06ec1007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:42.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.388+0000 7f06f4340700 1 -- 172.21.15.98:0/3508006599 shutdown_connections 2024-04-25T17:52:42.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:42.388+0000 7f06f4340700 1 -- 172.21.15.98:0/3508006599 wait complete. 2024-04-25T17:52:42.699 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:42 smithi098 ceph-mon[29192]: pgmap v531: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 9 op/s 2024-04-25T17:52:42.699 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:42 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3508006599' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T17:52:42.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:42 smithi203 ceph-mon[32834]: pgmap v531: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 9 op/s 2024-04-25T17:52:42.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:42 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3508006599' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T17:52:42.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:42 smithi173 ceph-mon[32856]: pgmap v531: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 9 op/s 2024-04-25T17:52:42.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:42 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3508006599' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T17:52:43.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T17:52:43.302 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:52:43.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:43 smithi098 ceph-mon[29192]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T17:52:43.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:43 smithi098 ceph-mon[29192]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T17:52:43.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:43 smithi098 ceph-mon[29192]: Cluster is now healthy 2024-04-25T17:52:43.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:43 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3508006599' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T17:52:43.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:43 smithi098 ceph-mon[29192]: fsmap 12 up:standby 2024-04-25T17:52:43.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:43 smithi203 ceph-mon[32834]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T17:52:43.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:43 smithi203 ceph-mon[32834]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T17:52:43.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:43 smithi203 ceph-mon[32834]: Cluster is now healthy 2024-04-25T17:52:43.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:43 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3508006599' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T17:52:43.712 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:43 smithi203 ceph-mon[32834]: fsmap 12 up:standby 2024-04-25T17:52:43.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:43 smithi173 ceph-mon[32856]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T17:52:43.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:43 smithi173 ceph-mon[32856]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T17:52:43.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:43 smithi173 ceph-mon[32856]: Cluster is now healthy 2024-04-25T17:52:43.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:43 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3508006599' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T17:52:43.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:43 smithi173 ceph-mon[32856]: fsmap 12 up:standby 2024-04-25T17:52:44.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:44 smithi098 ceph-mon[29192]: pgmap v532: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2024-04-25T17:52:44.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:44 smithi203 ceph-mon[32834]: pgmap v532: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2024-04-25T17:52:44.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:44 smithi173 ceph-mon[32856]: pgmap v532: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2024-04-25T17:52:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.025+0000 7f5b5528a700 1 -- 172.21.15.98:0/2224826150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b50107990 msgr2=0x7f5b50107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.025+0000 7f5b5528a700 1 --2- 172.21.15.98:0/2224826150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b50107990 0x7f5b50107d70 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009990 tx=0x7f5b40009ca0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.025+0000 7f5b5528a700 1 -- 172.21.15.98:0/2224826150 shutdown_connections 2024-04-25T17:52:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.025+0000 7f5b5528a700 1 --2- 172.21.15.98:0/2224826150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b50102300 0x7f5b5010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.025+0000 7f5b5528a700 1 --2- 172.21.15.98:0/2224826150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b50107990 0x7f5b50107d70 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.025+0000 7f5b5528a700 1 --2- 172.21.15.98:0/2224826150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b50101940 0x7f5b50101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.025+0000 7f5b5528a700 1 -- 172.21.15.98:0/2224826150 >> 172.21.15.98:0/2224826150 conn(0x7f5b500788f0 msgr2=0x7f5b50078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.025+0000 7f5b5528a700 1 -- 172.21.15.98:0/2224826150 shutdown_connections 2024-04-25T17:52:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.025+0000 7f5b5528a700 1 -- 172.21.15.98:0/2224826150 wait complete. 2024-04-25T17:52:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b5528a700 1 Processor -- start 2024-04-25T17:52:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b5528a700 1 -- start start 2024-04-25T17:52:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b5528a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b50101940 0x7f5b501034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b5528a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b50102300 0x7f5b50103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:45.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b5528a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b50107990 0x7f5b50103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b5528a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5b50111b00 con 0x7f5b50107990 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b5528a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b50111980 con 0x7f5b50101940 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b5528a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5b50111c80 con 0x7f5b50102300 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b4f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b50107990 0x7f5b50103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b4f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b50107990 0x7f5b50103f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52050/0 (socket says 172.21.15.98:52050) 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b4f7fe700 1 -- 172.21.15.98:0/3138008305 learned_addr learned my addr 172.21.15.98:0/3138008305 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b467fc700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b50102300 0x7f5b50103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.026+0000 7f5b4effd700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b50101940 0x7f5b501034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b4f7fe700 1 -- 172.21.15.98:0/3138008305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b50102300 msgr2=0x7f5b50103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b4f7fe700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b50102300 0x7f5b50103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b4f7fe700 1 -- 172.21.15.98:0/3138008305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b50101940 msgr2=0x7f5b501034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b4f7fe700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b50101940 0x7f5b501034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b4f7fe700 1 -- 172.21.15.98:0/3138008305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b501a7600 con 0x7f5b50107990 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b4f7fe700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b50107990 0x7f5b50103f70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5b48007ee0 tx=0x7f5b4800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:45.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b4cff9700 1 -- 172.21.15.98:0/3138008305 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4800dbd0 con 0x7f5b50107990 2024-04-25T17:52:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b4cff9700 1 -- 172.21.15.98:0/3138008305 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b4800dd70 con 0x7f5b50107990 2024-04-25T17:52:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b4cff9700 1 -- 172.21.15.98:0/3138008305 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b48005990 con 0x7f5b50107990 2024-04-25T17:52:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b5528a700 1 -- 172.21.15.98:0/3138008305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b501a78f0 con 0x7f5b50107990 2024-04-25T17:52:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.027+0000 7f5b5528a700 1 -- 172.21.15.98:0/3138008305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b501a7de0 con 0x7f5b50107990 2024-04-25T17:52:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.028+0000 7f5b5528a700 1 -- 172.21.15.98:0/3138008305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b50069730 con 0x7f5b50107990 2024-04-25T17:52:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.029+0000 7f5b4cff9700 1 -- 172.21.15.98:0/3138008305 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5b4801a020 con 0x7f5b50107990 2024-04-25T17:52:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.030+0000 7f5b4cff9700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5b3c0a3db0 0x7f5b3c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.030+0000 7f5b4effd700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5b3c0a3db0 0x7f5b3c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.030+0000 7f5b4cff9700 1 -- 172.21.15.98:0/3138008305 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8727+0+0 (secure 0 0 0) 0x7f5b480d06a0 con 0x7f5b50107990 2024-04-25T17:52:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.030+0000 7f5b4effd700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5b3c0a3db0 0x7f5b3c0a6270 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5b40006010 tx=0x7f5b4001a040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.033+0000 7f5b4cff9700 1 -- 172.21.15.98:0/3138008305 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b48010070 con 0x7f5b50107990 2024-04-25T17:52:45.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.232+0000 7f5b5528a700 1 -- 172.21.15.98:0/3138008305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5b50104a20 con 0x7f5b50107990 2024-04-25T17:52:45.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.407+0000 7f5b4cff9700 1 -- 172.21.15.98:0/3138008305 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) ==== 181+0+0 (secure 0 0 0) 0x7f5b50104a20 con 0x7f5b50107990 2024-04-25T17:52:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.410+0000 7f5b5528a700 1 -- 172.21.15.98:0/3138008305 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5b3c0a3db0 msgr2=0x7f5b3c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.410+0000 7f5b5528a700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5b3c0a3db0 0x7f5b3c0a6270 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5b40006010 tx=0x7f5b4001a040 comp rx=0 tx=0).stop 2024-04-25T17:52:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.410+0000 7f5b5528a700 1 -- 172.21.15.98:0/3138008305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b50107990 msgr2=0x7f5b50103f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.410+0000 7f5b5528a700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b50107990 0x7f5b50103f70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5b48007ee0 tx=0x7f5b4800d3b0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.411+0000 7f5b5528a700 1 -- 172.21.15.98:0/3138008305 shutdown_connections 2024-04-25T17:52:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.411+0000 7f5b5528a700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f5b3c0a3db0 0x7f5b3c0a6270 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.411+0000 7f5b5528a700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b50107990 0x7f5b50103f70 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.411+0000 7f5b5528a700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b50101940 0x7f5b501034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.411+0000 7f5b5528a700 1 --2- 172.21.15.98:0/3138008305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b50102300 0x7f5b50103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.411+0000 7f5b5528a700 1 -- 172.21.15.98:0/3138008305 >> 172.21.15.98:0/3138008305 conn(0x7f5b500788f0 msgr2=0x7f5b5006a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.411+0000 7f5b5528a700 1 -- 172.21.15.98:0/3138008305 shutdown_connections 2024-04-25T17:52:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:45.411+0000 7f5b5528a700 1 -- 172.21.15.98:0/3138008305 wait complete. 2024-04-25T17:52:45.412 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2024-04-25T17:52:45.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:45 smithi098 ceph-mon[29192]: pgmap v533: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2024-04-25T17:52:45.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:45 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3138008305' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T17:52:45.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:45 smithi203 ceph-mon[32834]: pgmap v533: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2024-04-25T17:52:45.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:45 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3138008305' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T17:52:45.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:45 smithi173 ceph-mon[32856]: pgmap v533: 97 pgs: 97 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2024-04-25T17:52:45.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:45 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3138008305' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T17:52:46.375 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid 79754014-0328-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T17:52:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T17:52:46.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:46 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/3138008305' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T17:52:46.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:46 smithi098 ceph-mon[29192]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T17:52:46.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:46 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/3138008305' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T17:52:46.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:46 smithi203 ceph-mon[32834]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T17:52:46.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:46 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/3138008305' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T17:52:46.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:46 smithi173 ceph-mon[32856]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T17:52:47.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:47 smithi098 ceph-mon[29192]: pgmap v535: 65 pgs: 65 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:52:47.711 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:47 smithi203 ceph-mon[32834]: pgmap v535: 65 pgs: 65 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:52:47.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:47 smithi173 ceph-mon[32856]: pgmap v535: 65 pgs: 65 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:52:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.354+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/1750561903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc103450 msgr2=0x7f3bcc10f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:48.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.354+0000 7f3bd0bb5700 1 --2- 172.21.15.98:0/1750561903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc103450 0x7f3bcc10f8b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400b0a0 tx=0x7f3bc400b3b0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.354+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/1750561903 shutdown_connections 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.354+0000 7f3bd0bb5700 1 --2- 172.21.15.98:0/1750561903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc103450 0x7f3bcc10f8b0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.354+0000 7f3bd0bb5700 1 --2- 172.21.15.98:0/1750561903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bcc108a50 0x7f3bcc108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.354+0000 7f3bd0bb5700 1 --2- 172.21.15.98:0/1750561903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bcc102a90 0x7f3bcc102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.354+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/1750561903 >> 172.21.15.98:0/1750561903 conn(0x7f3bcc079ba0 msgr2=0x7f3bcc079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.354+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/1750561903 shutdown_connections 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.354+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/1750561903 wait complete. 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bd0bb5700 1 Processor -- start 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bd0bb5700 1 -- start start 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bd0bb5700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bcc102a90 0x7f3bcc1a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bd0bb5700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bcc103450 0x7f3bcc1a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bd0bb5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc108a50 0x7f3bcc10c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:48.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bd0bb5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc112ce0 con 0x7f3bcc108a50 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bd0bb5700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc112b60 con 0x7f3bcc102a90 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bd0bb5700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc112e60 con 0x7f3bcc103450 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bcad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc108a50 0x7f3bcc10c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bcad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc108a50 0x7f3bcc10c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52076/0 (socket says 172.21.15.98:52076) 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bcad9d700 1 -- 172.21.15.98:0/488455603 learned_addr learned my addr 172.21.15.98:0/488455603 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bca59c700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bcc102a90 0x7f3bcc1a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.355+0000 7f3bc9d9b700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bcc103450 0x7f3bcc1a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bcad9d700 1 -- 172.21.15.98:0/488455603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bcc103450 msgr2=0x7f3bcc1a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bcad9d700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bcc103450 0x7f3bcc1a4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bcad9d700 1 -- 172.21.15.98:0/488455603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bcc102a90 msgr2=0x7f3bcc1a4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bcad9d700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bcc102a90 0x7f3bcc1a4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bcad9d700 1 -- 172.21.15.98:0/488455603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bcc10d160 con 0x7f3bcc108a50 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bca59c700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bcc102a90 0x7f3bcc1a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bcad9d700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc108a50 0x7f3bcc10c9e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3bc40062a0 tx=0x7f3bc4004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:48.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bbb7fe700 1 -- 172.21.15.98:0/488455603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4021070 con 0x7f3bcc108a50 2024-04-25T17:52:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/488455603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bcc10d3f0 con 0x7f3bcc108a50 2024-04-25T17:52:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bbb7fe700 1 -- 172.21.15.98:0/488455603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bc4007d00 con 0x7f3bcc108a50 2024-04-25T17:52:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bbb7fe700 1 -- 172.21.15.98:0/488455603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc401d450 con 0x7f3bcc108a50 2024-04-25T17:52:48.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.356+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/488455603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bcc1ac4b0 con 0x7f3bcc108a50 2024-04-25T17:52:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.357+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/488455603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bcc069730 con 0x7f3bcc108a50 2024-04-25T17:52:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.358+0000 7f3bbb7fe700 1 -- 172.21.15.98:0/488455603 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3bc4004000 con 0x7f3bcc108a50 2024-04-25T17:52:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.359+0000 7f3bbb7fe700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3bb40a3db0 0x7f3bb40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T17:52:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.359+0000 7f3bca59c700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3bb40a3db0 0x7f3bb40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T17:52:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.359+0000 7f3bbb7fe700 1 -- 172.21.15.98:0/488455603 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8173+0+0 (secure 0 0 0) 0x7f3bc400f020 con 0x7f3bcc108a50 2024-04-25T17:52:48.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.359+0000 7f3bca59c700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3bb40a3db0 0x7f3bb40a6270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc1a0fd0 tx=0x7f3bc0008040 comp rx=0 tx=0).ready entity=mgr.15129 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T17:52:48.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.363+0000 7f3bbb7fe700 1 -- 172.21.15.98:0/488455603 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3bc402a020 con 0x7f3bcc108a50 2024-04-25T17:52:48.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.561+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/488455603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3bcc02df80 con 0x7f3bcc108a50 2024-04-25T17:52:48.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.613+0000 7f3bbb7fe700 1 -- 172.21.15.98:0/488455603 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) ==== 169+0+0 (secure 0 0 0) 0x7f3bc409ea00 con 0x7f3bcc108a50 2024-04-25T17:52:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.615+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/488455603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3bb40a3db0 msgr2=0x7f3bb40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.615+0000 7f3bd0bb5700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3bb40a3db0 0x7f3bb40a6270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc1a0fd0 tx=0x7f3bc0008040 comp rx=0 tx=0).stop 2024-04-25T17:52:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.615+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/488455603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc108a50 msgr2=0x7f3bcc10c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T17:52:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.615+0000 7f3bd0bb5700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc108a50 0x7f3bcc10c9e0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3bc40062a0 tx=0x7f3bc4004ed0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.616+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/488455603 shutdown_connections 2024-04-25T17:52:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.616+0000 7f3bd0bb5700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.98:6800/844767972,v1:172.21.15.98:6801/844767972] conn(0x7f3bb40a3db0 0x7f3bb40a6270 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.616+0000 7f3bd0bb5700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc108a50 0x7f3bcc10c9e0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.616+0000 7f3bd0bb5700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bcc102a90 0x7f3bcc1a4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.616+0000 7f3bd0bb5700 1 --2- 172.21.15.98:0/488455603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3bcc103450 0x7f3bcc1a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T17:52:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.616+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/488455603 >> 172.21.15.98:0/488455603 conn(0x7f3bcc079ba0 msgr2=0x7f3bcc10df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T17:52:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.616+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/488455603 shutdown_connections 2024-04-25T17:52:48.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T17:52:48.616+0000 7f3bd0bb5700 1 -- 172.21.15.98:0/488455603 wait complete. 2024-04-25T17:52:48.619 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2024-04-25T17:52:48.954 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:48 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/488455603' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T17:52:48.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:48 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/488455603' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T17:52:49.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:48 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/488455603' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T17:52:49.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T17:52:49.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T17:52:49.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T17:52:49.621 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T17:52:49.621 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T17:52:49.647 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T17:52:49.667 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T17:52:49.690 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T17:52:49.690 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T17:52:49.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mon.a 2024-04-25T17:52:49.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:49 smithi098 ceph-mon[29192]: pgmap v536: 65 pgs: 65 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:52:49.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:49 smithi098 ceph-mon[29192]: from='client.? 172.21.15.98:0/488455603' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T17:52:49.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:49 smithi098 ceph-mon[29192]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T17:52:49.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:49 smithi203 ceph-mon[32834]: pgmap v536: 65 pgs: 65 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:52:49.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:49 smithi203 ceph-mon[32834]: from='client.? 172.21.15.98:0/488455603' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T17:52:49.961 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:49 smithi203 ceph-mon[32834]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T17:52:50.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:49 smithi173 ceph-mon[32856]: pgmap v536: 65 pgs: 65 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T17:52:50.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:49 smithi173 ceph-mon[32856]: from='client.? 172.21.15.98:0/488455603' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T17:52:50.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:49 smithi173 ceph-mon[32856]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T17:52:50.169 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:49 smithi098 systemd[1]: Stopping Ceph mon.a for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:52:50.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a[29188]: 2024-04-25T17:52:50.173+0000 7fc911dd1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:52:50.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a[29188]: 2024-04-25T17:52:50.173+0000 7fc911dd1700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T17:52:50.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:50 smithi098 podman[106606]: 2024-04-25 17:52:50.361340149 +0000 UTC m=+0.339607202 container died 2fd9d01e54efa100d598d8a466c31b356cfc67f9df8336a5e9522851b39dbe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:52:51.127 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:50 smithi098 podman[106606]: 2024-04-25 17:52:50.877057006 +0000 UTC m=+0.855324058 container cleanup 2fd9d01e54efa100d598d8a466c31b356cfc67f9df8336a5e9522851b39dbe6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:52:51.128 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 17:52:50 smithi098 bash[106606]: ceph-79754014-0328-11ef-bc93-c7b262605968-mon-a 2024-04-25T17:52:51.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T17:52:51.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:52:51.351 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T17:52:51.351 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T17:52:51.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mon.b 2024-04-25T17:52:51.771 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:51 smithi173 systemd[1]: Stopping Ceph mon.b for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:52:52.442 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mon-b[32852]: 2024-04-25T17:52:52.187+0000 7fd416f44700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:52:52.442 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mon-b[32852]: 2024-04-25T17:52:52.187+0000 7fd416f44700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T17:52:52.754 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:52 smithi173 podman[70176]: 2024-04-25 17:52:52.443808254 +0000 UTC m=+0.910277372 container died a6b8778b700b2ccf2eaac58c5a4ef4bb867e886b7e2d17b31a31b314111850b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T17:52:53.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:52 smithi173 podman[70176]: 2024-04-25 17:52:52.854854151 +0000 UTC m=+1.321323257 container cleanup a6b8778b700b2ccf2eaac58c5a4ef4bb867e886b7e2d17b31a31b314111850b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:52:53.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 17:52:52 smithi173 bash[70176]: ceph-79754014-0328-11ef-bc93-c7b262605968-mon-b 2024-04-25T17:52:53.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T17:52:53.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:52:53.504 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T17:52:53.504 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T17:52:53.504 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mon.c 2024-04-25T17:52:53.830 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:53 smithi203 systemd[1]: Stopping Ceph mon.c for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:52:54.165 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mon-c[32830]: 2024-04-25T17:52:53.830+0000 7f3c59347700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:52:54.165 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mon-c[32830]: 2024-04-25T17:52:53.830+0000 7f3c59347700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T17:52:54.165 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:53 smithi203 podman[71640]: 2024-04-25 17:52:53.892658568 +0000 UTC m=+0.194290468 container died 6c75708555c392a6a80063dad19099b1dfdff31e33f6c457fe5bf152ed0ca420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T17:52:54.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:54 smithi203 podman[71640]: 2024-04-25 17:52:54.265286523 +0000 UTC m=+0.566918423 container cleanup 6c75708555c392a6a80063dad19099b1dfdff31e33f6c457fe5bf152ed0ca420 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:52:54.461 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 17:52:54 smithi203 bash[71640]: ceph-79754014-0328-11ef-bc93-c7b262605968-mon-c 2024-04-25T17:52:54.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T17:52:54.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:52:54.769 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T17:52:54.769 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T17:52:54.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.x 2024-04-25T17:52:55.081 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:52:54 smithi098 systemd[1]: Stopping Ceph mgr.x for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:52:55.454 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:52:55 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:52:55.081+0000 7f6827158700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:52:55.455 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:52:55 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x[29422]: 2024-04-25T17:52:55.081+0000 7f6827158700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-25T17:52:55.455 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:52:55 smithi098 podman[106736]: 2024-04-25 17:52:55.113009373 +0000 UTC m=+0.150553584 container died d234cf7edff53886131a7ccf9ab3e36263a6e54e0f23ba504b8c2c146e222b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-25T17:52:55.954 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:52:55 smithi098 podman[106736]: 2024-04-25 17:52:55.666858763 +0000 UTC m=+0.704402970 container cleanup d234cf7edff53886131a7ccf9ab3e36263a6e54e0f23ba504b8c2c146e222b60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True) 2024-04-25T17:52:55.954 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 17:52:55 smithi098 bash[106736]: ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-x 2024-04-25T17:52:56.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T17:52:56.862 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:52:56.862 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T17:52:56.862 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T17:52:56.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.y 2024-04-25T17:52:57.271 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:52:56 smithi173 systemd[1]: Stopping Ceph mgr.y for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:52:57.540 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:52:57 smithi173 podman[70299]: 2024-04-25 17:52:57.389350043 +0000 UTC m=+0.338800405 container died 4799370d0d30dc8f63f2936ad7b9bc9df77c809a09b53b3e977576841f0b09a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T17:52:58.021 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:52:57 smithi173 podman[70299]: 2024-04-25 17:52:57.778420695 +0000 UTC m=+0.727871044 container cleanup 4799370d0d30dc8f63f2936ad7b9bc9df77c809a09b53b3e977576841f0b09a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T17:52:58.021 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 17:52:57 smithi173 bash[70299]: ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-y 2024-04-25T17:52:58.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T17:52:58.217 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:52:58.217 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T17:52:58.217 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T17:52:58.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.z 2024-04-25T17:52:58.575 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:52:58 smithi203 systemd[1]: Stopping Ceph mgr.z for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:52:58.846 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:52:58 smithi203 podman[71761]: 2024-04-25 17:52:58.576720657 +0000 UTC m=+0.155117765 container died 00d2f9f988580c123e9b17ac5b68f0f0553d3ff7d9837f34ea998ed7bed03de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T17:52:59.211 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:52:58 smithi203 podman[71761]: 2024-04-25 17:52:58.955012957 +0000 UTC m=+0.533410071 container cleanup 00d2f9f988580c123e9b17ac5b68f0f0553d3ff7d9837f34ea998ed7bed03de8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T17:52:59.211 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 17:52:58 smithi203 bash[71761]: ceph-79754014-0328-11ef-bc93-c7b262605968-mgr-z 2024-04-25T17:52:59.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T17:52:59.428 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:52:59.429 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T17:52:59.429 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T17:52:59.429 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.0 2024-04-25T17:52:59.704 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:52:59 smithi098 systemd[1]: Stopping Ceph osd.0 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:53:00.204 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:52:59 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0[44416]: 2024-04-25T17:52:59.872+0000 7f9875c02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:53:00.204 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:52:59 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0[44416]: 2024-04-25T17:52:59.872+0000 7f9875c02700 -1 osd.0 119 *** Got signal Terminated *** 2024-04-25T17:53:00.205 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:52:59 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0[44416]: 2024-04-25T17:52:59.872+0000 7f9875c02700 -1 osd.0 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:53:05.772 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:53:05 smithi098 podman[106856]: 2024-04-25 17:53:05.475958074 +0000 UTC m=+5.855674837 container died f9817654b41bbebcbf8d5414c7bec122d0587f9354d8790bdbdaf315e39e5ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:53:06.123 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:53:05 smithi098 podman[106856]: 2024-04-25 17:53:05.87333208 +0000 UTC m=+6.253048854 container cleanup f9817654b41bbebcbf8d5414c7bec122d0587f9354d8790bdbdaf315e39e5ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:53:06.124 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:53:05 smithi098 bash[106856]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0 2024-04-25T17:53:06.454 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:53:06 smithi098 podman[106874]: 2024-04-25 17:53:06.125167577 +0000 UTC m=+0.651748080 container remove f9817654b41bbebcbf8d5414c7bec122d0587f9354d8790bdbdaf315e39e5ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T17:53:06.769 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:53:06 smithi098 podman[106967]: 2024-04-25 17:53:06.388548971 +0000 UTC m=+0.018164537 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:53:06.769 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:53:06 smithi098 podman[106967]: 2024-04-25 17:53:06.509672705 +0000 UTC m=+0.139288271 container create d12a8b7fdf59483fab1ac60995086bc69245d4352a884e470bd12bae9c09519f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T17:53:06.769 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:53:06 smithi098 podman[106967]: 2024-04-25 17:53:06.770592864 +0000 UTC m=+0.400208430 container init d12a8b7fdf59483fab1ac60995086bc69245d4352a884e470bd12bae9c09519f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T17:53:07.061 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:53:06 smithi098 podman[106967]: 2024-04-25 17:53:06.773777577 +0000 UTC m=+0.403393152 container start d12a8b7fdf59483fab1ac60995086bc69245d4352a884e470bd12bae9c09519f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:53:07.061 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:53:06 smithi098 podman[106967]: 2024-04-25 17:53:06.870683185 +0000 UTC m=+0.500298765 container attach d12a8b7fdf59483fab1ac60995086bc69245d4352a884e470bd12bae9c09519f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T17:53:07.061 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 17:53:07 smithi098 podman[106967]: 2024-04-25 17:53:07.062615718 +0000 UTC m=+0.692231289 container died d12a8b7fdf59483fab1ac60995086bc69245d4352a884e470bd12bae9c09519f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-25T17:53:07.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T17:53:07.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:53:07.690 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T17:53:07.690 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T17:53:07.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.1 2024-04-25T17:53:07.982 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:07 smithi173 systemd[1]: Stopping Ceph osd.1 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:53:08.271 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:08 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1[37499]: 2024-04-25T17:53:08.008+0000 7f023926d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:53:08.271 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:08 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1[37499]: 2024-04-25T17:53:08.008+0000 7f023926d700 -1 osd.1 119 *** Got signal Terminated *** 2024-04-25T17:53:08.271 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:08 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1[37499]: 2024-04-25T17:53:08.008+0000 7f023926d700 -1 osd.1 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:53:13.771 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:13 smithi173 podman[70437]: 2024-04-25 17:53:13.499629525 +0000 UTC m=+5.611108657 container died 9c8e8c3a8c16470f2b0850515240fdc899b40cf03b1f7b68eb7e074dd71c0d9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-04-25T17:53:14.151 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:13 smithi173 podman[70437]: 2024-04-25 17:53:13.880089083 +0000 UTC m=+5.991568216 container cleanup 9c8e8c3a8c16470f2b0850515240fdc899b40cf03b1f7b68eb7e074dd71c0d9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T17:53:14.152 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:13 smithi173 bash[70437]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1 2024-04-25T17:53:14.488 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:14 smithi173 podman[70454]: 2024-04-25 17:53:14.153395751 +0000 UTC m=+0.655098544 container remove 9c8e8c3a8c16470f2b0850515240fdc899b40cf03b1f7b68eb7e074dd71c0d9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-04-25T17:53:14.771 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:14 smithi173 podman[70546]: 2024-04-25 17:53:14.392811846 +0000 UTC m=+0.017817520 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:53:14.771 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:14 smithi173 podman[70546]: 2024-04-25 17:53:14.521913701 +0000 UTC m=+0.146919427 container create ef6ecef772ea13dea1d57f2551abc81859d86703c86929343d467336ac1a5f68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-25T17:53:15.065 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:14 smithi173 podman[70546]: 2024-04-25 17:53:14.775538343 +0000 UTC m=+0.400544005 container init ef6ecef772ea13dea1d57f2551abc81859d86703c86929343d467336ac1a5f68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True) 2024-04-25T17:53:15.065 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:14 smithi173 podman[70546]: 2024-04-25 17:53:14.778942932 +0000 UTC m=+0.403948597 container start ef6ecef772ea13dea1d57f2551abc81859d86703c86929343d467336ac1a5f68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-25T17:53:15.065 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:14 smithi173 podman[70546]: 2024-04-25 17:53:14.883662106 +0000 UTC m=+0.508667783 container attach ef6ecef772ea13dea1d57f2551abc81859d86703c86929343d467336ac1a5f68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T17:53:15.341 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 17:53:15 smithi173 podman[70546]: 2024-04-25 17:53:15.066697829 +0000 UTC m=+0.691703486 container died ef6ecef772ea13dea1d57f2551abc81859d86703c86929343d467336ac1a5f68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:53:15.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T17:53:15.823 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:53:15.824 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T17:53:15.824 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T17:53:15.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.2 2024-04-25T17:53:16.160 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:15 smithi203 systemd[1]: Stopping Ceph osd.2 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:53:16.461 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2[37999]: 2024-04-25T17:53:16.160+0000 7fe7dfb12700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:53:16.461 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2[37999]: 2024-04-25T17:53:16.160+0000 7fe7dfb12700 -1 osd.2 119 *** Got signal Terminated *** 2024-04-25T17:53:16.461 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2[37999]: 2024-04-25T17:53:16.160+0000 7fe7dfb12700 -1 osd.2 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:53:22.102 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:21 smithi203 podman[71897]: 2024-04-25 17:53:21.766401303 +0000 UTC m=+5.744282998 container died b787bc3f334daed4bb7b860135d93226a02e0c2465cccc323f8a32089328ab6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-25T17:53:22.454 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:22 smithi203 podman[71897]: 2024-04-25 17:53:22.202974784 +0000 UTC m=+6.180856478 container cleanup b787bc3f334daed4bb7b860135d93226a02e0c2465cccc323f8a32089328ab6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T17:53:22.454 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:22 smithi203 bash[71897]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2 2024-04-25T17:53:22.711 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:22 smithi203 podman[71914]: 2024-04-25 17:53:22.455699009 +0000 UTC m=+0.687627890 container remove b787bc3f334daed4bb7b860135d93226a02e0c2465cccc323f8a32089328ab6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-25T17:53:23.193 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:22 smithi203 podman[72006]: 2024-04-25 17:53:22.799794481 +0000 UTC m=+0.018051578 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:53:23.193 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:22 smithi203 podman[72006]: 2024-04-25 17:53:22.92454092 +0000 UTC m=+0.142798069 container create 4e7248a907771bdaa51bb53dbbb21aa509ce64d457517db604b2d0f6100dd04f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T17:53:23.461 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:23 smithi203 podman[72006]: 2024-04-25 17:53:23.194892325 +0000 UTC m=+0.413149412 container init 4e7248a907771bdaa51bb53dbbb21aa509ce64d457517db604b2d0f6100dd04f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:53:23.462 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:23 smithi203 podman[72006]: 2024-04-25 17:53:23.198379447 +0000 UTC m=+0.416636538 container start 4e7248a907771bdaa51bb53dbbb21aa509ce64d457517db604b2d0f6100dd04f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T17:53:23.462 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:23 smithi203 podman[72006]: 2024-04-25 17:53:23.303517761 +0000 UTC m=+0.521774867 container attach 4e7248a907771bdaa51bb53dbbb21aa509ce64d457517db604b2d0f6100dd04f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T17:53:23.806 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 17:53:23 smithi203 podman[72006]: 2024-04-25 17:53:23.497729986 +0000 UTC m=+0.715987073 container died 4e7248a907771bdaa51bb53dbbb21aa509ce64d457517db604b2d0f6100dd04f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T17:53:24.151 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T17:53:24.191 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:53:24.192 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T17:53:24.192 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T17:53:24.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.3 2024-04-25T17:53:24.501 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:24 smithi098 systemd[1]: Stopping Ceph osd.3 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:53:24.759 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:24 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3[51380]: 2024-04-25T17:53:24.502+0000 7fe8b24f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:53:24.760 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:24 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3[51380]: 2024-04-25T17:53:24.502+0000 7fe8b24f8700 -1 osd.3 119 *** Got signal Terminated *** 2024-04-25T17:53:24.760 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:24 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3[51380]: 2024-04-25T17:53:24.502+0000 7fe8b24f8700 -1 osd.3 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:53:26.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:25.978+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:26.704 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:26 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3[51380]: 2024-04-25T17:53:26.251+0000 7fe8aeaec700 -1 osd.3 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.278973+0000 front 2024-04-25T17:53:02.278762+0000 (oldest deadline 2024-04-25T17:53:25.778809+0000) 2024-04-25T17:53:27.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:26.950+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:27.704 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:27 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3[51380]: 2024-04-25T17:53:27.287+0000 7fe8aeaec700 -1 osd.3 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.278973+0000 front 2024-04-25T17:53:02.278762+0000 (oldest deadline 2024-04-25T17:53:25.778809+0000) 2024-04-25T17:53:27.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:27 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:27.707+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:28.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:27.974+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:28.704 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:28 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3[51380]: 2024-04-25T17:53:28.284+0000 7fe8aeaec700 -1 osd.3 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.278973+0000 front 2024-04-25T17:53:02.278762+0000 (oldest deadline 2024-04-25T17:53:25.778809+0000) 2024-04-25T17:53:28.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:28 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:28.668+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:29.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:28 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:28.961+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:29.455 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:29 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3[51380]: 2024-04-25T17:53:29.316+0000 7fe8aeaec700 -1 osd.3 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.278973+0000 front 2024-04-25T17:53:02.278762+0000 (oldest deadline 2024-04-25T17:53:25.778809+0000) 2024-04-25T17:53:29.455 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:29 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:29.121+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:29.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:29 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:29.581+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:29.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:29 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:29.715+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:30.205 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:30 smithi098 podman[107084]: 2024-04-25 17:53:30.064860266 +0000 UTC m=+5.674283384 container died 751eb5674678685aac72c317524c0f161c0c61a7b0271a7b7ad21e6844f260de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True) 2024-04-25T17:53:30.208 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:30 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:30.092+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:30.208 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:29 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:29.912+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:30.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:29 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:29.950+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:30.704 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:30 smithi098 podman[107084]: 2024-04-25 17:53:30.60352042 +0000 UTC m=+6.212943569 container cleanup 751eb5674678685aac72c317524c0f161c0c61a7b0271a7b7ad21e6844f260de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:53:30.704 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:30 smithi098 bash[107084]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3 2024-04-25T17:53:30.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:30.570+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:30.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:30.759+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:31.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:30 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:30.932+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:31.205 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:30 smithi098 podman[107113]: 2024-04-25 17:53:30.8721379 +0000 UTC m=+0.805725436 container remove 751eb5674678685aac72c317524c0f161c0c61a7b0271a7b7ad21e6844f260de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-25T17:53:31.205 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:31 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:31.085+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:31.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:30 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:30.960+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:31.666 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:31 smithi098 podman[107201]: 2024-04-25 17:53:31.261872471 +0000 UTC m=+0.017412891 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:53:31.666 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:31 smithi098 podman[107201]: 2024-04-25 17:53:31.406545352 +0000 UTC m=+0.162085751 container create 65ba2af2448021895f7fd35c1cc7b2fbc261a155aad7b88afd3f625003adc13c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T17:53:31.954 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:31 smithi098 podman[107201]: 2024-04-25 17:53:31.667839977 +0000 UTC m=+0.423380375 container init 65ba2af2448021895f7fd35c1cc7b2fbc261a155aad7b88afd3f625003adc13c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:53:31.955 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:31 smithi098 podman[107201]: 2024-04-25 17:53:31.671226494 +0000 UTC m=+0.426766888 container start 65ba2af2448021895f7fd35c1cc7b2fbc261a155aad7b88afd3f625003adc13c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-25T17:53:31.955 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:31 smithi098 podman[107201]: 2024-04-25 17:53:31.775926046 +0000 UTC m=+0.531466451 container attach 65ba2af2448021895f7fd35c1cc7b2fbc261a155aad7b88afd3f625003adc13c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T17:53:31.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:31 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:31.558+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:31.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:31 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:31.743+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:32.269 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:32 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:32.100+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:32.269 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:31 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:31.977+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:32.270 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 17:53:31 smithi098 podman[107201]: 2024-04-25 17:53:31.961813294 +0000 UTC m=+0.717353706 container died 65ba2af2448021895f7fd35c1cc7b2fbc261a155aad7b88afd3f625003adc13c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T17:53:32.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:31 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:31.925+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:32.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T17:53:32.659 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:53:32.660 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T17:53:32.660 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T17:53:32.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.4 2024-04-25T17:53:32.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:32 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:32.607+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:32.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:32 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:32.787+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:32.982 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:32 smithi173 systemd[1]: Stopping Ceph osd.4 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:53:32.982 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:32 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:32.889+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:33.271 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:32 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4[44145]: 2024-04-25T17:53:32.983+0000 7fcff0da0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:53:33.271 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:32 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4[44145]: 2024-04-25T17:53:32.983+0000 7fcff0da0700 -1 osd.4 119 *** Got signal Terminated *** 2024-04-25T17:53:33.271 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:32 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4[44145]: 2024-04-25T17:53:32.983+0000 7fcff0da0700 -1 osd.4 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:53:33.454 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:33 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:33.143+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:33.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:33 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:33.000+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:33.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:33 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:33.756+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:33.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:33 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:33.649+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:33.962 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:33 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:33.579+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:34.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:33 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:33.862+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:34.454 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:34 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:34.096+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:34.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:33 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:33.986+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:34.835 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:34.601+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:34.835 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:34.533+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:34.835 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:34.719+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:35.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:34 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:34.878+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:35.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:35 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:35.003+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:35.454 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:35 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:35.064+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:35.454 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:35 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:35.064+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:35.698 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:35 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:35.554+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:35.698 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:35 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:35.520+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:35.771 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:35 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4[44145]: 2024-04-25T17:53:35.330+0000 7fcfe938c700 -1 osd.4 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.574469+0000 front 2024-04-25T17:53:12.574502+0000 (oldest deadline 2024-04-25T17:53:34.874597+0000) 2024-04-25T17:53:35.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:35 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:35.699+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:36.017 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:35 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:35.977+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:36.017 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:35 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:35.977+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:36.250 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:35 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:35.880+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:36.454 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:36 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:36.018+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:36.454 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:36 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:36.018+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:36.521 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:36 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4[44145]: 2024-04-25T17:53:36.336+0000 7fcfe938c700 -1 osd.4 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.574469+0000 front 2024-04-25T17:53:12.574502+0000 (oldest deadline 2024-04-25T17:53:34.874597+0000) 2024-04-25T17:53:36.521 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:36 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:36.251+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:36.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:36 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:36.667+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:36.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:36 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:36.569+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:36.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:36 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:36.569+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.042540+0000 front 2024-04-25T17:53:10.042382+0000 (oldest deadline 2024-04-25T17:53:35.942309+0000) 2024-04-25T17:53:36.962 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:36 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:36.495+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:37.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:37 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:37.040+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:37.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:37 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:37.040+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:37.205 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:36 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:36.840+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:37.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:36 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:36.952+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:37.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:36 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:36.952+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:37.521 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:37 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4[44145]: 2024-04-25T17:53:37.338+0000 7fcfe938c700 -1 osd.4 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.574469+0000 front 2024-04-25T17:53:12.574502+0000 (oldest deadline 2024-04-25T17:53:34.874597+0000) 2024-04-25T17:53:37.521 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:37 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:37.204+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:37.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:37 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:37.540+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:37.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:37 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:37.623+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:37.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:37 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:37.623+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:37.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:37 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:37.532+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:37.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:37 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:37.532+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.042540+0000 front 2024-04-25T17:53:10.042382+0000 (oldest deadline 2024-04-25T17:53:35.942309+0000) 2024-04-25T17:53:38.173 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:37 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:37.806+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:38.173 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:37 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:37.806+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:38.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:38 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:38.061+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:38.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:38 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:38.061+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:38.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:37 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:37.943+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:38.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:37 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:37.943+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:38.521 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:38.173+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:38.950 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:38 smithi173 podman[70677]: 2024-04-25 17:53:38.644177209 +0000 UTC m=+5.784780522 container died 93d596e474a4404220d10bebac0cf53a67aea9b2f12f9c283b160109453762be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T17:53:38.951 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:38.805+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:38.951 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:38 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:38.805+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:38.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:38.590+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:38.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:38.658+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:38.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:38.658+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:38.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:38.573+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:38.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:38 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:38.573+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.042540+0000 front 2024-04-25T17:53:10.042382+0000 (oldest deadline 2024-04-25T17:53:35.942309+0000) 2024-04-25T17:53:39.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:39 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:39.016+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:39.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:39 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:39.016+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:39.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:38 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:38.950+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:39.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:38 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:38.950+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:39.271 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:39 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:39.131+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:39.271 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:39 smithi173 podman[70677]: 2024-04-25 17:53:39.050864471 +0000 UTC m=+6.191467777 container cleanup 93d596e474a4404220d10bebac0cf53a67aea9b2f12f9c283b160109453762be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T17:53:39.271 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:39 smithi173 bash[70677]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4 2024-04-25T17:53:39.659 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:39 smithi173 podman[70694]: 2024-04-25 17:53:39.336501046 +0000 UTC m=+0.690216305 container remove 93d596e474a4404220d10bebac0cf53a67aea9b2f12f9c283b160109453762be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T17:53:39.922 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:39 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:39.816+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:39.922 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:39 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:39.816+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:39.922 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:39 smithi173 podman[70786]: 2024-04-25 17:53:39.563956541 +0000 UTC m=+0.018683982 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:53:39.922 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:39 smithi173 podman[70786]: 2024-04-25 17:53:39.687566896 +0000 UTC m=+0.142294314 container create 12b3086dfea2f80e5b3310846bf668a19c08ae1d0c01ea1d6c93f85912246b47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:53:39.922 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:39 smithi173 podman[70786]: 2024-04-25 17:53:39.923804088 +0000 UTC m=+0.378531506 container init 12b3086dfea2f80e5b3310846bf668a19c08ae1d0c01ea1d6c93f85912246b47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-25T17:53:39.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:39 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:39.592+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:39.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:39 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:39.576+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:39.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:39 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:39.576+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.042540+0000 front 2024-04-25T17:53:10.042382+0000 (oldest deadline 2024-04-25T17:53:35.942309+0000) 2024-04-25T17:53:39.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:39 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:39.653+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:39.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:39 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:39.653+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:40.214 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:40 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:40.108+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:40.214 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:39 smithi173 podman[70786]: 2024-04-25 17:53:39.927103151 +0000 UTC m=+0.381830570 container start 12b3086dfea2f80e5b3310846bf668a19c08ae1d0c01ea1d6c93f85912246b47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-25T17:53:40.214 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:40 smithi173 podman[70786]: 2024-04-25 17:53:40.032344453 +0000 UTC m=+0.487071885 container attach 12b3086dfea2f80e5b3310846bf668a19c08ae1d0c01ea1d6c93f85912246b47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T17:53:40.454 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:40.004+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:40.454 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:40.004+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:40.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:39 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:39.977+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:40.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:39 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:39.977+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:40.521 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 17:53:40 smithi173 podman[70786]: 2024-04-25 17:53:40.21594756 +0000 UTC m=+0.670674982 container died 12b3086dfea2f80e5b3310846bf668a19c08ae1d0c01ea1d6c93f85912246b47 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T17:53:40.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T17:53:40.895 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:53:40.895 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T17:53:40.895 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T17:53:40.895 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.5 2024-04-25T17:53:40.922 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:40 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:40.624+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:40.922 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:40 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:40.599+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:40.922 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:40 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:40.599+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.042540+0000 front 2024-04-25T17:53:10.042382+0000 (oldest deadline 2024-04-25T17:53:35.942309+0000) 2024-04-25T17:53:40.922 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:40 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:40.668+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:40.922 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:40 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:40.668+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:41.148 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:40 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:40.842+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:41.148 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:40 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:40.842+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:41.211 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:40 smithi203 systemd[1]: Stopping Ceph osd.5 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:53:41.454 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:40.982+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:41.454 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:40.982+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:41.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:40.997+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:41.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:40 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:40.997+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:41.521 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:41 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:41.148+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:41.562 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:41 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:41.219+0000 7f5ac5162700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:53:41.562 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:41 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:41.219+0000 7f5ac5162700 -1 osd.5 119 *** Got signal Terminated *** 2024-04-25T17:53:41.562 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:41 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:41.219+0000 7f5ac5162700 -1 osd.5 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:53:41.562 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:41 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:41.563+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:41.562 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:41 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:41.563+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.042540+0000 front 2024-04-25T17:53:10.042382+0000 (oldest deadline 2024-04-25T17:53:35.942309+0000) 2024-04-25T17:53:41.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:41 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:41.663+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:41.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:41 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:41.660+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:41.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:41 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:41.660+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:42.197 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:41 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:41.874+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:42.197 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:41 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:41.874+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:42.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:42 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:42.028+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:42.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:42 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:42.028+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:42.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:41 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:41.943+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:42.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:41 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:41.943+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:42.521 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:42 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:42.197+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:42.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:42 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:42.620+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:42.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:42 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:42.599+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:42.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:42 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:42.599+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.042540+0000 front 2024-04-25T17:53:10.042382+0000 (oldest deadline 2024-04-25T17:53:35.942309+0000) 2024-04-25T17:53:42.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:42 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:42.650+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:42.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:42 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:42.650+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:43.193 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:42 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:42.827+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:43.193 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:42 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:42.827+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:43.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:42 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:42.900+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:43.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:42 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:42.900+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:43.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:43 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:43.010+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:43.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:43 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:43.010+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:43.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:43 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:43.010+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:43.521 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:43 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:43.194+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:43.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:43 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:43.602+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:43.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:43 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:43.646+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:43.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:43 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:43.646+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.042540+0000 front 2024-04-25T17:53:10.042382+0000 (oldest deadline 2024-04-25T17:53:35.942309+0000) 2024-04-25T17:53:43.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:43 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:43.646+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.443187+0000 front 2024-04-25T17:53:19.443214+0000 (oldest deadline 2024-04-25T17:53:43.543327+0000) 2024-04-25T17:53:43.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:43 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:43.659+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:43.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:43 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:43.659+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:43.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:43 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:43.659+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:44.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:43 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:43.868+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:44.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:43 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:43.868+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:44.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:44 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:44.041+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:44.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:44 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:44.041+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:44.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:44 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:44.041+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:44.228 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:43 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:43.876+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:44.228 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:43 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:43.876+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:44.228 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:43 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:43.876+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:44.521 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:44 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:44.228+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:44.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:44 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:44.588+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:44.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:44 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:44.683+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:44.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:44 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:44.683+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.042540+0000 front 2024-04-25T17:53:10.042382+0000 (oldest deadline 2024-04-25T17:53:35.942309+0000) 2024-04-25T17:53:44.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:44 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:44.683+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.443187+0000 front 2024-04-25T17:53:19.443214+0000 (oldest deadline 2024-04-25T17:53:43.543327+0000) 2024-04-25T17:53:44.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:44 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:44.664+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:44.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:44 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:44.664+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:44.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:44 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:44.664+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:45.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:44 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:44.840+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:45.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:44 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:44.840+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:45.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:44 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:44.995+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:45.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:44 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:44.995+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:45.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:44 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:44.995+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:45.266 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:44 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:44.902+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:45.266 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:44 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:44.902+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:45.266 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:44 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:44.902+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:45.521 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:45 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:45.266+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:45.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:45 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:45.546+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:45.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:45 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:45.666+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.841365+0000 front 2024-04-25T17:53:04.841435+0000 (oldest deadline 2024-04-25T17:53:28.941172+0000) 2024-04-25T17:53:45.961 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:45 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:45.666+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.042540+0000 front 2024-04-25T17:53:10.042382+0000 (oldest deadline 2024-04-25T17:53:35.942309+0000) 2024-04-25T17:53:45.962 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:45 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5[44513]: 2024-04-25T17:53:45.666+0000 7f5abd74e700 -1 osd.5 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.443187+0000 front 2024-04-25T17:53:19.443214+0000 (oldest deadline 2024-04-25T17:53:43.543327+0000) 2024-04-25T17:53:45.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:45 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:45.694+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:45.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:45 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:45.694+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:45.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:45 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:45.694+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:46.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:45 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:45.844+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:46.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:45 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:45.844+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:46.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:45 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:45.844+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.458202+0000 front 2024-04-25T17:53:19.458333+0000 (oldest deadline 2024-04-25T17:53:45.358280+0000) 2024-04-25T17:53:46.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:45 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:45.954+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:46.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:45 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:45.954+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:46.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:45 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:45.954+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:46.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:45 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:45.874+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:46.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:45 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:45.874+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:46.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:45 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:45.874+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:46.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:46 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:46.286+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:46.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:46 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:46.286+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:46.855 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:46 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:46.571+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:46.855 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:46 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:46.571+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:46.855 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:46 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:46.711+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:46.855 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:46 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:46.711+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:46.856 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:46 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:46.711+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:47.177 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:46 smithi203 podman[72124]: 2024-04-25 17:53:46.856498221 +0000 UTC m=+5.767836639 container died b266277954d20eda77250d8c283e9a14dda5714e56735ce9ef86d6df824d9e9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-25T17:53:47.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:46.855+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:47.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:46.855+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:47.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:46.855+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.458202+0000 front 2024-04-25T17:53:19.458333+0000 (oldest deadline 2024-04-25T17:53:45.358280+0000) 2024-04-25T17:53:47.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:46.969+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:47.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:46.969+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:47.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:46.969+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:47.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:46 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:46.845+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:47.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:46 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:46.845+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:47.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:46 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:46.845+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:47.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:47 smithi203 podman[72124]: 2024-04-25 17:53:47.287479955 +0000 UTC m=+6.198818376 container cleanup b266277954d20eda77250d8c283e9a14dda5714e56735ce9ef86d6df824d9e9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T17:53:47.461 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:47 smithi203 bash[72124]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5 2024-04-25T17:53:47.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:47.332+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:47.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:47.332+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:47.948 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:47 smithi203 podman[72143]: 2024-04-25 17:53:47.505415188 +0000 UTC m=+0.645659763 container remove b266277954d20eda77250d8c283e9a14dda5714e56735ce9ef86d6df824d9e9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T17:53:47.948 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:47.524+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:47.949 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:47.524+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:47.949 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:47.741+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:47.949 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:47.741+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:47.949 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:47 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:47.741+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:48.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:47.872+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:48.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:47.872+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:48.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:47.872+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.458202+0000 front 2024-04-25T17:53:19.458333+0000 (oldest deadline 2024-04-25T17:53:45.358280+0000) 2024-04-25T17:53:48.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:47.922+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:48.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:47.922+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:48.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:47 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:47.922+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:48.211 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:47 smithi203 podman[72234]: 2024-04-25 17:53:47.852918671 +0000 UTC m=+0.018678208 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:53:48.211 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:48 smithi203 podman[72234]: 2024-04-25 17:53:48.014588464 +0000 UTC m=+0.180347984 container create 4451e4a049c33ad0108e339aaf6855a65ca7cebda0e3fbb2aaf57d1e7f221172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:53:48.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:47.836+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:48.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:47.836+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:48.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:47 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:47.836+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:48.520 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:48 smithi203 podman[72234]: 2024-04-25 17:53:48.260398459 +0000 UTC m=+0.426157984 container init 4451e4a049c33ad0108e339aaf6855a65ca7cebda0e3fbb2aaf57d1e7f221172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:53:48.520 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:48 smithi203 podman[72234]: 2024-04-25 17:53:48.263976423 +0000 UTC m=+0.429735946 container start 4451e4a049c33ad0108e339aaf6855a65ca7cebda0e3fbb2aaf57d1e7f221172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T17:53:48.520 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:48 smithi203 podman[72234]: 2024-04-25 17:53:48.342697658 +0000 UTC m=+0.508457181 container attach 4451e4a049c33ad0108e339aaf6855a65ca7cebda0e3fbb2aaf57d1e7f221172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:53:48.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:48 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:48.334+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:48.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:48 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:48.334+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:48.903 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 17:53:48 smithi203 podman[72234]: 2024-04-25 17:53:48.554831389 +0000 UTC m=+0.720590911 container died 4451e4a049c33ad0108e339aaf6855a65ca7cebda0e3fbb2aaf57d1e7f221172 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T17:53:48.903 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:48 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:48.520+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:48.903 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:48 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:48.520+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:48.904 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:48 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:48.754+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:48.904 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:48 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:48.754+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:48.904 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:48 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:48.754+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:49.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:48 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:48.850+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:49.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:48 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:48.850+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:49.205 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:48 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:48.850+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.458202+0000 front 2024-04-25T17:53:19.458333+0000 (oldest deadline 2024-04-25T17:53:45.358280+0000) 2024-04-25T17:53:49.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:48 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:48.956+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:49.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:48 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:48.956+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:49.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:48 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:48.956+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:49.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T17:53:49.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:48 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:48.843+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:49.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:48 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:48.843+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:49.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:48 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:48.843+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:49.321 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:53:49.321 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T17:53:49.321 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T17:53:49.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.6 2024-04-25T17:53:49.640 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:49 smithi098 systemd[1]: Stopping Ceph osd.6 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:53:49.725 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:49 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:49.473+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:49.726 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:49 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:49.473+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:49.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:49 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:49.302+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:49.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:49 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:49.302+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:49.954 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:49.640+0000 7f2c6b77b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:53:49.954 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:49.640+0000 7f2c6b77b700 -1 osd.6 119 *** Got signal Terminated *** 2024-04-25T17:53:49.955 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:49.640+0000 7f2c6b77b700 -1 osd.6 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:53:49.955 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:49.884+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:49.955 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:49.884+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:49.955 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:49.884+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.458202+0000 front 2024-04-25T17:53:19.458333+0000 (oldest deadline 2024-04-25T17:53:45.358280+0000) 2024-04-25T17:53:50.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:49 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:49.726+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:50.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:49 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:49.726+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:50.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:49 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:49.726+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:50.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:49 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:49.862+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:50.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:49 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:49.862+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:50.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:49 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:49.862+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:50.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:49.974+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:50.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:49.974+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:50.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:49 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:49.974+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:50.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:50 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:50.280+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:50.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:50 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:50.280+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:50.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:50.478+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:50.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:50.478+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:50.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:50.716+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:50.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:50.716+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:50.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:50.716+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:50.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:50 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:50.716+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:53:51.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:50.844+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:51.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:50.844+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:51.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:50.844+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.458202+0000 front 2024-04-25T17:53:19.458333+0000 (oldest deadline 2024-04-25T17:53:45.358280+0000) 2024-04-25T17:53:51.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:50.981+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:51.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:50.981+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:51.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:50 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:50.981+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:51.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:50 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:50.828+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:51.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:50 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:50.828+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:51.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:50 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:50.828+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:51.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:51 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:51.295+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:51.772 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:51 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:51.295+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:51.962 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:51 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:51.484+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:51.962 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:51 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:51.484+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:51.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:51 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:51.673+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:51.963 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:51 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:51.673+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:51.963 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:51 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:51.673+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:51.963 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:51 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:51.673+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:53:52.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:51.827+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:52.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:51.827+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:52.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:51.827+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.458202+0000 front 2024-04-25T17:53:19.458333+0000 (oldest deadline 2024-04-25T17:53:45.358280+0000) 2024-04-25T17:53:52.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:51.938+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:52.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:51.938+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:52.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:51.938+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:52.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:51 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:51.938+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:53:52.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:51 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:51.824+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:52.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:51 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:51.824+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:52.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:51 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:51.824+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:52.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:52.313+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:52.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:52.313+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:52.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:52.518+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:52.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:52.518+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:52.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:52.518+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:53:52.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:52.715+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:52.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:52.715+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:52.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:52.715+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:52.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:52 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:52.715+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:53:53.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:52.807+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:53.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:52.807+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:53.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:52.807+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.458202+0000 front 2024-04-25T17:53:19.458333+0000 (oldest deadline 2024-04-25T17:53:45.358280+0000) 2024-04-25T17:53:53.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:52.929+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:53.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:52.929+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:53.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:52.929+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:53.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:52 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:52.929+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:53:53.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:52.853+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:53.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:52.853+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:53.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:52 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:52.853+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:53.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:53.346+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:53.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:53.346+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:53.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:53.346+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.948282+0000 front 2024-04-25T17:53:27.948351+0000 (oldest deadline 2024-04-25T17:53:53.248272+0000) 2024-04-25T17:53:53.962 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:53.520+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:53.963 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:53.520+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:53.963 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:53.520+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:53:53.963 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:53.688+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:53.963 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:53.688+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:53.963 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:53.688+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:53.963 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:53 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:53.688+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:53:54.106 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:53.936+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:54.106 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:53.936+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:54.106 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:53.936+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:54.106 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:53.936+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:53:54.106 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:53.804+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.456117+0000 front 2024-04-25T17:53:02.456264+0000 (oldest deadline 2024-04-25T17:53:28.356285+0000) 2024-04-25T17:53:54.106 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:53.804+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:08.356744+0000 front 2024-04-25T17:53:08.356842+0000 (oldest deadline 2024-04-25T17:53:34.256823+0000) 2024-04-25T17:53:54.106 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:53.804+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.458202+0000 front 2024-04-25T17:53:19.458333+0000 (oldest deadline 2024-04-25T17:53:45.358280+0000) 2024-04-25T17:53:54.106 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:53 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6[57923]: 2024-04-25T17:53:53.804+0000 7f2c67d6f700 -1 osd.6 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:28.859298+0000 front 2024-04-25T17:53:28.859329+0000 (oldest deadline 2024-04-25T17:53:52.959486+0000) 2024-04-25T17:53:54.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:53.803+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:54.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:53.804+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:54.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:53.804+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:54.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:53.804+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:53:54.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:54 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:54.363+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:54.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:54 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:54.363+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:54.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:54 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:54.363+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.948282+0000 front 2024-04-25T17:53:27.948351+0000 (oldest deadline 2024-04-25T17:53:53.248272+0000) 2024-04-25T17:53:54.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:54 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:54.523+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:54.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:54 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:54.523+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:54.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:54 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:54.523+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:53:54.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:54 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:54.647+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:54.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:54 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:54.647+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:54.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:54 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:54.647+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:54.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:54 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:54.647+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:53:55.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:54.897+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:55.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:54.897+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:55.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:54.897+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:55.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:54 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:54.897+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:53:55.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:54 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:54.782+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:55.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:54 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:54.782+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:55.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:54 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:54.782+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:55.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:54 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:54.782+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:53:55.632 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:55 smithi098 podman[107305]: 2024-04-25 17:53:55.326056056 +0000 UTC m=+5.805467306 container died 0de12907b00649888a5162d83bb79508af6e7fe79bf3decd94d0b91ca62690a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:53:55.766 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:55 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:55.379+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:55.766 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:55 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:55.379+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:55.766 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:55 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:55.379+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.948282+0000 front 2024-04-25T17:53:27.948351+0000 (oldest deadline 2024-04-25T17:53:53.248272+0000) 2024-04-25T17:53:55.928 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:55 smithi098 podman[107305]: 2024-04-25 17:53:55.833033679 +0000 UTC m=+6.312444922 container cleanup 0de12907b00649888a5162d83bb79508af6e7fe79bf3decd94d0b91ca62690a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T17:53:55.929 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:55 smithi098 bash[107305]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6 2024-04-25T17:53:55.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:55 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:55.558+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:55.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:55 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:55.558+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:55.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:55 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:55.558+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:53:55.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:55 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:55.687+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:55.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:55 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:55.687+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:55.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:55 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:55.687+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:55.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:55 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:55.687+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:53:56.021 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:55 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:55.767+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:56.021 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:55 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:55.767+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:56.021 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:55 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:55.767+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:56.021 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:55 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:55.767+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:53:56.204 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:56 smithi098 podman[107326]: 2024-04-25 17:53:56.151427443 +0000 UTC m=+0.822675043 container remove 0de12907b00649888a5162d83bb79508af6e7fe79bf3decd94d0b91ca62690a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:53:56.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:55 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:55.929+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:56.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:55 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:55.929+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:56.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:55 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:55.929+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:56.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:55 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:55.929+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:53:56.650 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:56 smithi098 podman[107428]: 2024-04-25 17:53:56.43448707 +0000 UTC m=+0.017777905 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:53:56.650 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:56 smithi098 podman[107428]: 2024-04-25 17:53:56.552611959 +0000 UTC m=+0.135902778 container create 83953b33773cc6eab45da949d779d1f9b866d9c2306acace752cdd2cad513998 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-25T17:53:56.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:56 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:56.383+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:56.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:56 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:56.383+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:56.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:56 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:56.383+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.948282+0000 front 2024-04-25T17:53:27.948351+0000 (oldest deadline 2024-04-25T17:53:53.248272+0000) 2024-04-25T17:53:56.954 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:56 smithi098 podman[107428]: 2024-04-25 17:53:56.763463355 +0000 UTC m=+0.346754174 container init 83953b33773cc6eab45da949d779d1f9b866d9c2306acace752cdd2cad513998 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T17:53:56.954 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:56 smithi098 podman[107428]: 2024-04-25 17:53:56.766640968 +0000 UTC m=+0.349931791 container start 83953b33773cc6eab45da949d779d1f9b866d9c2306acace752cdd2cad513998 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:53:56.954 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:56 smithi098 podman[107428]: 2024-04-25 17:53:56.821583611 +0000 UTC m=+0.404874441 container attach 83953b33773cc6eab45da949d779d1f9b866d9c2306acace752cdd2cad513998 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T17:53:56.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:56 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:56.574+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:56.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:56 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:56.574+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:56.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:56 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:56.574+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:53:56.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:56 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:56.705+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:56.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:56 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:56.705+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:56.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:56 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:56.705+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:56.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:56 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:56.705+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:53:57.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:56 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:56.799+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:57.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:56 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:56.799+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:57.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:56 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:56.799+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:57.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:56 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:56.799+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:53:57.323 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 17:53:57 smithi098 podman[107428]: 2024-04-25 17:53:57.057248158 +0000 UTC m=+0.640538980 container died 83953b33773cc6eab45da949d779d1f9b866d9c2306acace752cdd2cad513998 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:53:57.323 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:56 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:56.957+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:57.323 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:56 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:56.957+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:57.323 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:56 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:56.957+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:57.323 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:56 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:56.957+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:53:57.632 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T17:53:57.677 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:53:57.678 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T17:53:57.678 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T17:53:57.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.7 2024-04-25T17:53:57.694 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:57 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:57.412+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:57.694 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:57 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:57.412+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:57.695 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:57 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:57.412+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.948282+0000 front 2024-04-25T17:53:27.948351+0000 (oldest deadline 2024-04-25T17:53:53.248272+0000) 2024-04-25T17:53:57.936 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:57 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:57.936+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:57.936 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:57 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:57.936+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:57.937 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:57 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:57.936+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:57.937 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:57 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:57.936+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:53:57.962 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:57 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:57.541+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:57.962 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:57 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:57.541+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:57.962 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:57 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:57.541+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:53:57.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:57 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:57.718+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:57.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:57 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:57.718+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:57.963 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:57 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:57.718+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:57.963 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:57 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:57.718+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:53:57.968 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:57 smithi173 systemd[1]: Stopping Ceph osd.7 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:53:57.969 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:57 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:57.837+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:57.969 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:57 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:57.837+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:57.969 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:57 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:57.837+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:57.969 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:57 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:57.837+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:53:58.280 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:57 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:57.988+0000 7f7fe9a08700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:53:58.280 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:57 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:57.988+0000 7f7fe9a08700 -1 osd.7 119 *** Got signal Terminated *** 2024-04-25T17:53:58.280 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:57 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:57.988+0000 7f7fe9a08700 -1 osd.7 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:53:58.780 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:58 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:58.381+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:58.780 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:58 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:58.381+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:58.780 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:58 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:58.381+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.948282+0000 front 2024-04-25T17:53:27.948351+0000 (oldest deadline 2024-04-25T17:53:53.248272+0000) 2024-04-25T17:53:58.863 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:58 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:58.507+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:58.864 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:58 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:58.507+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:58.864 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:58 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:58.507+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:53:58.864 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:58 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:58.756+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:58.864 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:58 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:58.756+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:58.865 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:58 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:58.756+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:58.865 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:58 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:58.756+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:53:59.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:58 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:58.894+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:53:59.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:58 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:58.894+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:53:59.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:58 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:58.894+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:53:59.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:58 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:58.894+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:53:59.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:58 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:58.894+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:53:59.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:58 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:58.856+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:53:59.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:58 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:58.856+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:53:59.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:58 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:58.856+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:53:59.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:58 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:58.856+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:53:59.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:59 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:59.428+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:53:59.772 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:59 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:59.428+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:53:59.772 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:53:59 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:53:59.428+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.948282+0000 front 2024-04-25T17:53:27.948351+0000 (oldest deadline 2024-04-25T17:53:53.248272+0000) 2024-04-25T17:53:59.961 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:59 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:59.495+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:53:59.962 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:59 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:59.495+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:53:59.962 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:53:59 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:53:59.495+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:53:59.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:59 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:59.737+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:53:59.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:59 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:59.737+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:53:59.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:59 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:59.737+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:53:59.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:53:59 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:53:59.737+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:00.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:59 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:59.896+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:00.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:59 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:59.896+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:00.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:59 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:59.896+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:00.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:59 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:59.896+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:00.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:53:59 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:53:59.896+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:00.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:59 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:59.852+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:00.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:59 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:59.852+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:00.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:59 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:59.852+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:00.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:53:59 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:53:59.852+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:00.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:00 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:00.471+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:54:00.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:00 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:00.471+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:54:00.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:00 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:00.471+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.948282+0000 front 2024-04-25T17:53:27.948351+0000 (oldest deadline 2024-04-25T17:53:53.248272+0000) 2024-04-25T17:54:00.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:00 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:00.471+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:37.949237+0000 front 2024-04-25T17:53:37.949264+0000 (oldest deadline 2024-04-25T17:53:59.649464+0000) 2024-04-25T17:54:00.778 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:00 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:00.513+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:00.778 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:00 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:00.513+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:00.778 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:00 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:00.513+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:00.778 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:00 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:00.513+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:01.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:00 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:00.898+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:01.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:00 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:00.898+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:01.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:00 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:00.898+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:01.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:00 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:00.898+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:01.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:00 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:00.898+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:01.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:00 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:00.779+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:01.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:00 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:00.779+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:01.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:00 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:00.779+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:01.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:00 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:00.779+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:01.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:00 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:00.902+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:01.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:00 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:00.902+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:01.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:00 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:00.902+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:01.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:00 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:00.902+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:01.770 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:01 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:01.482+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:01.770 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:01 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:01.482+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:01.770 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:01 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:01.482+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:01.770 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:01 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:01.482+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:01.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:01 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:01.508+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:54:01.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:01 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:01.508+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:54:01.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:01 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:01.508+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.948282+0000 front 2024-04-25T17:53:27.948351+0000 (oldest deadline 2024-04-25T17:53:53.248272+0000) 2024-04-25T17:54:01.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:01 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:01.508+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:37.949237+0000 front 2024-04-25T17:53:37.949264+0000 (oldest deadline 2024-04-25T17:53:59.649464+0000) 2024-04-25T17:54:02.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:01 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:01.943+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:02.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:01 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:01.943+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:02.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:01 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:01.943+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:02.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:01 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:01.943+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:02.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:01 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:01.943+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:02.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:01 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:01.771+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:02.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:01 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:01.771+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:02.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:01 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:01.771+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:02.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:01 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:01.771+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:02.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:01 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:01.939+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:02.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:01 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:01.939+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:02.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:01 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:01.939+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:02.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:01 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:01.939+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:02.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:02 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:02.452+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:02.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:02 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:02.452+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:02.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:02 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:02.452+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:02.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:02 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:02.452+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:02.935 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:02 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:02.530+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:10.445512+0000 front 2024-04-25T17:53:10.445541+0000 (oldest deadline 2024-04-25T17:53:35.745628+0000) 2024-04-25T17:54:02.935 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:02 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:02.530+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:19.747455+0000 front 2024-04-25T17:53:19.747379+0000 (oldest deadline 2024-04-25T17:53:45.647243+0000) 2024-04-25T17:54:02.935 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:02 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:02.530+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.948282+0000 front 2024-04-25T17:53:27.948351+0000 (oldest deadline 2024-04-25T17:53:53.248272+0000) 2024-04-25T17:54:02.935 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:02 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7[50696]: 2024-04-25T17:54:02.530+0000 7f7fe1ff4700 -1 osd.7 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:37.949237+0000 front 2024-04-25T17:53:37.949264+0000 (oldest deadline 2024-04-25T17:53:59.649464+0000) 2024-04-25T17:54:03.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:02 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:02.812+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:03.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:02 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:02.812+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:03.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:02 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:02.812+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:03.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:02 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:02.812+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:03.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:02 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:02.936+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:03.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:02 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:02.936+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:03.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:02 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:02.936+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:03.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:02 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:02.936+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:03.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:02 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:02.936+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:03.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:02.964+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:03.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:02.964+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:03.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:02.964+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:03.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:02.964+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:03.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:02 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:02.964+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:03.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:03.419+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:03.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:03.419+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:03.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:03.419+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:03.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:03.419+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:03.771 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:03 smithi173 podman[70892]: 2024-04-25 17:54:03.507270973 +0000 UTC m=+5.639134359 container died 94152295af34700ba8e6e0565807218d25fb8d4d02af32fce1ddbc153534d3d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T17:54:04.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:03.804+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:04.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:03.804+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:04.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:03.804+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:04.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:03.804+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:04.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:03 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:03.804+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:04.265 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:03 smithi173 podman[70892]: 2024-04-25 17:54:03.914805845 +0000 UTC m=+6.046669213 container cleanup 94152295af34700ba8e6e0565807218d25fb8d4d02af32fce1ddbc153534d3d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-25T17:54:04.265 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:03 smithi173 bash[70892]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7 2024-04-25T17:54:04.265 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:03 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:03.914+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:04.265 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:03 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:03.914+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:04.265 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:03 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:03.914+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:04.265 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:03 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:03.914+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:04.265 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:03 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:03.914+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:04.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:03 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:03.998+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:04.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:03 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:03.998+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:04.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:03 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:03.998+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:04.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:03 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:03.998+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:04.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:03 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:03.998+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:04.521 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:04 smithi173 podman[70920]: 2024-04-25 17:54:04.266243437 +0000 UTC m=+0.753302478 container remove 94152295af34700ba8e6e0565807218d25fb8d4d02af32fce1ddbc153534d3d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T17:54:04.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:04.373+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:04.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:04.373+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:04.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:04.373+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:04.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:04.373+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:04.915 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:04 smithi173 podman[71012]: 2024-04-25 17:54:04.483889923 +0000 UTC m=+0.018152171 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:54:04.915 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:04 smithi173 podman[71012]: 2024-04-25 17:54:04.583992087 +0000 UTC m=+0.118254363 container create 21efe484935965bca85b2450a708162b720fc3f07dba7c22a63d08cba4763387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T17:54:04.915 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:04 smithi173 podman[71012]: 2024-04-25 17:54:04.820299514 +0000 UTC m=+0.354561757 container init 21efe484935965bca85b2450a708162b720fc3f07dba7c22a63d08cba4763387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T17:54:04.915 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:04 smithi173 podman[71012]: 2024-04-25 17:54:04.823645282 +0000 UTC m=+0.357907521 container start 21efe484935965bca85b2450a708162b720fc3f07dba7c22a63d08cba4763387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:54:05.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:04.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:05.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:04.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:05.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:04.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:05.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:04.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:05.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:04 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:04.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:05.271 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:04 smithi173 podman[71012]: 2024-04-25 17:54:04.919860116 +0000 UTC m=+0.454122355 container attach 21efe484935965bca85b2450a708162b720fc3f07dba7c22a63d08cba4763387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T17:54:05.271 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 17:54:05 smithi173 podman[71012]: 2024-04-25 17:54:05.11803812 +0000 UTC m=+0.652300346 container died 21efe484935965bca85b2450a708162b720fc3f07dba7c22a63d08cba4763387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T17:54:05.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:04 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:04.915+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:05.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:04 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:04.915+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:05.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:04 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:04.915+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:05.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:04 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:04.915+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:05.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:04 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:04.915+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:05.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:05.011+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:05.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:05.011+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:05.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:05.011+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:05.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:05.011+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:05.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:05.011+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:05.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:05 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:05.343+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:05.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:05 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:05.343+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:05.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:05 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:05.343+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:05.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:05 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:05.343+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:05.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T17:54:05.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:05.898 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T17:54:05.898 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T17:54:05.899 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.8 2024-04-25T17:54:06.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:05.977+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:06.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:05.977+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:06.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:05.977+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:06.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:05.977+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:06.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:05 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:05.977+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:06.211 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:05 smithi203 systemd[1]: Stopping Ceph osd.8 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:54:06.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:05 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:05.854+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:06.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:05 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:05.854+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:06.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:05 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:05.854+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:06.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:05 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:05.854+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:06.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:05 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:05.854+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:06.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:05 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:05.923+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:06.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:05 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:05.923+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:06.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:05 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:05.923+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:06.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:05 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:05.923+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:06.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:05 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:05.923+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:06.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:06.257+0000 7f38c1efe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:54:06.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:06.257+0000 7f38c1efe700 -1 osd.8 119 *** Got signal Terminated *** 2024-04-25T17:54:06.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:06.257+0000 7f38c1efe700 -1 osd.8 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:54:06.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:06.296+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:06.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:06.296+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:06.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:06.296+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:06.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:06.296+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:07.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:06.890+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:07.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:06.890+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:07.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:06.890+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:07.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:06.890+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:07.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:06.890+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:07.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:06 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:06.969+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:07.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:06 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:06.969+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:07.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:06 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:06.969+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:07.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:06 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:06.969+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:07.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:06 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:06.969+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:07.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:07.026+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:07.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:07.026+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:07.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:07.026+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:07.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:07.026+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:07.456 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:07 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:07.026+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:07.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:07.284+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:07.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:07.284+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:07.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:07.284+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:07.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:07.284+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:07.713 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:07.284+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.646262+0000 front 2024-04-25T17:53:45.646294+0000 (oldest deadline 2024-04-25T17:54:07.247164+0000) 2024-04-25T17:54:08.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:07.858+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:08.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:07.858+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:08.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:07.858+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:08.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:07.858+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:08.213 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:07 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:07.858+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:08.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:07 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:07.921+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:08.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:07 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:07.921+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:08.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:07 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:07.921+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:08.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:07 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:07.921+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:08.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:07 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:07.921+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:08.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:08.004+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:08.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:08.004+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:08.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:08.004+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:08.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:08.004+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:08.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:08.004+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:08.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:08.263+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:08.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:08.263+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:08.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:08.263+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:08.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:08.263+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:08.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:08.263+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.646262+0000 front 2024-04-25T17:53:45.646294+0000 (oldest deadline 2024-04-25T17:54:07.247164+0000) 2024-04-25T17:54:09.214 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:08.896+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:09.214 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:08.896+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:09.214 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:08.896+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:09.214 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:08.896+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:09.214 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:08 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:08.896+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:09.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:08 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:08.937+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:09.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:08 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:08.937+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:09.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:08 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:08.937+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:09.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:08 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:08.937+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:09.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:08 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:08.937+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:09.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:08 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:08.937+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:09.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:08.965+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:09.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:08.965+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:09.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:08.965+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:09.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:08.965+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:09.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:08 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:08.965+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:09.572 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:09 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:09.300+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:09.572 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:09 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:09.300+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:09.572 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:09 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:09.300+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:09.572 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:09 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:09.300+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:09.572 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:09 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:09.300+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.646262+0000 front 2024-04-25T17:53:45.646294+0000 (oldest deadline 2024-04-25T17:54:07.247164+0000) 2024-04-25T17:54:10.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:09 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:09.912+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:10.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:09 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:09.912+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:10.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:09 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:09.912+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:10.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:09 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:09.912+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:10.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:09 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:09.912+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:10.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:09 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:09.941+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:10.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:09 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:09.941+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:10.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:09 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:09.941+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:10.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:09 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:09.941+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:10.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:09 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:09.941+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:10.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:09 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:09.941+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:10.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.010+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:10.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.010+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:10.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.010+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:10.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.010+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:10.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.010+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:10.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:10.268+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:10.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:10.268+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:10.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:10.268+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:10.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:10.268+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:10.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:10.268+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.646262+0000 front 2024-04-25T17:53:45.646294+0000 (oldest deadline 2024-04-25T17:54:07.247164+0000) 2024-04-25T17:54:11.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:10.869+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:11.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:10.869+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:11.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:10.869+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:11.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:10.869+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:11.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:10.869+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:11.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:10 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:10.869+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:11.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:10 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:10.951+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:11.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:10 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:10.951+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:11.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:10 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:10.951+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:11.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:10 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:10.951+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:11.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:10 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:10.951+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:11.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:10 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:10.951+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:11.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.999+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:11.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.999+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:11.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.999+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:11.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.999+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:11.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.999+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:11.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:10 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:10.999+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.892288+0000 front 2024-04-25T17:53:45.892273+0000 (oldest deadline 2024-04-25T17:54:10.592125+0000) 2024-04-25T17:54:11.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:11.222+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.440579+0000 front 2024-04-25T17:53:12.440742+0000 (oldest deadline 2024-04-25T17:53:33.540493+0000) 2024-04-25T17:54:11.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:11.222+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.542323+0000 front 2024-04-25T17:53:17.542350+0000 (oldest deadline 2024-04-25T17:53:45.743086+0000) 2024-04-25T17:54:11.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:11.222+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:25.743608+0000 front 2024-04-25T17:53:25.743663+0000 (oldest deadline 2024-04-25T17:53:51.643641+0000) 2024-04-25T17:54:11.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:11.222+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:35.744637+0000 front 2024-04-25T17:53:35.744594+0000 (oldest deadline 2024-04-25T17:54:00.444650+0000) 2024-04-25T17:54:11.712 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8[51079]: 2024-04-25T17:54:11.222+0000 7f38be4f2700 -1 osd.8 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.646262+0000 front 2024-04-25T17:53:45.646294+0000 (oldest deadline 2024-04-25T17:54:07.247164+0000) 2024-04-25T17:54:12.149 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:11 smithi203 podman[72351]: 2024-04-25 17:54:11.834483329 +0000 UTC m=+5.722932109 container died 241567a7fa069b8897c3947522b0488762733e5663a23847086f43fa1279cf4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T17:54:12.149 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:11.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:12.149 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:11.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:12.150 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:11.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:12.150 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:11.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:12.150 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:11.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:12.150 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:11.820+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:12.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:11 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:11.995+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:12.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:11 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:11.995+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:12.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:11 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:11.995+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:12.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:11 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:11.995+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:12.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:11 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:11.995+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:12.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:11 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:11.995+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:12.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.003+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:12.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.003+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:12.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.003+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:12.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.003+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:12.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.003+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:12.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.003+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.892288+0000 front 2024-04-25T17:53:45.892273+0000 (oldest deadline 2024-04-25T17:54:10.592125+0000) 2024-04-25T17:54:12.461 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:12 smithi203 podman[72351]: 2024-04-25 17:54:12.258237577 +0000 UTC m=+6.146686399 container cleanup 241567a7fa069b8897c3947522b0488762733e5663a23847086f43fa1279cf4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T17:54:12.461 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:12 smithi203 bash[72351]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8 2024-04-25T17:54:12.794 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:12 smithi203 podman[72368]: 2024-04-25 17:54:12.484650272 +0000 UTC m=+0.645705904 container remove 241567a7fa069b8897c3947522b0488762733e5663a23847086f43fa1279cf4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T17:54:13.051 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:12 smithi203 podman[72456]: 2024-04-25 17:54:12.830272255 +0000 UTC m=+0.017766473 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:54:13.051 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:12 smithi203 podman[72456]: 2024-04-25 17:54:12.935750174 +0000 UTC m=+0.123244420 container create eb20899dc83febc9a9050b2c896b2480e344e8934578691035145316d15046f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:54:13.052 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:12 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:12.794+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:13.052 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:12 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:12.794+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:13.052 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:12 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:12.794+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:13.052 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:12 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:12.794+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:13.052 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:12 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:12.794+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:13.052 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:12 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:12.794+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:13.453 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:13 smithi203 podman[72456]: 2024-04-25 17:54:13.16308989 +0000 UTC m=+0.350584094 container init eb20899dc83febc9a9050b2c896b2480e344e8934578691035145316d15046f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T17:54:13.453 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:13 smithi203 podman[72456]: 2024-04-25 17:54:13.166315804 +0000 UTC m=+0.353810009 container start eb20899dc83febc9a9050b2c896b2480e344e8934578691035145316d15046f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-25T17:54:13.454 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:13 smithi203 podman[72456]: 2024-04-25 17:54:13.263042516 +0000 UTC m=+0.450536732 container attach eb20899dc83febc9a9050b2c896b2480e344e8934578691035145316d15046f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-04-25T17:54:13.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:13.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:13.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:13.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:13.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:13.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:12 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:12.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.892288+0000 front 2024-04-25T17:53:45.892273+0000 (oldest deadline 2024-04-25T17:54:10.592125+0000) 2024-04-25T17:54:13.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.023+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:13.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.023+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:13.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.023+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:13.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.024+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:13.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.024+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:13.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.024+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:13.711 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 17:54:13 smithi203 podman[72456]: 2024-04-25 17:54:13.45503445 +0000 UTC m=+0.642528656 container died eb20899dc83febc9a9050b2c896b2480e344e8934578691035145316d15046f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:54:14.082 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:13 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:13.762+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:14.082 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:13 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:13.762+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:14.082 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:13 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:13.762+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:14.082 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:13 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:13.762+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:14.083 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:13 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:13.762+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:14.083 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:13 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:13.762+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:14.103 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T17:54:14.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:14.144 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T17:54:14.144 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T17:54:14.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.9 2024-04-25T17:54:14.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.978+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:14.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.978+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:14.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.978+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:14.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.978+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:14.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.978+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:14.272 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:13 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:13.978+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:14.443 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:14 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:14.016+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:14.444 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:14 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:14.016+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:14.444 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:14 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:14.016+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:14.444 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:14 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:14.016+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:14.444 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:14 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:14.016+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:14.444 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:14 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:14.016+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.892288+0000 front 2024-04-25T17:53:45.892273+0000 (oldest deadline 2024-04-25T17:54:10.592125+0000) 2024-04-25T17:54:14.444 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:14 smithi098 systemd[1]: Stopping Ceph osd.9 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:54:14.444 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:14 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:14.444+0000 7f338ae68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:54:14.444 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:14 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:14.444+0000 7f338ae68700 -1 osd.9 119 *** Got signal Terminated *** 2024-04-25T17:54:14.444 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:14 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:14.444+0000 7f338ae68700 -1 osd.9 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:54:15.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:14 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:14.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:15.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:14 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:14.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:15.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:14 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:14.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:15.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:14 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:14.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:15.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:14 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:14.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:15.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:14 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:14.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:15.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:14 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:14.985+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:15.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:14 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:14.985+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:15.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:14 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:14.985+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:15.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:14 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:14.985+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:15.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:14 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:14.985+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:15.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:14 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:14.985+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:15.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:15 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:15.059+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:15.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:15 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:15.059+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:15.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:15 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:15.059+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:15.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:15 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:15.059+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:15.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:15 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:15.059+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:15.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:15 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:15.059+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.892288+0000 front 2024-04-25T17:53:45.892273+0000 (oldest deadline 2024-04-25T17:54:10.592125+0000) 2024-04-25T17:54:16.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:15 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:15.753+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:16.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:15 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:15.753+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:16.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:15 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:15.753+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:16.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:15 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:15.753+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:16.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:15 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:15.753+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:16.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:15 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:15.753+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:16.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:16 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:16.013+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:16.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:16 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:16.013+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:16.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:16 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:16.013+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:16.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:16 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:16.013+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:16.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:16 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:16.013+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:16.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:16 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:16.013+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:16.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:16 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:16.074+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:16.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:16 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:16.074+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:16.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:16 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:16.074+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:16.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:16 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:16.074+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:16.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:16 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:16.074+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:16.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:16 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:16.074+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.892288+0000 front 2024-04-25T17:53:45.892273+0000 (oldest deadline 2024-04-25T17:54:10.592125+0000) 2024-04-25T17:54:17.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:16.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:17.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:16.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:17.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:16.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:17.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:16.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:17.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:16.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:17.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:16.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:17.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:17 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:17.061+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:17.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:17 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:17.061+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:17.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:17 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:17.061+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:17.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:17 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:17.061+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:17.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:17 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:17.061+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:17.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:17 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:17.062+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.892288+0000 front 2024-04-25T17:53:45.892273+0000 (oldest deadline 2024-04-25T17:54:10.592125+0000) 2024-04-25T17:54:17.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:17 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:17.030+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:17.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:17 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:17.030+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:17.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:17 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:17.030+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:17.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:17 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:17.030+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:17.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:17 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:17.030+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:17.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:17 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:17.030+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:17.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:17 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:17.030+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:17.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:17 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:17.691+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:17.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:17 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:17.691+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:17.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:17 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:17.691+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:17.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:17 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:17.691+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:17.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:17 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:17.691+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:17.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:17 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:17.691+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:18.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.018+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:18.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.018+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:18.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.018+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:18.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.018+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:18.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.018+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:18.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.018+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.892288+0000 front 2024-04-25T17:53:45.892273+0000 (oldest deadline 2024-04-25T17:54:10.592125+0000) 2024-04-25T17:54:18.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:18 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:18.077+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:18.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:18 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:18.077+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:18.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:18 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:18.077+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:18.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:18 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:18.077+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:18.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:18 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:18.077+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:18.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:18 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:18.077+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:18.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:18 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:18.077+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:19.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:18 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:18.723+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:19.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:18 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:18.723+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:19.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:18 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:18.723+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:19.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:18 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:18.723+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:19.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:18 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:18.723+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:19.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:18 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:18.723+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:19.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:04.585213+0000 front 2024-04-25T17:53:04.585144+0000 (oldest deadline 2024-04-25T17:53:29.885314+0000) 2024-04-25T17:54:19.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:09.886001+0000 front 2024-04-25T17:53:09.886166+0000 (oldest deadline 2024-04-25T17:53:35.785859+0000) 2024-04-25T17:54:19.454 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:20.387727+0000 front 2024-04-25T17:53:20.387832+0000 (oldest deadline 2024-04-25T17:53:42.687753+0000) 2024-04-25T17:54:19.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.988951+0000 front 2024-04-25T17:53:27.988940+0000 (oldest deadline 2024-04-25T17:53:51.389318+0000) 2024-04-25T17:54:19.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:33.090369+0000 front 2024-04-25T17:53:33.090341+0000 (oldest deadline 2024-04-25T17:53:58.890640+0000) 2024-04-25T17:54:19.455 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:18 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9[64980]: 2024-04-25T17:54:18.970+0000 7f3383454700 -1 osd.9 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:45.892288+0000 front 2024-04-25T17:53:45.892273+0000 (oldest deadline 2024-04-25T17:54:10.592125+0000) 2024-04-25T17:54:19.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:19 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:19.127+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:19.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:19 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:19.127+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:19.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:19 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:19.127+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:19.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:19 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:19.127+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:19.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:19 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:19.127+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:19.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:19 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:19.127+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:19.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:19 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:19.127+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:20.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:19 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:19.739+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:20.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:19 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:19.739+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:20.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:19 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:19.739+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:20.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:19 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:19.739+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:20.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:19 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:19.739+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:20.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:19 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:19.739+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:20.267 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:20 smithi098 podman[107534]: 2024-04-25 17:54:20.010613456 +0000 UTC m=+5.668933404 container died 7625593bf82e9cfec694c3166427fbc130ceeba8ab4ad74bffa45d94b5ba10c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416) 2024-04-25T17:54:20.267 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:20 smithi098 podman[107534]: 2024-04-25 17:54:20.269057302 +0000 UTC m=+5.927377245 container cleanup 7625593bf82e9cfec694c3166427fbc130ceeba8ab4ad74bffa45d94b5ba10c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T17:54:20.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:20 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:20.140+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:20.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:20 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:20.140+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:20.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:20 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:20.140+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:20.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:20 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:20.140+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:20.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:20 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:20.140+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:20.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:20 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:20.140+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:20.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:20 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:20.140+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:20.569 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:20 smithi098 bash[107534]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9 2024-04-25T17:54:20.883 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:20 smithi098 podman[107550]: 2024-04-25 17:54:20.570563428 +0000 UTC m=+0.562584093 container remove 7625593bf82e9cfec694c3166427fbc130ceeba8ab4ad74bffa45d94b5ba10c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T17:54:21.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:20 smithi098 podman[107641]: 2024-04-25 17:54:20.78808009 +0000 UTC m=+0.018144610 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:54:21.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:20 smithi098 podman[107641]: 2024-04-25 17:54:20.896258268 +0000 UTC m=+0.126322759 container create b218c84ff9450f1ae6bbbdfc69530c5387d2b468840923488bd9b2d2bb5ab03d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T17:54:21.204 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:21 smithi098 podman[107641]: 2024-04-25 17:54:21.123710663 +0000 UTC m=+0.353775155 container init b218c84ff9450f1ae6bbbdfc69530c5387d2b468840923488bd9b2d2bb5ab03d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-25T17:54:21.205 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:21 smithi098 podman[107641]: 2024-04-25 17:54:21.126823791 +0000 UTC m=+0.356888284 container start b218c84ff9450f1ae6bbbdfc69530c5387d2b468840923488bd9b2d2bb5ab03d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-25T17:54:21.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:20 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:20.763+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:21.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:20 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:20.763+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:21.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:20 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:20.763+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:21.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:20 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:20.763+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:21.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:20 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:20.763+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:21.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:20 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:20.763+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:21.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:21 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:21.176+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:21.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:21 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:21.176+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:21.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:21 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:21.176+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:21.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:21 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:21.176+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:21.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:21 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:21.176+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:21.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:21 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:21.176+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:21.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:21 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:21.176+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:21.700 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:21 smithi098 podman[107641]: 2024-04-25 17:54:21.232191418 +0000 UTC m=+0.462255911 container attach b218c84ff9450f1ae6bbbdfc69530c5387d2b468840923488bd9b2d2bb5ab03d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T17:54:21.700 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:21 smithi098 podman[107641]: 2024-04-25 17:54:21.418261719 +0000 UTC m=+0.648326227 container died b218c84ff9450f1ae6bbbdfc69530c5387d2b468840923488bd9b2d2bb5ab03d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-25T17:54:21.954 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 17:54:21 smithi098 podman[107666]: 2024-04-25 17:54:21.943777446 +0000 UTC m=+0.519849337 container remove b218c84ff9450f1ae6bbbdfc69530c5387d2b468840923488bd9b2d2bb5ab03d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T17:54:22.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T17:54:22.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:22.069 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T17:54:22.069 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T17:54:22.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.10 2024-04-25T17:54:22.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:21 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:21.766+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:22.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:21 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:21.766+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:22.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:21 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:21.766+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:22.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:21 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:21.766+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:22.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:21 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:21.766+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:22.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:21 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:21.766+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:22.384 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 systemd[1]: Stopping Ceph osd.10 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:54:22.384 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:22.211+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:22.384 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:22.211+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:22.384 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:22.211+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:22.384 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:22.211+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:22.384 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:22.211+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:22.384 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:22.211+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:22.384 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:22.211+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:22.385 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:22.384+0000 7f2afbff9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:54:22.385 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:22.384+0000 7f2afbff9700 -1 osd.10 119 *** Got signal Terminated *** 2024-04-25T17:54:22.385 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:22 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:22.384+0000 7f2afbff9700 -1 osd.10 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:54:23.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:22.726+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:23.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:22.726+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:23.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:22.726+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:23.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:22.726+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:23.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:22.726+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:23.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:22 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:22.726+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:23.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:23 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:23.190+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:23.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:23 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:23.190+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:23.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:23 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:23.190+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:23.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:23 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:23.190+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:23.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:23 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:23.190+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:23.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:23 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:23.190+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:23.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:23 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:23.190+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:24.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:23 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:23.759+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:24.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:23 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:23.759+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:24.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:23 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:23.759+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:24.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:23 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:23.759+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:24.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:23 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:23.759+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:24.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:23 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:23.759+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:24.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:24.239+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:24.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:24.239+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:24.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:24.239+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:24.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:24.239+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:24.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:24.239+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:24.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:24.239+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:24.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:24 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:24.239+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:25.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:24.757+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:25.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:24.757+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:25.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:24.757+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:25.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:24.757+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:25.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:24.757+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:25.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:24.757+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:25.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:24 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:24.757+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:25.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:25.216+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:25.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:25.216+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:25.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:25.216+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:25.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:25.216+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:25.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:25.216+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:25.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:25.216+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:25.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:25 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:25.216+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:26.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:25.724+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:26.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:25.724+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:26.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:25.724+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:26.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:25.724+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:26.213 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:25.724+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:26.213 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:25.724+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:26.213 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:25 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:25.724+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:26.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:26.200+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:26.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:26.200+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:26.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:26.200+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:26.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:26.200+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:26.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:26.200+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:26.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:26.200+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:26.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:26 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:26.200+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:27.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:26 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:26.738+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:27.213 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:26 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:26.738+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:27.213 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:26 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:26.738+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:27.213 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:26 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:26.738+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:27.213 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:26 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:26.738+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:27.213 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:26 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:26.738+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:27.214 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:26 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:26.738+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:27.521 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:27.198+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.857468+0000 front 2024-04-25T17:53:02.857491+0000 (oldest deadline 2024-04-25T17:53:25.757459+0000) 2024-04-25T17:54:27.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:27.198+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:11.658490+0000 front 2024-04-25T17:53:11.658411+0000 (oldest deadline 2024-04-25T17:53:37.558556+0000) 2024-04-25T17:54:27.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:27.198+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:17.559105+0000 front 2024-04-25T17:53:17.559285+0000 (oldest deadline 2024-04-25T17:53:42.859036+0000) 2024-04-25T17:54:27.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:27.198+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:27.560074+0000 front 2024-04-25T17:53:27.560126+0000 (oldest deadline 2024-04-25T17:53:53.360645+0000) 2024-04-25T17:54:27.522 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:27.198+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.762163+0000 front 2024-04-25T17:53:36.762185+0000 (oldest deadline 2024-04-25T17:54:02.662304+0000) 2024-04-25T17:54:27.523 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:27.198+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:42.663003+0000 front 2024-04-25T17:53:42.662879+0000 (oldest deadline 2024-04-25T17:54:07.962886+0000) 2024-04-25T17:54:27.523 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:27 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10[57487]: 2024-04-25T17:54:27.198+0000 7f2af85ed700 -1 osd.10 119 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T17:53:50.263765+0000 front 2024-04-25T17:53:50.263745+0000 (oldest deadline 2024-04-25T17:54:16.163740+0000) 2024-04-25T17:54:28.113 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:27 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:27.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:28.113 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:27 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:27.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:28.114 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:27 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:27.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:28.114 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:27 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:27.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:28.114 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:27 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:27.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:28.114 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:27 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:27.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:28.115 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:27 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:27.748+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:28.328 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:28 smithi173 podman[71119]: 2024-04-25 17:54:28.062182467 +0000 UTC m=+5.789980648 container died d6eb83e1cc041078198a28ab2197eb04800d0459dfaadb6b502867011081481b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-25T17:54:28.670 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:28 smithi173 podman[71119]: 2024-04-25 17:54:28.445618432 +0000 UTC m=+6.173416614 container cleanup d6eb83e1cc041078198a28ab2197eb04800d0459dfaadb6b502867011081481b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T17:54:28.670 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:28 smithi173 bash[71119]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10 2024-04-25T17:54:29.020 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:28 smithi173 podman[71146]: 2024-04-25 17:54:28.70540676 +0000 UTC m=+0.636185862 container remove d6eb83e1cc041078198a28ab2197eb04800d0459dfaadb6b502867011081481b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-25T17:54:29.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:28 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:28.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:29.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:28 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:28.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:29.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:28 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:28.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:29.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:28 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:28.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:29.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:28 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:28.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:29.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:28 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:28.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:29.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:28 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:28.717+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:29.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:29 smithi173 podman[71241]: 2024-04-25 17:54:28.924345797 +0000 UTC m=+0.018178840 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:54:29.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:29 smithi173 podman[71241]: 2024-04-25 17:54:29.031028045 +0000 UTC m=+0.124861082 container create 49f978eb047f8472101fc1de87b0f95c167292eed35158ddb7365ca78b96bf67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T17:54:29.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:29 smithi173 podman[71241]: 2024-04-25 17:54:29.224905646 +0000 UTC m=+0.318738682 container init 49f978eb047f8472101fc1de87b0f95c167292eed35158ddb7365ca78b96bf67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T17:54:29.271 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:29 smithi173 podman[71241]: 2024-04-25 17:54:29.227993924 +0000 UTC m=+0.321826966 container start 49f978eb047f8472101fc1de87b0f95c167292eed35158ddb7365ca78b96bf67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T17:54:29.771 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:29 smithi173 podman[71241]: 2024-04-25 17:54:29.308403112 +0000 UTC m=+0.402236158 container attach 49f978eb047f8472101fc1de87b0f95c167292eed35158ddb7365ca78b96bf67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T17:54:29.771 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 17:54:29 smithi173 podman[71241]: 2024-04-25 17:54:29.523677625 +0000 UTC m=+0.617510657 container died 49f978eb047f8472101fc1de87b0f95c167292eed35158ddb7365ca78b96bf67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T17:54:30.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T17:54:30.162 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:30.162 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T17:54:30.162 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T17:54:30.162 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@osd.11 2024-04-25T17:54:30.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:29 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:29.721+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:30.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:29 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:29.721+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:30.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:29 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:29.721+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:30.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:29 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:29.721+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:30.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:29 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:29.721+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:30.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:29 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:29.721+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:30.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:29 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:29.721+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:30.461 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 systemd[1]: Stopping Ceph osd.11 for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:54:30.740 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:30.469+0000 7fee06a7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:54:30.740 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:30.469+0000 7fee06a7f700 -1 osd.11 119 *** Got signal Terminated *** 2024-04-25T17:54:30.740 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:30.469+0000 7fee06a7f700 -1 osd.11 119 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T17:54:31.211 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:30.740+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:31.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:30.740+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:31.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:30.740+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:31.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:30.740+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:31.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:30.740+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:31.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:30.740+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:31.212 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:30 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:30.740+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:31.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:31 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:31.700+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:31.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:31 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:31.700+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:31.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:31 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:31.700+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:31.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:31 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:31.700+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:31.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:31 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:31.700+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:31.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:31 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:31.700+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:31.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:31 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:31.700+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:32.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:32 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:32.674+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:32.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:32 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:32.674+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:32.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:32 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:32.674+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:32.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:32 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:32.674+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:32.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:32 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:32.674+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:32.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:32 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:32.674+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:32.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:32 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:32.674+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:33.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:33 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:33.630+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:33.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:33 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:33.630+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:33.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:33 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:33.630+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:33.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:33 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:33.630+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:33.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:33 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:33.630+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:33.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:33 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:33.630+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:33.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:33 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:33.630+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:34.961 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:34.651+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T17:53:02.359706+0000 front 2024-04-25T17:53:02.359784+0000 (oldest deadline 2024-04-25T17:53:27.560309+0000) 2024-04-25T17:54:34.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:34.651+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T17:53:12.261481+0000 front 2024-04-25T17:53:12.261591+0000 (oldest deadline 2024-04-25T17:53:37.561481+0000) 2024-04-25T17:54:34.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:34.651+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T17:53:18.662482+0000 front 2024-04-25T17:53:18.662726+0000 (oldest deadline 2024-04-25T17:53:42.762619+0000) 2024-04-25T17:54:34.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:34.651+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T17:53:29.064544+0000 front 2024-04-25T17:53:29.064520+0000 (oldest deadline 2024-04-25T17:53:50.164507+0000) 2024-04-25T17:54:34.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:34.651+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T17:53:36.065804+0000 front 2024-04-25T17:53:36.065989+0000 (oldest deadline 2024-04-25T17:54:03.666327+0000) 2024-04-25T17:54:34.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:34.651+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T17:53:44.667870+0000 front 2024-04-25T17:53:44.667918+0000 (oldest deadline 2024-04-25T17:54:09.968000+0000) 2024-04-25T17:54:34.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:34.651+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T17:54:00.970697+0000 front 2024-04-25T17:54:00.970774+0000 (oldest deadline 2024-04-25T17:54:23.771321+0000) 2024-04-25T17:54:34.962 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:34 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11[58132]: 2024-04-25T17:54:34.651+0000 7fee03073700 -1 osd.11 119 heartbeat_check: no reply from 172.21.15.203:6820 osd.8 since back 2024-04-25T17:54:09.572758+0000 front 2024-04-25T17:54:09.572696+0000 (oldest deadline 2024-04-25T17:54:34.272766+0000) 2024-04-25T17:54:36.419 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:36 smithi203 podman[72585]: 2024-04-25 17:54:36.134276428 +0000 UTC m=+5.777940565 container died 6b92e27d9b89d26a362330868fc4d63c4c21819a67bfd0cf5e58513a3be7bc99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:54:36.711 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:36 smithi203 podman[72585]: 2024-04-25 17:54:36.528329906 +0000 UTC m=+6.171994048 container cleanup 6b92e27d9b89d26a362330868fc4d63c4c21819a67bfd0cf5e58513a3be7bc99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-25T17:54:36.711 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:36 smithi203 bash[72585]: ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11 2024-04-25T17:54:37.036 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:36 smithi203 podman[72602]: 2024-04-25 17:54:36.737613979 +0000 UTC m=+0.603333133 container remove 6b92e27d9b89d26a362330868fc4d63c4c21819a67bfd0cf5e58513a3be7bc99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:54:37.373 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:37 smithi203 podman[72696]: 2024-04-25 17:54:37.0384663 +0000 UTC m=+0.108067755 container create 5e6a5a12bad96c58337ade64e9e8dae1668256d58ac4c3d0d0d1118025acbe82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:54:37.373 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:37 smithi203 podman[72696]: 2024-04-25 17:54:36.948370986 +0000 UTC m=+0.017972457 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T17:54:37.373 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:37 smithi203 podman[72696]: 2024-04-25 17:54:37.274398407 +0000 UTC m=+0.343999862 container init 5e6a5a12bad96c58337ade64e9e8dae1668256d58ac4c3d0d0d1118025acbe82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-25T17:54:37.373 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:37 smithi203 podman[72696]: 2024-04-25 17:54:37.277438081 +0000 UTC m=+0.347039539 container start 5e6a5a12bad96c58337ade64e9e8dae1668256d58ac4c3d0d0d1118025acbe82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T17:54:37.373 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:37 smithi203 podman[72696]: 2024-04-25 17:54:37.374374367 +0000 UTC m=+0.443975831 container attach 5e6a5a12bad96c58337ade64e9e8dae1668256d58ac4c3d0d0d1118025acbe82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:54:37.711 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 17:54:37 smithi203 podman[72696]: 2024-04-25 17:54:37.569650737 +0000 UTC m=+0.639252190 container died 5e6a5a12bad96c58337ade64e9e8dae1668256d58ac4c3d0d0d1118025acbe82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T17:54:38.166 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T17:54:38.244 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:38.245 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T17:54:38.245 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T17:54:38.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.a 2024-04-25T17:54:38.544 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:54:38 smithi098 systemd[1]: Stopping Ceph mds.a for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:54:38.807 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:54:38 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-a[71893]: 2024-04-25T17:54:38.544+0000 7f135e3f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:54:38.807 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:54:38 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-a[71893]: 2024-04-25T17:54:38.544+0000 7f135e3f1700 -1 mds.a *** got signal Terminated *** 2024-04-25T17:54:40.694 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:54:40 smithi098 podman[107745]: 2024-04-25 17:54:40.398631622 +0000 UTC m=+1.966928759 container died f9496264578803285d00b0e04d960c1249a81b106a55671e0cc85581006131b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-25T17:54:40.954 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:54:40 smithi098 podman[107745]: 2024-04-25 17:54:40.870265713 +0000 UTC m=+2.438562892 container cleanup f9496264578803285d00b0e04d960c1249a81b106a55671e0cc85581006131b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-25T17:54:40.954 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 17:54:40 smithi098 bash[107745]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-a 2024-04-25T17:54:41.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T17:54:41.329 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:41.329 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T17:54:41.329 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T17:54:41.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.d 2024-04-25T17:54:41.639 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:54:41 smithi098 systemd[1]: Stopping Ceph mds.d for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:54:41.954 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:54:41 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-d[70584]: 2024-04-25T17:54:41.640+0000 7f4b71e05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:54:41.954 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:54:41 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-d[70584]: 2024-04-25T17:54:41.640+0000 7f4b71e05700 -1 mds.d *** got signal Terminated *** 2024-04-25T17:54:45.373 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:54:45 smithi098 podman[107878]: 2024-04-25 17:54:45.094542427 +0000 UTC m=+3.550158578 container died beecde30ebb8f2ad4cc9c5bcd638ed1111268cc747de198de52664ba884a1eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-d, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T17:54:45.698 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:54:45 smithi098 podman[107878]: 2024-04-25 17:54:45.465957216 +0000 UTC m=+3.921573363 container cleanup beecde30ebb8f2ad4cc9c5bcd638ed1111268cc747de198de52664ba884a1eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T17:54:45.698 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 17:54:45 smithi098 bash[107878]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-d 2024-04-25T17:54:45.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T17:54:45.876 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:45.876 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T17:54:45.877 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T17:54:45.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.g 2024-04-25T17:54:46.204 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:54:45 smithi098 systemd[1]: Stopping Ceph mds.g for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:54:46.204 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:54:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-g[71423]: 2024-04-25T17:54:46.177+0000 7f64e1114700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:54:46.204 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:54:46 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-g[71423]: 2024-04-25T17:54:46.177+0000 7f64e1114700 -1 mds.g *** got signal Terminated *** 2024-04-25T17:54:47.902 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:54:47 smithi098 podman[108000]: 2024-04-25 17:54:47.623095501 +0000 UTC m=+1.534943975 container died b13d255fce754143c699aa75e47599d7501a19ef036a62c8436f6103a29c6948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-25T17:54:48.204 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:54:48 smithi098 podman[108000]: 2024-04-25 17:54:48.002859548 +0000 UTC m=+1.914708030 container cleanup b13d255fce754143c699aa75e47599d7501a19ef036a62c8436f6103a29c6948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T17:54:48.204 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 17:54:48 smithi098 bash[108000]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-g 2024-04-25T17:54:48.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T17:54:48.399 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:48.400 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T17:54:48.400 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T17:54:48.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.j 2024-04-25T17:54:48.698 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 17:54:48 smithi098 systemd[1]: Stopping Ceph mds.j for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:54:48.954 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 17:54:48 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-j[70158]: 2024-04-25T17:54:48.699+0000 7f10a4d1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:54:48.954 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 17:54:48 smithi098 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-j[70158]: 2024-04-25T17:54:48.699+0000 7f10a4d1c700 -1 mds.j *** got signal Terminated *** 2024-04-25T17:54:52.998 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 17:54:52 smithi098 podman[108120]: 2024-04-25 17:54:52.712166751 +0000 UTC m=+4.103096028 container died 7541741ae1be611249b8babd86b0deed5f9ae7941cf5ebdb609bba16d2fc5e15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:54:52.998 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 17:54:53 smithi098 podman[108120]: 2024-04-25 17:54:53.000082412 +0000 UTC m=+4.391011696 container cleanup 7541741ae1be611249b8babd86b0deed5f9ae7941cf5ebdb609bba16d2fc5e15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T17:54:53.335 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 17:54:53 smithi098 bash[108120]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-j 2024-04-25T17:54:53.335 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 17:54:53 smithi098 podman[108134]: 2024-04-25 17:54:53.217753516 +0000 UTC m=+0.501701421 container remove 7541741ae1be611249b8babd86b0deed5f9ae7941cf5ebdb609bba16d2fc5e15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T17:54:53.345 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T17:54:53.375 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:53.375 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T17:54:53.376 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T17:54:53.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.b 2024-04-25T17:54:53.669 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:54:53 smithi173 systemd[1]: Stopping Ceph mds.b for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:54:54.021 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:54:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b[63135]: 2024-04-25T17:54:53.669+0000 7fa153fc5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:54:54.021 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:54:53 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b[63135]: 2024-04-25T17:54:53.669+0000 7fa153fc5700 -1 mds.b *** got signal Terminated *** 2024-04-25T17:54:55.070 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:54:54 smithi173 podman[71351]: 2024-04-25 17:54:54.807221727 +0000 UTC m=+1.237386655 container died d433fc0d571006ea16003765d341c2e9833b29a4c7cbc2a59c844e0be79d288a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5) 2024-04-25T17:54:55.336 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:54:55 smithi173 podman[71351]: 2024-04-25 17:54:55.145416027 +0000 UTC m=+1.575580947 container cleanup d433fc0d571006ea16003765d341c2e9833b29a4c7cbc2a59c844e0be79d288a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T17:54:55.337 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:54:55 smithi173 bash[71351]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b 2024-04-25T17:54:55.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T17:54:55.586 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:54:55 smithi173 podman[71365]: 2024-04-25 17:54:55.338122385 +0000 UTC m=+0.529184796 container remove d433fc0d571006ea16003765d341c2e9833b29a4c7cbc2a59c844e0be79d288a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:54:55.587 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:54:55 smithi173 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.b.service: Deactivated successfully. 2024-04-25T17:54:55.587 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:54:55 smithi173 systemd[1]: Stopped Ceph mds.b for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:54:55.587 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 17:54:55 smithi173 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.b.service: Consumed 29.710s CPU time. 2024-04-25T17:54:55.599 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:55.599 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T17:54:55.599 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T17:54:55.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.e 2024-04-25T17:54:55.931 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:54:55 smithi173 systemd[1]: Stopping Ceph mds.e for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:54:56.271 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:54:55 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-e[62517]: 2024-04-25T17:54:55.931+0000 7f0467ca0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:54:56.271 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:54:55 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-e[62517]: 2024-04-25T17:54:55.931+0000 7f0467ca0700 -1 mds.e *** got signal Terminated *** 2024-04-25T17:54:59.098 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:54:58 smithi173 podman[71472]: 2024-04-25 17:54:58.801040913 +0000 UTC m=+2.978702972 container died bad144b5616ef50ad6f39420e819aee8f12974b3e976725b3545468ad7ad69f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-25T17:54:59.439 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:54:59 smithi173 podman[71472]: 2024-04-25 17:54:59.215473851 +0000 UTC m=+3.393135924 container cleanup bad144b5616ef50ad6f39420e819aee8f12974b3e976725b3545468ad7ad69f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-25T17:54:59.440 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 17:54:59 smithi173 bash[71472]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-e 2024-04-25T17:54:59.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T17:54:59.645 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:54:59.645 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T17:54:59.645 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T17:54:59.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.h 2024-04-25T17:54:59.977 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:54:59 smithi173 systemd[1]: Stopping Ceph mds.h for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:55:00.244 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:54:59 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h[62828]: 2024-04-25T17:54:59.978+0000 7faea0ad1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:55:00.245 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:54:59 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h[62828]: 2024-04-25T17:54:59.978+0000 7faea0ad1700 -1 mds.h *** got signal Terminated *** 2024-04-25T17:55:03.069 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:55:02 smithi173 podman[71602]: 2024-04-25 17:55:02.803633341 +0000 UTC m=+2.935328823 container died ab34c40f9d5861c64532b8f216cf5ed836600e762e42db09cf0381b3f6e2c4b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416) 2024-04-25T17:55:03.419 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:55:03 smithi173 podman[71602]: 2024-04-25 17:55:03.16145161 +0000 UTC m=+3.293147109 container cleanup ab34c40f9d5861c64532b8f216cf5ed836600e762e42db09cf0381b3f6e2c4b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T17:55:03.419 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 17:55:03 smithi173 bash[71602]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-h 2024-04-25T17:55:03.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T17:55:03.618 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:55:03.618 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T17:55:03.619 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T17:55:03.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.k 2024-04-25T17:55:04.021 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:55:03 smithi173 systemd[1]: Stopping Ceph mds.k for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:55:04.021 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:55:03 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-k[62196]: 2024-04-25T17:55:03.915+0000 7f68f73d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:55:04.021 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:55:03 smithi173 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-k[62196]: 2024-04-25T17:55:03.915+0000 7f68f73d0700 -1 mds.k *** got signal Terminated *** 2024-04-25T17:55:05.147 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:55:04 smithi173 podman[71723]: 2024-04-25 17:55:04.876972881 +0000 UTC m=+1.043586106 container died 48a48d1de11cafabf0c31d7f6e576ea5f3f83e1c42b5bc0f47646489d97abf9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-25T17:55:05.456 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:55:05 smithi173 podman[71723]: 2024-04-25 17:55:05.223131369 +0000 UTC m=+1.389744595 container cleanup 48a48d1de11cafabf0c31d7f6e576ea5f3f83e1c42b5bc0f47646489d97abf9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-25T17:55:05.456 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:55:05 smithi173 bash[71723]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-k 2024-04-25T17:55:05.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T17:55:05.725 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:55:05 smithi173 podman[71737]: 2024-04-25 17:55:05.457788277 +0000 UTC m=+0.572134773 container remove 48a48d1de11cafabf0c31d7f6e576ea5f3f83e1c42b5bc0f47646489d97abf9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True) 2024-04-25T17:55:05.725 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:55:05 smithi173 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.k.service: Deactivated successfully. 2024-04-25T17:55:05.725 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:55:05 smithi173 systemd[1]: Stopped Ceph mds.k for 79754014-0328-11ef-bc93-c7b262605968. 2024-04-25T17:55:05.725 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 17:55:05 smithi173 systemd[1]: ceph-79754014-0328-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.137s CPU time. 2024-04-25T17:55:05.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:55:05.736 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T17:55:05.736 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T17:55:05.736 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c 2024-04-25T17:55:06.095 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:55:05 smithi203 systemd[1]: Stopping Ceph mds.c for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:55:06.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:55:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[70008]: 2024-04-25T17:55:06.095+0000 7fe72a389700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:55:06.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:55:06 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c[70008]: 2024-04-25T17:55:06.095+0000 7fe72a389700 -1 mds.c *** got signal Terminated *** 2024-04-25T17:55:10.461 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:55:10 smithi203 podman[72803]: 2024-04-25 17:55:10.100645631 +0000 UTC m=+4.176606856 container died 9436a344c342df02358250f0b9762281e92e06b51366b45e6a602de1c288fc31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:55:10.861 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:55:10 smithi203 podman[72803]: 2024-04-25 17:55:10.56115519 +0000 UTC m=+4.637116416 container cleanup 9436a344c342df02358250f0b9762281e92e06b51366b45e6a602de1c288fc31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:55:10.861 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 17:55:10 smithi203 bash[72803]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-c 2024-04-25T17:55:10.995 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T17:55:11.029 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:55:11.029 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T17:55:11.029 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T17:55:11.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.f 2024-04-25T17:55:11.211 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:55:11 smithi203 systemd[1]: Stopping Ceph mds.f for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:55:11.711 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:55:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-f[63249]: 2024-04-25T17:55:11.339+0000 7fbfb59ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:55:11.711 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:55:11 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-f[63249]: 2024-04-25T17:55:11.339+0000 7fbfb59ed700 -1 mds.f *** got signal Terminated *** 2024-04-25T17:55:15.172 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:55:14 smithi203 podman[72934]: 2024-04-25 17:55:14.820973604 +0000 UTC m=+3.572523674 container died 2eb811a66a3f7e58215dab476cfdc81ae19d6e952f711ce2729516e883a42eee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T17:55:15.461 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:55:15 smithi203 podman[72934]: 2024-04-25 17:55:15.273208663 +0000 UTC m=+4.024758728 container cleanup 2eb811a66a3f7e58215dab476cfdc81ae19d6e952f711ce2729516e883a42eee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T17:55:15.461 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 17:55:15 smithi203 bash[72934]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-f 2024-04-25T17:55:15.743 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T17:55:15.775 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:55:15.775 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T17:55:15.775 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T17:55:15.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.i 2024-04-25T17:55:15.961 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:55:15 smithi203 systemd[1]: Stopping Ceph mds.i for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:55:16.461 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:55:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i[63871]: 2024-04-25T17:55:16.093+0000 7fbefcd75700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:55:16.461 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:55:16 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i[63871]: 2024-04-25T17:55:16.093+0000 7fbefcd75700 -1 mds.i *** got signal Terminated *** 2024-04-25T17:55:19.159 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:55:18 smithi203 podman[73056]: 2024-04-25 17:55:18.81625336 +0000 UTC m=+2.827877173 container died 8e4ed0f945d224be0531bc734f2839fefca641b1031ba0d1d04cb310be1c0b34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T17:55:19.159 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:55:19 smithi203 podman[73056]: 2024-04-25 17:55:19.160865457 +0000 UTC m=+3.172489250 container cleanup 8e4ed0f945d224be0531bc734f2839fefca641b1031ba0d1d04cb310be1c0b34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T17:55:19.461 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:55:19 smithi203 bash[73056]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i 2024-04-25T17:55:19.461 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 17:55:19 smithi203 podman[73070]: 2024-04-25 17:55:19.387097936 +0000 UTC m=+0.562395037 container remove 8e4ed0f945d224be0531bc734f2839fefca641b1031ba0d1d04cb310be1c0b34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-25T17:55:19.630 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T17:55:19.660 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:55:19.660 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T17:55:19.660 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T17:55:19.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-79754014-0328-11ef-bc93-c7b262605968@mds.l 2024-04-25T17:55:19.961 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:55:19 smithi203 systemd[1]: Stopping Ceph mds.l for 79754014-0328-11ef-bc93-c7b262605968... 2024-04-25T17:55:20.461 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:55:19 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-l[62940]: 2024-04-25T17:55:19.972+0000 7fdb0794b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T17:55:20.461 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:55:19 smithi203 ceph-79754014-0328-11ef-bc93-c7b262605968-mds-l[62940]: 2024-04-25T17:55:19.972+0000 7fdb0794b700 -1 mds.l *** got signal Terminated *** 2024-04-25T17:55:22.211 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:55:21 smithi203 podman[73178]: 2024-04-25 17:55:21.951155019 +0000 UTC m=+2.075183387 container died e1e5757a8f297031bbadb2f1a58944e0d73b9f7cf0ec1a17c26b9d27bb27e1a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T17:55:22.581 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:55:22 smithi203 podman[73178]: 2024-04-25 17:55:22.248240646 +0000 UTC m=+2.372269017 container cleanup e1e5757a8f297031bbadb2f1a58944e0d73b9f7cf0ec1a17c26b9d27bb27e1a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-25T17:55:22.582 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:55:22 smithi203 bash[73178]: ceph-79754014-0328-11ef-bc93-c7b262605968-mds-l 2024-04-25T17:55:22.582 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 17:55:22 smithi203 podman[73192]: 2024-04-25 17:55:22.473987962 +0000 UTC m=+0.519097900 container remove e1e5757a8f297031bbadb2f1a58944e0d73b9f7cf0ec1a17c26b9d27bb27e1a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-79754014-0328-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T17:55:22.599 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79754014-0328-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T17:55:22.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T17:55:22.629 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T17:55:22.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79754014-0328-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T17:55:22.819 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:55:25.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79754014-0328-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T17:55:25.395 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:55:27.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79754014-0328-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T17:55:27.883 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:55:30.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T17:55:30.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T17:55:30.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T17:55:30.356 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T17:55:30.359 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560/remote/smithi098/crash 2024-04-25T17:55:30.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/crash -- . 2024-04-25T17:55:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T17:55:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-04-25T17:55:30.408 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560/remote/smithi173/crash 2024-04-25T17:55:30.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/crash -- . 2024-04-25T17:55:30.456 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T17:55:30.457 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-04-25T17:55:30.458 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560/remote/smithi203/crash 2024-04-25T17:55:30.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/crash -- . 2024-04-25T17:55:30.502 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/79754014-0328-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T17:55:30.502 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-04-25T17:55:30.503 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T17:55:30.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-25T17:55:30.540 INFO:tasks.cephadm:Compressing logs... 2024-04-25T17:55:30.540 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T17:55:30.583 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T17:55:30.585 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T17:55:30.606 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.log 2024-04-25T17:55:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:: No such file or directory 2024-04-25T17:55:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T17:55:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T17:55:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T17:55:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T17:55:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T17:55:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T17:55:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T17:55:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T17:55:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mon.a.log: 90.4% 87.2% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T17:55:30.610 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T17:55:30.618 INFO:teuthology.orchestra.run.smithi203.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.log: No such file or directory 2024-04-25T17:55:30.618 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T17:55:30.618 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T17:55:30.618 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T17:55:30.618 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T17:55:30.619 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T17:55:30.619 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T17:55:30.619 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T17:55:30.619 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T17:55:30.619 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T17:55:30.619 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.2% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T17:55:30.619 INFO:teuthology.orchestra.run.smithi203.stderr: 90.6% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T17:55:30.625 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T17:55:30.625 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T17:55:30.625 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.2% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T17:55:30.625 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T17:55:30.625 INFO:teuthology.orchestra.run.smithi098.stderr: 94.5% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T17:55:30.625 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T17:55:30.625 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T17:55:30.625 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T17:55:30.625 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.5% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T17:55:30.626 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T17:55:30.626 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.5% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T17:55:30.626 INFO:teuthology.orchestra.run.smithi203.stderr: 92.3% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T17:55:30.626 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T17:55:30.627 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T17:55:30.627 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T17:55:30.627 INFO:teuthology.orchestra.run.smithi203.stderr: 83.8% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T17:55:30.627 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.log 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mgr.y.log: 90.5% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr: 87.2% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T17:55:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.10.log: 92.2% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T17:55:30.629 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T17:55:30.629 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T17:55:30.629 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T17:55:30.629 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T17:55:30.631 INFO:teuthology.orchestra.run.smithi098.stderr: 94.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T17:55:30.635 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T17:55:30.636 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T17:55:30.643 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T17:55:30.648 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T17:55:30.654 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T17:55:30.655 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T17:55:30.661 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.k.log: 91.0% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T17:55:30.669 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 94.5% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T17:55:30.670 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T17:55:30.674 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cephadm.log: 92.1% 94.4% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T17:55:30.674 INFO:teuthology.orchestra.run.smithi203.stderr: 91.0% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T17:55:30.676 INFO:teuthology.orchestra.run.smithi173.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T17:55:30.935 INFO:teuthology.orchestra.run.smithi173.stderr: 93.7% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T17:55:30.936 INFO:teuthology.orchestra.run.smithi203.stderr: 93.9% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T17:55:31.073 INFO:teuthology.orchestra.run.smithi098.stderr: 90.3% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T17:55:31.084 INFO:teuthology.orchestra.run.smithi203.stderr: 93.8% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T17:55:31.132 INFO:teuthology.orchestra.run.smithi203.stderr: 91.8% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T17:55:31.214 INFO:teuthology.orchestra.run.smithi173.stderr: 91.8% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T17:55:32.154 INFO:teuthology.orchestra.run.smithi098.stderr: 90.3% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T17:55:34.589 INFO:teuthology.orchestra.run.smithi203.stderr: 94.0% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T17:55:34.738 INFO:teuthology.orchestra.run.smithi173.stderr: 95.5% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T17:55:36.542 INFO:teuthology.orchestra.run.smithi173.stderr: 93.9% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T17:55:54.510 INFO:teuthology.orchestra.run.smithi098.stderr: 91.7% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T17:56:03.150 INFO:teuthology.orchestra.run.smithi203.stderr: 91.4% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T17:56:04.499 INFO:teuthology.orchestra.run.smithi173.stderr: 91.7% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T17:56:09.133 INFO:teuthology.orchestra.run.smithi173.stderr: 91.3% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T17:56:09.421 INFO:teuthology.orchestra.run.smithi203.stderr: 91.5% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T17:56:09.459 INFO:teuthology.orchestra.run.smithi203.stderr: 91.4% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T17:56:09.709 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T17:56:10.473 INFO:teuthology.orchestra.run.smithi203.stderr: 91.2% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T17:56:10.475 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T17:56:10.475 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m39.876s 2024-04-25T17:56:10.475 INFO:teuthology.orchestra.run.smithi203.stderr:user 2m30.406s 2024-04-25T17:56:10.475 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m4.632s 2024-04-25T17:56:12.275 INFO:teuthology.orchestra.run.smithi173.stderr: 91.2% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T17:56:16.950 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T17:56:18.369 INFO:teuthology.orchestra.run.smithi098.stderr: 91.2% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T17:56:18.371 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-04-25T17:56:18.371 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m47.775s 2024-04-25T17:56:18.371 INFO:teuthology.orchestra.run.smithi098.stderr:user 2m34.206s 2024-04-25T17:56:18.371 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m4.683s 2024-04-25T17:56:19.247 INFO:teuthology.orchestra.run.smithi173.stderr: 91.3% -- replaced with /var/log/ceph/79754014-0328-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T17:56:19.249 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-04-25T17:56:19.249 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m48.650s 2024-04-25T17:56:19.249 INFO:teuthology.orchestra.run.smithi173.stderr:user 2m47.815s 2024-04-25T17:56:19.249 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m5.108s 2024-04-25T17:56:19.249 INFO:tasks.cephadm:Archiving logs... 2024-04-25T17:56:19.249 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560/remote/smithi098/log 2024-04-25T17:56:19.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T17:56:44.482 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560/remote/smithi173/log 2024-04-25T17:56:44.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T17:57:10.935 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560/remote/smithi203/log 2024-04-25T17:57:10.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T17:57:35.546 INFO:tasks.cephadm:Removing cluster... 2024-04-25T17:57:35.546 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79754014-0328-11ef-bc93-c7b262605968 --force 2024-04-25T17:57:35.739 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:57:36.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79754014-0328-11ef-bc93-c7b262605968 --force 2024-04-25T17:57:36.438 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:57:36.918 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79754014-0328-11ef-bc93-c7b262605968 --force 2024-04-25T17:57:37.112 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: 79754014-0328-11ef-bc93-c7b262605968 2024-04-25T17:57:37.640 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T17:57:37.640 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T17:57:37.658 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T17:57:37.677 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T17:57:37.693 INFO:tasks.cephadm:Teardown complete 2024-04-25T17:57:37.694 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T17:57:37.711 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T17:57:37.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T17:57:37.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T17:57:37.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T17:57:37.835 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T17:57:37.847 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T17:57:37.847 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T17:57:37.850 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T17:57:37.852 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T17:57:37.861 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-04-25T17:57:37.865 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T17:57:37.865 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-04-25T17:57:37.865 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +608us[ +625us] +/- 47ms 2024-04-25T17:57:37.865 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -1408us[-1392us] +/- 70ms 2024-04-25T17:57:37.865 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +636us[ +653us] +/- 82ms 2024-04-25T17:57:37.865 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:57:37.866 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-04-25T17:57:37.866 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-25T17:57:37.870 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T17:57:37.870 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-25T17:57:37.870 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 +590us[ +574us] +/- 47ms 2024-04-25T17:57:37.870 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 -1418us[-1418us] +/- 70ms 2024-04-25T17:57:37.870 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +631us[ +631us] +/- 82ms 2024-04-25T17:57:37.870 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:57:37.872 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T17:57:37.872 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-04-25T17:57:37.872 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 +592us[ +595us] +/- 47ms 2024-04-25T17:57:37.872 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -1418us[-1418us] +/- 70ms 2024-04-25T17:57:37.872 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +638us[ +641us] +/- 82ms 2024-04-25T17:57:37.872 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:57:37.872 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T17:57:37.882 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T17:57:37.883 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T17:57:37.892 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T17:57:37.982 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T17:57:38.053 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T17:57:38.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T17:57:38.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:57:38.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T17:57:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:57:38.174 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T17:57:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:57:38.200 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T17:57:38.211 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T17:57:38.221 INFO:teuthology.task.internal:Duration was 2409.659051 seconds 2024-04-25T17:57:38.221 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T17:57:38.231 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T17:57:38.231 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T17:57:38.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T17:57:38.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T17:57:38.268 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T17:57:38.274 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T17:57:38.276 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T17:57:38.559 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T17:57:38.559 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:57:38.559 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T17:57:38.579 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T17:57:38.580 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T17:57:38.627 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T17:57:38.627 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T17:57:38.657 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T17:57:38.657 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T17:57:38.660 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T17:57:38.670 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T17:57:38.722 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T17:57:38.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T17:57:38.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T17:57:38.738 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T17:57:39.457 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T17:57:39.469 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T17:57:39.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T17:57:39.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T17:57:39.519 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T17:57:39.543 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T17:57:39.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:57:39.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:57:39.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:57:39.577 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-04-25T17:57:39.583 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-04-25T17:57:39.606 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-04-25T17:57:39.643 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:57:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:57:39.658 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:57:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:57:39.673 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:57:39.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:57:39.698 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T17:57:39.710 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T17:57:39.711 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560/remote/smithi098 2024-04-25T17:57:39.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T17:57:39.763 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560/remote/smithi173 2024-04-25T17:57:39.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T17:57:39.800 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560/remote/smithi203 2024-04-25T17:57:39.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T17:57:39.836 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T17:57:39.836 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T17:57:39.839 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T17:57:39.841 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T17:57:39.884 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T17:57:39.898 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T17:57:39.898 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T17:57:39.909 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T17:57:39.909 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T17:57:39.912 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T17:57:39.914 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T17:57:39.923 INFO:teuthology.orchestra.run.smithi098.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 17:57 /home/ubuntu/cephtest 2024-04-25T17:57:39.927 INFO:teuthology.orchestra.run.smithi173.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 17:57 /home/ubuntu/cephtest 2024-04-25T17:57:39.939 INFO:teuthology.orchestra.run.smithi203.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 17:57 /home/ubuntu/cephtest 2024-04-25T17:57:39.940 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T17:57:39.952 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T17:57:39.976 INFO:teuthology.nuke:Checking targets against current locks 2024-04-25T17:57:40.000 DEBUG:teuthology.nuke:shortname: smithi098 2024-04-25T17:57:40.001 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T17:57:40.021 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 17:11:52.628813', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNv7hWN3xpohe53vRVrnAB4MkRwwoJqHgMy5TMFJE7KVoRXVGBWbK//6orJckZSGDLq9weekpkgTK7IkWpjaAME='} 2024-04-25T17:57:40.027 DEBUG:teuthology.nuke:shortname: smithi173 2024-04-25T17:57:40.027 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T17:57:40.047 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 17:11:52.630778', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJ8S3aYybZ0BCo44CqPJv9BgZsE7uxa63inWi1qLMetgIZ5oPKN8ASaJCRdSHRCqiZ0dmKVDgQGV5U5WXrd3pA='} 2024-04-25T17:57:40.051 DEBUG:teuthology.nuke:shortname: smithi203 2024-04-25T17:57:40.051 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T17:57:40.069 INFO:teuthology.orchestra.console.smithi098:Power off 2024-04-25T17:57:40.070 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T17:57:40.080 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_16:54:09-fs-wip-lusov-quiescer-distro-default-smithi/7673560', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 17:11:52.632534', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH8irGOdpa7WxLAHIoGAsqHoyzfYqo/eum94UeChJ7UmDaWwbWu9Ix33CpBZrMtH9O/6tgpVpNKUG+yHYVzz7uE='} 2024-04-25T17:57:40.092 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-04-25T17:57:40.092 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:40.110 INFO:teuthology.orchestra.console.smithi173:Power off 2024-04-25T17:57:40.110 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T17:57:40.122 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-25T17:57:40.131 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-04-25T17:57:40.131 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:40.148 INFO:teuthology.orchestra.console.smithi203:Power off 2024-04-25T17:57:40.148 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T17:57:40.157 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-25T17:57:40.170 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-04-25T17:57:40.170 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:40.192 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-25T17:57:44.123 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:44.158 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:44.193 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:44.233 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-25T17:57:44.267 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-25T17:57:44.302 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-25T17:57:48.237 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:48.269 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:48.303 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:48.346 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-25T17:57:48.377 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-25T17:57:48.412 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-04-25T17:57:48.512 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-04-25T17:57:52.351 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:52.378 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:57:52.460 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-04-25T17:57:52.486 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-04-25T17:57:52.562 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-04-25T17:57:52.587 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-04-25T17:57:52.692 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} duration: 2409.6590507030487 failure_reason: 'error during quiesce thrashing: Error releasing set ''d4deafbb'': 1 (EPERM)' flavor: default owner: scheduled_leonidus@teuthology status: fail success: false 2024-04-25T17:57:52.693 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T17:57:52.776 INFO:teuthology.run:FAIL